{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "D7UvwwCM0wC7"
},
"source": [
"# Notebook\n",
"\n",
"**Authors:** Colin Small (crs1031@wildcats.unh.edu), Matthew Argall (Matthew.Argall@unh.edu), Marek Petrik (Marek.Petrik@unh.edu)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "tdDrn6mBTUgT"
},
"source": [
"[](https://upload.wikimedia.org/wikipedia/commons/c/c9/NASA_Spacecraft_Finds_New_Magnetic_Process_in_Turbulent_Space.webm)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "rL5p5pJQ39Dd"
},
"source": [
"## Introduction\n",
"Global-scale energy flow throughout Earth’s magnetosphere is catalyzed by processes that occur at Earth’s magnetopause (MP) in the electron diffusion region (EDR) of magnetic reconnection. Until the launch of the Magnetospheric Multiscale (MMS) mission, only rare, fortuitous circumstances permitted a glimpse of the electron dynamics that break magnetic field lines and energize plasma. MMS employs automated burst triggers onboard the spacecraft and a Scientist-in-the-Loop (SITL) on the ground to select intervals likely to contain diffusion regions. Only low-resolution survey data is available to the SITL, which is insufficient to resolve electron dynamics. A strategy for the SITL, then, is to select all MP crossings. This has resulted in over 35 potential MP EDR encounters but is labor- and resource-intensive; after manual reclassification, just ∼ 0.7% of MP crossings, or 0.0001% of the mission lifetime during MMS’s first two years contained an EDR.\n",
"\n",
"In this notebook, we develop a Long-Short Term Memory (LSTM) neural network to detect magnetopause crossings and automate the SITL classification process. An LSTM developed with this notebook has been implemented in the MMS data stream to provide automated predictions to the SITL.\n",
"\n",
"\n",
"This model facilitates EDR studies and helps free-up mission operation costs by consolidating manual classification processes into automated routines."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "xRLV_B1CtKIP"
},
"source": [
"**Authors' notes:** \n",
"\n",
"1. This notebook was developed after the development of the original model in use at the SDC. We have tried our best to replicate the development steps and hyperparameters of that model, but we cannot guarantee that models developed with this notebook will exactly match the performance of the original.\n",
"\n",
"2. This notebook was designed on, and is best run on, Google Colab. It must either be run on Colab or on a machine with an NVIDIA GPU and cuDNN installed. If your machine does not have an NVIDIA GPU, does not have cuDNN installed, or if you run into issues running this notebook yourself, please open the notebook in Google Colab, which provides you with a virtual GPU to run the notebook. (If TF Keras is unable to identify a GPU to run on, make sure the notebook is set to use one by clicking the \"Runtime\" tab in the top menu bar, selecting \"Change runtime type\", selecting \"GPU\" in the dropdown menu under \"Hardware accelerator\", and clicking save. Colab will refresh your timetime, and you will need to re-run all cells.):"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "K8513yuEtKIP"
},
"source": [
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "4Em6RZ8Z6UFl"
},
"source": [
"## Import Libraries\n",
"\n",
"To start, we import the neccesary libraries for this notebook."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 318
},
"colab_type": "code",
"id": "3WshufyyiiLd",
"outputId": "fb577231-e76a-4f6e-e81d-1c191e5e164e"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: nasa-pymms in /usr/local/lib/python3.6/dist-packages (0.3.1)\n",
"Requirement already satisfied: tqdm>=4.36.1 in /usr/local/lib/python3.6/dist-packages (from nasa-pymms) (4.41.1)\n",
"Requirement already satisfied: matplotlib>=3.1.1 in /usr/local/lib/python3.6/dist-packages (from nasa-pymms) (3.2.2)\n",
"Requirement already satisfied: requests>=2.22.0 in /usr/local/lib/python3.6/dist-packages (from nasa-pymms) (2.23.0)\n",
"Requirement already satisfied: scipy>=1.4.1 in /usr/local/lib/python3.6/dist-packages (from nasa-pymms) (1.4.1)\n",
"Requirement already satisfied: cdflib in /usr/local/lib/python3.6/dist-packages (from nasa-pymms) (0.3.19)\n",
"Requirement already satisfied: numpy>=1.8 in /usr/local/lib/python3.6/dist-packages (from nasa-pymms) (1.18.5)\n",
"Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=3.1.1->nasa-pymms) (1.2.0)\n",
"Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=3.1.1->nasa-pymms) (0.10.0)\n",
"Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=3.1.1->nasa-pymms) (2.4.7)\n",
"Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=3.1.1->nasa-pymms) (2.8.1)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests>=2.22.0->nasa-pymms) (2020.6.20)\n",
"Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests>=2.22.0->nasa-pymms) (3.0.4)\n",
"Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests>=2.22.0->nasa-pymms) (1.24.3)\n",
"Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests>=2.22.0->nasa-pymms) (2.10)\n",
"Requirement already satisfied: attrs>=19.2.0 in /usr/local/lib/python3.6/dist-packages (from cdflib->nasa-pymms) (20.2.0)\n",
"Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from cycler>=0.10->matplotlib>=3.1.1->nasa-pymms) (1.15.0)\n"
]
}
],
"source": [
"!pip install nasa-pymms"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 87
},
"colab_type": "code",
"id": "MwnLUN2cUd4S",
"outputId": "784c11cf-a6ce-4e24-ad6e-2afe19051981"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"TensorFlow 1.x selected.\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Using TensorFlow backend.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Creating root data directory /root/data/mms\n",
"Creating root data directory /root/data/mms/dropbox\n"
]
}
],
"source": [
"from pathlib import Path\n",
"from sklearn import preprocessing\n",
"from tensorflow.keras.models import Sequential\n",
"from tensorflow.keras.layers import Dense, Dropout, LSTM, CuDNNLSTM, BatchNormalization, Bidirectional, Reshape, TimeDistributed\n",
"from tensorflow.keras.callbacks import TensorBoard, ModelCheckpoint\n",
"from matplotlib import pyplot\n",
"from sklearn.metrics import roc_curve, auc, confusion_matrix\n",
"from keras import backend as K\n",
"from pymms.sdc import mrmms_sdc_api as mms\n",
"import keras.backend.tensorflow_backend as tfb\n",
"import tensorflow as tf\n",
"import numpy as np\n",
"import pandas as pd\n",
"import tensorflow as tf\n",
"import matplotlib.pyplot as plt\n",
"plt.rcParams.update({'font.size': 18})\n",
"import datetime as dt\n",
"import os\n",
"import time\n",
"import sklearn\n",
"import scipy\n",
"import pickle\n",
"import random\n",
"import requests"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "x-LAsDIG3XD7"
},
"source": [
"## Download, Preprocess, and Format MMS Data"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "gWIpfE2zSCOP"
},
"source": [
"After installing and importinng the neccesary libraries, we download our training and validation data. "
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 407
},
"colab_type": "code",
"id": "mcytuTHUR8U8",
"outputId": "04175787-010b-43d0-9951-104ec5782768"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--2020-09-16 16:36:14-- https://zenodo.org/record/3884266/files/original_training_data.csv?download=1\n",
"Resolving zenodo.org (zenodo.org)... 188.184.117.155\n",
"Connecting to zenodo.org (zenodo.org)|188.184.117.155|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 447842635 (427M) [text/plain]\n",
"Saving to: ‘training_data.csv’\n",
"\n",
"training_data.csv 100%[===================>] 427.10M 7.45MB/s in 24s \n",
"\n",
"2020-09-16 16:36:39 (18.0 MB/s) - ‘training_data.csv’ saved [447842635/447842635]\n",
"\n",
"--2020-09-16 16:36:39-- https://zenodo.org/record/3884266/files/original_validation_data.csv?download=1\n",
"Resolving zenodo.org (zenodo.org)... 188.184.117.155\n",
"Connecting to zenodo.org (zenodo.org)|188.184.117.155|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 90314951 (86M) [text/plain]\n",
"Saving to: ‘validation_data.csv’\n",
"\n",
"validation_data.csv 100%[===================>] 86.13M 8.10MB/s in 10s \n",
"\n",
"2020-09-16 16:36:49 (8.45 MB/s) - ‘validation_data.csv’ saved [90314951/90314951]\n",
"\n"
]
}
],
"source": [
"!wget -O training_data.csv https://zenodo.org/record/3884266/files/original_training_data.csv?download=1\n",
"!wget -O validation_data.csv https://zenodo.org/record/3884266/files/original_validation_data.csv?download=1"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "aDg-HJ0MAtjS"
},
"source": [
"After downloading the training and validation data, we preprocess our training data in preparation for training the neural network."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "fp4mcnLfFZXw"
},
"source": [
"We first load the data we downloaded above. The data is a table of measurements from the MMS spacecraft, where each row represents individual measurements taken at a given time and where each column represents a feature (variable) recorded at that time. There is an additional column representing the ground truths for each measurement (whether this measurement was selected by a SITL or not). Then, we will adjust the formatting and datatypes of several of the columns and sort the data by the time of the measurement."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "tojTr8i472HY"
},
"outputs": [],
"source": [
"mms_data = pd.read_csv('training_data.csv', index_col=0, infer_datetime_format=True,\n",
"\t\t\t\t\t\t parse_dates=[0])"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 623
},
"colab_type": "code",
"id": "U2yZv1MwGqWQ",
"outputId": "78dc7f6b-3f8c-43e6-b7b3-0bda18e92966"
},
"outputs": [
{
"data": {
"text/html": [
"
\n", " | mms1_des_energyspectr_omni_fast_0 | \n", "mms1_des_energyspectr_omni_fast_1 | \n", "mms1_des_energyspectr_omni_fast_2 | \n", "mms1_des_energyspectr_omni_fast_3 | \n", "mms1_des_energyspectr_omni_fast_4 | \n", "mms1_des_energyspectr_omni_fast_5 | \n", "mms1_des_energyspectr_omni_fast_6 | \n", "mms1_des_energyspectr_omni_fast_7 | \n", "mms1_des_energyspectr_omni_fast_8 | \n", "mms1_des_energyspectr_omni_fast_9 | \n", "mms1_des_energyspectr_omni_fast_10 | \n", "mms1_des_energyspectr_omni_fast_11 | \n", "mms1_des_energyspectr_omni_fast_12 | \n", "mms1_des_energyspectr_omni_fast_13 | \n", "mms1_des_energyspectr_omni_fast_14 | \n", "mms1_des_energyspectr_omni_fast_15 | \n", "mms1_des_energyspectr_omni_fast_16 | \n", "mms1_des_energyspectr_omni_fast_17 | \n", "mms1_des_energyspectr_omni_fast_18 | \n", "mms1_des_energyspectr_omni_fast_19 | \n", "mms1_des_energyspectr_omni_fast_20 | \n", "mms1_des_energyspectr_omni_fast_21 | \n", "mms1_des_energyspectr_omni_fast_22 | \n", "mms1_des_energyspectr_omni_fast_23 | \n", "mms1_des_energyspectr_omni_fast_24 | \n", "mms1_des_energyspectr_omni_fast_25 | \n", "mms1_des_energyspectr_omni_fast_26 | \n", "mms1_des_energyspectr_omni_fast_27 | \n", "mms1_des_energyspectr_omni_fast_28 | \n", "mms1_des_energyspectr_omni_fast_29 | \n", "mms1_des_energyspectr_omni_fast_30 | \n", "mms1_des_numberdensity_fast | \n", "mms1_des_bulkv_dbcs_fast_0 | \n", "mms1_des_bulkv_dbcs_fast_1 | \n", "mms1_des_heatq_dbcs_fast_0 | \n", "mms1_des_heatq_dbcs_fast_1 | \n", "mms1_des_temppara_fast | \n", "mms1_des_tempperp_fast | \n", "mms1_des_prestensor_dbcs_fast_x1_y1 | \n", "mms1_des_prestensor_dbcs_fast_x2_y1 | \n", "... | \n", "mms1_dis_energyspectr_omni_fast_29 | \n", "mms1_dis_energyspectr_omni_fast_30 | \n", "mms1_dis_numberdensity_fast | \n", "mms1_dis_bulkv_dbcs_fast_0 | \n", "mms1_dis_bulkv_dbcs_fast_1 | \n", "mms1_dis_heatq_dbcs_fast_0 | \n", "mms1_dis_heatq_dbcs_fast_1 | \n", "mms1_dis_temppara_fast | \n", "mms1_dis_tempperp_fast | \n", "mms1_dis_prestensor_dbcs_fast_x1_y1 | \n", "mms1_dis_prestensor_dbcs_fast_x2_y1 | \n", "mms1_dis_prestensor_dbcs_fast_x2_y2 | \n", "mms1_dis_prestensor_dbcs_fast_x3_y1 | \n", "mms1_dis_prestensor_dbcs_fast_x3_y2 | \n", "mms1_dis_prestensor_dbcs_fast_x3_y3 | \n", "mms1_dis_temptensor_dbcs_fast_x1_y1 | \n", "mms1_dis_temptensor_dbcs_fast_x2_y1 | \n", "mms1_dis_temptensor_dbcs_fast_x2_y2 | \n", "mms1_dis_temptensor_dbcs_fast_x3_y1 | \n", "mms1_dis_temptensor_dbcs_fast_x3_y2 | \n", "mms1_dis_temptensor_dbcs_fast_x3_y3 | \n", "mms1_dis_temp_anisotropy | \n", "mms1_dis_scalar_temperature | \n", "mms1_dis_N_Q | \n", "mms1_dis_Vz_Q | \n", "mms1_dis_nV_Q | \n", "mms1_afg_srvy_dmpa_Bx | \n", "mms1_afg_srvy_dmpa_By | \n", "mms1_afg_srvy_dmpa_Bz | \n", "mms1_afg_srvy_dmpa_|B| | \n", "mms1_afg_magnetic_pressure | \n", "mms1_afg_clock_angle | \n", "mms1_afg_Bz_Q | \n", "mms1_edp_x | \n", "mms1_edp_y | \n", "mms1_edp_z | \n", "mms1_edp_|E| | \n", "mms1_temp_ratio | \n", "mms1_plasma_beta | \n", "selected | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Epoch | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
2017-01-01 01:49:08.736524 | \n", "172560370.0 | \n", "141811650.0 | \n", "115564310.0 | \n", "103489660.0 | \n", "109156240.0 | \n", "138017710.0 | \n", "199794740.0 | \n", "313011970.0 | \n", "495821400.0 | \n", "7.675890e+08 | \n", "1.102645e+09 | \n", "1.355750e+09 | \n", "1.283706e+09 | \n", "876457860.0 | \n", "437018100.0 | \n", "178497440.0 | \n", "61324772.0 | \n", "17706034.0 | \n", "5950672.0 | \n", "1453186.0 | \n", "1230702.8 | \n", "1290793.0 | \n", "1361365.00 | \n", "1443553.0 | \n", "1538581.10 | \n", "1600447.1 | \n", "1693706.6 | \n", "1860683.8 | \n", "1821310.1 | \n", "1785482.6 | \n", "2143655.0 | \n", "47.289574 | \n", "-64.066220 | \n", "-59.850384 | \n", "0.067417 | \n", "-0.033087 | \n", "68.663414 | \n", "69.643720 | \n", "0.530681 | \n", "0.000513 | \n", "... | \n", "1847614.4 | \n", "1368769.60 | \n", "45.166210 | \n", "-65.301070 | \n", "-75.960210 | \n", "-0.017214 | \n", "-0.054081 | \n", "496.30145 | \n", "558.62010 | \n", "4.105384 | \n", "0.039305 | \n", "3.979379 | \n", "0.077544 | \n", "0.372712 | \n", "3.591420 | \n", "4.105384 | \n", "0.039305 | \n", "3.979379 | \n", "0.077544 | \n", "0.372712 | \n", "3.591420 | \n", "-0.111558 | \n", "537.84720 | \n", "0.000000 | \n", "0.000000 | \n", "5217.443037 | \n", "-0.026197 | \n", "33.162117 | \n", "-39.656940 | \n", "51.695260 | \n", "2.126628e+09 | \n", "1.571586 | \n", "1.467709 | \n", "-0.862275 | \n", "1.665013 | \n", "0.827068 | \n", "2.049348 | \n", "1.0 | \n", "524.89600 | \n", "False | \n", "
2017-01-01 01:49:13.236552 | \n", "160474430.0 | \n", "134115120.0 | \n", "113082936.0 | \n", "106597736.0 | \n", "118977260.0 | \n", "157128700.0 | \n", "234196500.0 | \n", "368190240.0 | \n", "588617700.0 | \n", "9.143196e+08 | \n", "1.280595e+09 | \n", "1.386385e+09 | \n", "1.020292e+09 | \n", "546922600.0 | \n", "238565300.0 | \n", "95543030.0 | \n", "34221956.0 | \n", "10795790.0 | \n", "3346869.8 | \n", "1132098.6 | \n", "1230606.6 | \n", "1290689.8 | \n", "1370548.60 | \n", "1217164.6 | \n", "1303508.00 | \n", "1396131.6 | \n", "1496823.2 | \n", "1930027.9 | \n", "1878429.5 | \n", "675747.1 | \n", "2085773.5 | \n", "47.806107 | \n", "-34.529873 | \n", "-128.342510 | \n", "0.091984 | \n", "-0.000412 | \n", "59.059840 | \n", "62.034630 | \n", "0.473132 | \n", "0.000187 | \n", "... | \n", "1385553.0 | \n", "1053661.50 | \n", "46.629120 | \n", "-60.415287 | \n", "-118.971080 | \n", "-0.003761 | \n", "0.163968 | \n", "415.75192 | \n", "523.83150 | \n", "3.883746 | \n", "0.074054 | \n", "3.943084 | \n", "0.049551 | \n", "0.003202 | \n", "3.105979 | \n", "3.883746 | \n", "0.074054 | \n", "3.943084 | \n", "0.049551 | \n", "0.003202 | \n", "3.105979 | \n", "-0.206325 | \n", "487.80496 | \n", "1.097182 | \n", "39.256162 | \n", "6031.144668 | \n", "3.003209 | \n", "-17.920328 | \n", "-18.677692 | \n", "26.057890 | \n", "5.403419e+08 | \n", "-1.404753 | \n", "2.396316 | \n", "-0.572842 | \n", "0.670425 | \n", "-0.570290 | \n", "1.050165 | \n", "1.0 | \n", "929.00616 | \n", "False | \n", "
2017-01-01 01:49:17.736573 | \n", "143446660.0 | \n", "120096200.0 | \n", "103888320.0 | \n", "99487384.0 | \n", "114008050.0 | \n", "152906400.0 | \n", "228558100.0 | \n", "359750530.0 | \n", "572418500.0 | \n", "8.799912e+08 | \n", "1.192516e+09 | \n", "1.208780e+09 | \n", "8.369016e+08 | \n", "437589570.0 | \n", "191214960.0 | \n", "78865330.0 | \n", "28987052.0 | \n", "9752265.0 | \n", "3037489.2 | \n", "1132141.2 | \n", "1230653.4 | \n", "1290740.1 | \n", "1361307.60 | \n", "1453176.6 | \n", "1303684.90 | \n", "1417762.9 | \n", "1178981.4 | \n", "1254457.6 | \n", "1834328.2 | \n", "1900604.5 | \n", "1562550.4 | \n", "43.848970 | \n", "-17.841991 | \n", "-80.636430 | \n", "-0.017504 | \n", "0.007353 | \n", "55.005360 | \n", "61.100243 | \n", "0.432611 | \n", "0.002111 | \n", "... | \n", "1216220.0 | \n", "1003711.90 | \n", "42.998802 | \n", "-37.589535 | \n", "-101.116325 | \n", "0.002284 | \n", "0.062132 | \n", "379.23505 | \n", "442.85294 | \n", "3.202901 | \n", "-0.023277 | \n", "2.898829 | \n", "0.074439 | \n", "-0.131550 | \n", "2.612594 | \n", "3.202901 | \n", "-0.023277 | \n", "2.898829 | \n", "0.074439 | \n", "-0.131550 | \n", "2.612594 | \n", "-0.143655 | \n", "421.64697 | \n", "1.625556 | \n", "47.437222 | \n", "5204.241455 | \n", "0.250873 | \n", "-3.805478 | \n", "-25.772310 | \n", "26.052958 | \n", "5.401373e+08 | \n", "-1.504967 | \n", "1.586905 | \n", "-0.357424 | \n", "0.633575 | \n", "-0.160308 | \n", "0.744894 | \n", "1.0 | \n", "1132.09920 | \n", "False | \n", "
2017-01-01 01:49:22.236602 | \n", "143115380.0 | \n", "120483730.0 | \n", "103335910.0 | \n", "99983890.0 | \n", "115342700.0 | \n", "156445900.0 | \n", "236307790.0 | \n", "374869920.0 | \n", "598786900.0 | \n", "9.304425e+08 | \n", "1.259985e+09 | \n", "1.203378e+09 | \n", "7.605617e+08 | \n", "361202240.0 | \n", "147414610.0 | \n", "58348600.0 | \n", "21312232.0 | \n", "7609467.0 | \n", "3276093.5 | \n", "1132188.6 | \n", "1230705.2 | \n", "1290795.8 | \n", "1361368.00 | \n", "1443556.0 | \n", "1548742.10 | \n", "1391640.8 | \n", "1787557.8 | \n", "1927728.0 | \n", "1754844.5 | \n", "1890987.5 | \n", "2347546.8 | \n", "44.116190 | \n", "-24.291311 | \n", "-51.504208 | \n", "-0.010656 | \n", "-0.065078 | \n", "56.319736 | \n", "57.496597 | \n", "0.405184 | \n", "0.001348 | \n", "... | \n", "1146827.8 | \n", "996494.75 | \n", "42.281338 | \n", "-15.571530 | \n", "-88.482050 | \n", "-0.039505 | \n", "0.037287 | \n", "350.79077 | \n", "375.92334 | \n", "2.608262 | \n", "0.016238 | \n", "2.484871 | \n", "-0.012547 | \n", "-0.062605 | \n", "2.376314 | \n", "2.608262 | \n", "0.016238 | \n", "2.484871 | \n", "-0.012547 | \n", "-0.062605 | \n", "2.376314 | \n", "-0.066856 | \n", "367.54580 | \n", "2.437950 | \n", "63.604532 | \n", "4158.129041 | \n", "0.571565 | \n", "-26.909445 | \n", "-33.616943 | \n", "43.064415 | \n", "1.475799e+09 | \n", "-1.549559 | \n", "0.414231 | \n", "1.962965 | \n", "1.813941 | \n", "-1.001314 | \n", "2.854163 | \n", "1.0 | \n", "257.55072 | \n", "False | \n", "
2017-01-01 01:49:26.736624 | \n", "154996270.0 | \n", "129635576.0 | \n", "109879810.0 | \n", "104667430.0 | \n", "118939120.0 | \n", "159722750.0 | \n", "240743730.0 | \n", "384687520.0 | \n", "622868740.0 | \n", "9.923178e+08 | \n", "1.407102e+09 | \n", "1.379264e+09 | \n", "8.729052e+08 | \n", "412877600.0 | \n", "165255970.0 | \n", "64202252.0 | \n", "22482496.0 | \n", "7437224.5 | \n", "3127818.0 | \n", "1132103.5 | \n", "1230611.9 | \n", "1290695.2 | \n", "1361258.80 | \n", "1443435.9 | \n", "1492447.10 | \n", "1570121.2 | \n", "1725425.4 | \n", "1605647.4 | \n", "2037316.2 | \n", "1869736.6 | \n", "2320483.0 | \n", "47.787876 | \n", "-43.337650 | \n", "-68.804990 | \n", "-0.138768 | \n", "-0.233581 | \n", "56.954014 | \n", "58.321754 | \n", "0.449897 | \n", "-0.001409 | \n", "... | \n", "1509011.9 | \n", "1205417.50 | \n", "44.735374 | \n", "-29.033060 | \n", "-83.403510 | \n", "-0.047671 | \n", "-0.056385 | \n", "396.54170 | \n", "400.25833 | \n", "2.947306 | \n", "0.007026 | \n", "2.790271 | \n", "-0.076785 | \n", "-0.311423 | \n", "2.842150 | \n", "2.947306 | \n", "0.007026 | \n", "2.790271 | \n", "-0.076785 | \n", "-0.311423 | \n", "2.842150 | \n", "-0.009286 | \n", "399.01944 | \n", "0.083262 | \n", "57.780416 | \n", "4519.709417 | \n", "-1.338079 | \n", "-25.596290 | \n", "-35.260128 | \n", "43.591710 | \n", "1.512161e+09 | \n", "-1.623025 | \n", "0.009667 | \n", "1.674221 | \n", "0.755573 | \n", "-0.422870 | \n", "1.884867 | \n", "1.0 | \n", "423.39264 | \n", "False | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
2017-01-31 01:59:37.612761 | \n", "195522660.0 | \n", "169047760.0 | \n", "151144740.0 | \n", "155255970.0 | \n", "187461070.0 | \n", "260878460.0 | \n", "391024700.0 | \n", "601519170.0 | \n", "933239230.0 | \n", "1.431671e+09 | \n", "2.001644e+09 | \n", "1.911962e+09 | \n", "9.213409e+08 | \n", "266009100.0 | \n", "70228190.0 | \n", "23561248.0 | \n", "8219571.0 | \n", "3132344.0 | \n", "1069388.6 | \n", "1156007.0 | \n", "1205684.8 | \n", "1285194.1 | \n", "776630.40 | \n", "1443075.8 | \n", "1507529.50 | \n", "1614608.1 | \n", "1747582.8 | \n", "1609458.8 | \n", "2050434.2 | \n", "2227281.8 | \n", "2083288.2 | \n", "66.366800 | \n", "-114.178600 | \n", "-66.354935 | \n", "0.118533 | \n", "-0.167590 | \n", "54.738000 | \n", "49.998760 | \n", "0.542538 | \n", "0.017298 | \n", "... | \n", "1173978.5 | \n", "946332.50 | \n", "46.211296 | \n", "-99.599365 | \n", "-137.597990 | \n", "-0.087668 | \n", "-0.111775 | \n", "269.56824 | \n", "291.43158 | \n", "2.067940 | \n", "-0.059106 | \n", "2.042181 | \n", "0.032945 | \n", "0.023344 | \n", "2.201118 | \n", "2.067940 | \n", "-0.059106 | \n", "2.042181 | \n", "0.032945 | \n", "0.023344 | \n", "2.201118 | \n", "-0.075020 | \n", "284.14380 | \n", "2.026775 | \n", "10.799598 | \n", "722.083772 | \n", "-12.985076 | \n", "-21.649717 | \n", "3.804093 | \n", "25.530249 | \n", "5.186808e+08 | \n", "-2.111054 | \n", "1.825885 | \n", "0.569317 | \n", "-1.014906 | \n", "-0.478203 | \n", "1.258107 | \n", "1.0 | \n", "451.70038 | \n", "False | \n", "
2017-01-31 01:59:42.112805 | \n", "188999890.0 | \n", "167058830.0 | \n", "153385100.0 | \n", "160276850.0 | \n", "194781820.0 | \n", "265345520.0 | \n", "388861400.0 | \n", "597449200.0 | \n", "947348600.0 | \n", "1.491336e+09 | \n", "1.999952e+09 | \n", "1.605463e+09 | \n", "6.863398e+08 | \n", "206139580.0 | \n", "60198680.0 | \n", "21560316.0 | \n", "8027363.5 | \n", "3163052.5 | \n", "1069362.9 | \n", "1155979.5 | \n", "1205655.9 | \n", "1227593.4 | \n", "1093269.80 | \n", "1310439.9 | \n", "1257120.90 | \n", "1626893.6 | \n", "1687317.9 | \n", "1547724.2 | \n", "1889413.2 | \n", "1736235.6 | \n", "2222297.0 | \n", "64.247070 | \n", "-114.180830 | \n", "-57.873524 | \n", "0.079878 | \n", "-0.027051 | \n", "49.837917 | \n", "48.950280 | \n", "0.507997 | \n", "-0.001615 | \n", "... | \n", "1173978.5 | \n", "946332.50 | \n", "46.211296 | \n", "-99.599365 | \n", "-137.597990 | \n", "-0.087668 | \n", "-0.111775 | \n", "269.56824 | \n", "291.43158 | \n", "2.067940 | \n", "-0.059106 | \n", "2.042181 | \n", "0.032945 | \n", "0.023344 | \n", "2.201118 | \n", "2.067940 | \n", "-0.059106 | \n", "2.042181 | \n", "0.032945 | \n", "0.023344 | \n", "2.201118 | \n", "-0.075020 | \n", "284.14380 | \n", "2.026775 | \n", "10.799598 | \n", "722.083772 | \n", "-15.436447 | \n", "11.214994 | \n", "4.809845 | \n", "19.677261 | \n", "3.081197e+08 | \n", "2.513284 | \n", "2.099205 | \n", "1.721005 | \n", "-1.033757 | \n", "1.900864 | \n", "2.764742 | \n", "1.0 | \n", "205.54817 | \n", "False | \n", "
2017-01-31 01:59:46.612839 | \n", "156668940.0 | \n", "136574260.0 | \n", "123834980.0 | \n", "129313670.0 | \n", "158046480.0 | \n", "222452110.0 | \n", "343151600.0 | \n", "552882500.0 | \n", "898671700.0 | \n", "1.377260e+09 | \n", "1.645098e+09 | \n", "1.142623e+09 | \n", "4.697292e+08 | \n", "148775630.0 | \n", "47467200.0 | \n", "18703922.0 | \n", "7264802.5 | \n", "2508120.0 | \n", "1085820.6 | \n", "1155923.6 | \n", "1167832.6 | \n", "1042989.0 | \n", "1352492.10 | \n", "1176428.9 | \n", "1383997.80 | \n", "1534980.2 | \n", "1675194.2 | \n", "1783336.1 | \n", "1992431.1 | \n", "1171974.1 | \n", "2034246.5 | \n", "53.937880 | \n", "-147.811950 | \n", "-66.332436 | \n", "0.031849 | \n", "0.021172 | \n", "48.725334 | \n", "47.522090 | \n", "0.415525 | \n", "-0.001351 | \n", "... | \n", "1173978.5 | \n", "946332.50 | \n", "46.211296 | \n", "-99.599365 | \n", "-137.597990 | \n", "-0.087668 | \n", "-0.111775 | \n", "269.56824 | \n", "291.43158 | \n", "2.067940 | \n", "-0.059106 | \n", "2.042181 | \n", "0.032945 | \n", "0.023344 | \n", "2.201118 | \n", "2.067940 | \n", "-0.059106 | \n", "2.042181 | \n", "0.032945 | \n", "0.023344 | \n", "2.201118 | \n", "-0.075020 | \n", "284.14380 | \n", "2.026775 | \n", "10.799598 | \n", "722.083772 | \n", "9.732903 | \n", "-5.534757 | \n", "21.615046 | \n", "24.342825 | \n", "4.715547e+08 | \n", "-0.517060 | \n", "2.347574 | \n", "2.787715 | \n", "-2.293623 | \n", "-2.082477 | \n", "4.167586 | \n", "1.0 | \n", "136.35893 | \n", "False | \n", "
2017-01-31 01:59:51.112882 | \n", "148339340.0 | \n", "131617280.0 | \n", "123383950.0 | \n", "133689650.0 | \n", "169121890.0 | \n", "242394990.0 | \n", "371708160.0 | \n", "592586500.0 | \n", "940750800.0 | \n", "1.356828e+09 | \n", "1.457272e+09 | \n", "9.310631e+08 | \n", "3.592990e+08 | \n", "107206430.0 | \n", "34867150.0 | \n", "14601406.0 | \n", "5957955.0 | \n", "1804840.9 | \n", "1102969.0 | \n", "1155993.6 | \n", "1165338.0 | \n", "1181596.9 | \n", "630540.94 | \n", "1371131.6 | \n", "1339956.00 | \n", "1473016.6 | \n", "1658352.9 | \n", "1457381.9 | \n", "1135691.6 | \n", "2071354.1 | \n", "2416306.0 | \n", "52.308780 | \n", "-141.634200 | \n", "-85.162560 | \n", "-0.031770 | \n", "0.035757 | \n", "45.698080 | \n", "44.961994 | \n", "0.374102 | \n", "-0.003214 | \n", "... | \n", "1173978.5 | \n", "946332.50 | \n", "46.211296 | \n", "-99.599365 | \n", "-137.597990 | \n", "-0.087668 | \n", "-0.111775 | \n", "269.56824 | \n", "291.43158 | \n", "2.067940 | \n", "-0.059106 | \n", "2.042181 | \n", "0.032945 | \n", "0.023344 | \n", "2.201118 | \n", "2.067940 | \n", "-0.059106 | \n", "2.042181 | \n", "0.032945 | \n", "0.023344 | \n", "2.201118 | \n", "-0.075020 | \n", "284.14380 | \n", "2.026775 | \n", "10.799598 | \n", "722.083772 | \n", "13.513677 | \n", "-6.461386 | \n", "29.848564 | \n", "33.396190 | \n", "8.875320e+08 | \n", "-0.446005 | \n", "3.369051 | \n", "3.261839 | \n", "-4.248680 | \n", "-2.128106 | \n", "5.763654 | \n", "1.0 | \n", "98.59849 | \n", "False | \n", "
2017-01-31 01:59:55.612917 | \n", "135440930.0 | \n", "119457550.0 | \n", "111787400.0 | \n", "121276030.0 | \n", "155176540.0 | \n", "227165100.0 | \n", "355259520.0 | \n", "570692000.0 | \n", "891547700.0 | \n", "1.239421e+09 | \n", "1.285931e+09 | \n", "8.412920e+08 | \n", "3.517979e+08 | \n", "116526360.0 | \n", "41865636.0 | \n", "18613406.0 | \n", "8321585.5 | \n", "3825070.0 | \n", "1326246.5 | \n", "1155995.0 | \n", "1205672.4 | \n", "1224776.4 | \n", "1275854.50 | \n", "1217636.2 | \n", "838192.25 | \n", "1057988.5 | \n", "1786348.4 | \n", "1358997.8 | \n", "2065777.1 | \n", "2144108.0 | \n", "2296880.8 | \n", "48.376137 | \n", "-109.131290 | \n", "-104.492080 | \n", "-0.023458 | \n", "-0.053285 | \n", "46.801380 | \n", "45.059790 | \n", "0.350553 | \n", "-0.002084 | \n", "... | \n", "1173978.5 | \n", "946332.50 | \n", "46.211296 | \n", "-99.599365 | \n", "-137.597990 | \n", "-0.087668 | \n", "-0.111775 | \n", "269.56824 | \n", "291.43158 | \n", "2.067940 | \n", "-0.059106 | \n", "2.042181 | \n", "0.032945 | \n", "0.023344 | \n", "2.201118 | \n", "2.067940 | \n", "-0.059106 | \n", "2.042181 | \n", "0.032945 | \n", "0.023344 | \n", "2.201118 | \n", "-0.075020 | \n", "284.14380 | \n", "2.026775 | \n", "10.799598 | \n", "722.083772 | \n", "-9.631133 | \n", "29.820732 | \n", "-9.856265 | \n", "32.850887 | \n", "8.587848e+08 | \n", "1.883189 | \n", "3.069619 | \n", "-0.251488 | \n", "1.193291 | \n", "0.494789 | \n", "1.316057 | \n", "1.0 | \n", "431.81090 | \n", "False | \n", "
302188 rows × 124 columns
\n", "\n", " | EndDate | \n", "
---|---|
BeginDate | \n", "\n", " |
2017-02-03 22:27:28.784841 | \n", "2017-02-03 22:29:03.285435 | \n", "
2017-02-03 22:31:13.786249 | \n", "2017-02-03 22:33:55.787267 | \n", "
2017-02-03 23:35:52.811378 | \n", "2017-02-03 23:56:48.319683 | \n", "
2017-02-04 00:09:19.824671 | \n", "2017-02-04 00:16:04.827367 | \n", "
2017-02-04 00:38:52.836503 | \n", "2017-02-04 00:41:03.337379 | \n", "
... | \n", "... | \n", "
2017-02-09 08:00:15.435380 | \n", "2017-02-09 08:00:24.435452 | \n", "
2017-02-09 08:00:51.435666 | \n", "2017-02-09 08:01:27.435952 | \n", "
2017-02-09 08:02:52.936624 | \n", "2017-02-09 08:07:00.438590 | \n", "
2017-02-09 08:15:01.942399 | \n", "2017-02-09 08:46:04.957144 | \n", "
2017-02-09 09:43:13.984273 | \n", "2017-02-09 09:46:58.986053 | \n", "
97 rows × 1 columns
\n", "