The 'local time' determination of orbits was the root code that evolved into the current class. It started with the C/NOFS project. As real world data isn't always perfect, it includes filtering of the orbits that aren't present in the other orbit breakdowns (longitude, latitude, orbit number).

Plan :

• Break out filtering of local time orbits and generalize to any orbit
• Will likely need a variable input for the periodic value (local time is 24 hours, longitude is 360)
• Make the application of the filtering code optional

See pull #118

To close:

• Need to address #309
• Update default directory template - platform/name/inst_id/tag
• Need a method to translate from one directory structure to another (pysat 2 -> pysat 3)

RTD is having problems building pysat's develop branch. For some reason, the build is failing with, pandas 0.17.1 is installed but xarray requires versions > 0.19, or something like that. I don't know why it isn't just updating the pandas library right there......

It should work, but it doesn't. May not really be a pysat issue, but it still needs to be sorted out.

It would be very useful to be able to retrieve a range of dates for data that can be remotely accessed.

Pull request template needed. Also, Contributing.md needs to be updated to specify that pull requests should go into develop rather than master.

This affects COSMIC data:

Dear CDAAC Users,

Please note our Webpage now uses Secure Socket Layer (SSL) and thus it starts with https:// instead of http://
https://cdaac-www.cosmic.ucar.edu/cdaac/index.html

We have added a redirect from http:// to https://, so your bookmarks should just redirect.
However if you have any scripts that fetch data with http, please change your scripts to use https://

Madrigal will redownload files, appending a v*. This can cause a pysat file parsing crash. We need to improve pysat robustness to these situations.

The simple satellite simulations (instruments/pysat_testing.py) repeatedly use a np.mod function to create simple periodic signals (local time, longitude, etc.). These calls should be replaced with a function in the same file.

Despite providing a file for each data, the Madrigal JRO ISR data within a file may span multiple days. This is problematic when trying to do any of the longer term analysis things in pysat. Not sure what the most pysatic way to fix this is.

pysat currently assumes that files will be downloaded in a temporally consistent order. This means that the stored_file_info.txt files will assign dates between two files to the earlier file. This results in the loading of incorrect data and makes it impossible to use the .empty flag to download missing data.

This is probably not a priority but it would help with workflow for new users. Here is what help(pysat) outputs.

And here is what a more popular package outputs (help(pandas)).

Space weather indices, such as Dst, or solar indices, may require analysis on time scales longer than a day. The current Instrument object is optimized around a day. While it supports loading by file instead of day, not the most user friendly. Works for many basic data needs, however it doesn't offer all of pysat's functionality. All data should have best effort support from pysat.

I think we can fake monthly and yearly support without having to reimplement a new class. We may need a class, but it can just pull mostly everything from the current one.

Plan:

• Create a new class, MonthlyInstrument ? - to be determined
• subclass the Instrument object
• make a new load routine, optimized around loading by month, that just has a different front end but really calls the existing load
• when a load is requested by month, or by year, then the new load routine makes a call to the hidden load routine using filenames
• To make everything work out, the files for the monthly instrument need to be by month; yearly data by year. This can be faked on a per instrument basis, as needed.
• The iterators likely need some work in this setup
• Data padding may not work. Something to look at.
• Alternative is to create a new load routine with all of the features of the current one, but based around month
• The alternative is most ideally implemented by pulling out as much core code as possible so as to reduce the amount of code upkeep in the future.

To support future missions and pysat's use as a science operations center, basic functionality has been added to the files class to check if there are new files present. The idea is that pysat could run in the background, polling directories, and process files as they appear.

The is support for storing and loading the files in a directory, along with a basic routine to get a list of new files. Files are identified by name only.
Need support for :

• marking files as old after being processed
• marking files as new, so they can be processed again
• better identification of file, rather than just a filename

The command
sat[idx, :] = later_fit.data

where later_fit and sat are both Instrument objects. idx is a numpy array for indexing.

Produces the following error:

envs/Python_2/lib/python2.7/site-packages/pysat/_meta.pyc in setitem(self, names, input_data)
421 # convert given names into ones Meta has already seen
422 # if new, then input names become the standard
--> 423 names = [self.var_case_name(name) for name in names]

TypeError: 'slice' object is not iterable

Things don't work with
sat[idx] = later_fit.data
either.

From a fresh install and newly downloaded data in a conda virtual environment:

>>>ivm = pysat.Instrument(platform='cnofs', name='ivm')
>>>start = pysat.datetime(2013,12,30)
>>>stop = pysat.datetime(2014,1,30)
>>>ivm.bounds = (start, stop)
>>> ivm.load(date=stop)

This yields the output :

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~~~/anaconda3/envs/pysat_test/lib/python3.6/site-packages/pysat/_instrument.py", 
line 879, in load
    self.data, meta = self._load_data(date=self.date, fid=self._fid) 
  File "~~~/anaconda3/envs/pysat_test/lib/python3.6/site-packages/pysat/_instrument.py", 
line 622, in _load_data
    sat_id=self.sat_id, **self.kwargs)
  File "~~~/anaconda3/envs/pysat_test/lib/python3.6/site-packages/pysat/instruments/nasa_cdaweb_methods.py", 
line 157, in load
    return cdf.to_pysat(flatten_twod=flatten_twod)
  File "~~~/anaconda3/envs/pysat_test/lib/python3.6/site-packages/pysatCDF/_cdf.py", 
line 658, in to_pysat
    columns=new_names)
  File "~~~/anaconda3/envs/pysat_test/lib/python3.6/site-packages/pandas/core/frame.py", 
line 379, in __init__
    copy=copy)
  File "~~~/anaconda3/envs/pysat_test/lib/python3.6/site-packages/pandas/core/frame.py", 
line 527, in _init_ndarray
    index, columns = _get_axes(*values.shape)
  File "~~~/anaconda3/envs/pysat_test/lib/python3.6/site-packages/pandas/core/frame.py", 
line 484, in _get_axes
    index = _ensure_index(index)
  File "~~~/anaconda3/envs/pysat_test/lib/python3.6/site-packages/pandas/core/indexes/base.py", 
line 4974, in _ensure_index
    return Index(index_like)
  File "~~~/anaconda3/envs/pysat_test/lib/python3.6/site-packages/pandas/core/indexes/base.py", 
line 294, in __new__
    dtype=dtype, **kwargs)
  File "~~~/anaconda3/envs/pysat_test/lib/python3.6/site-packages/pandas/core/indexes/datetimes.py", 
line 422, in __new__
    subarr = conversion.ensure_datetime64ns(data)
  File "pandas/_libs/tslibs/conversion.pyx", line 119, in pandas._libs.tslibs.conversion.ensure_datetime64ns
  File "pandas/_libs/tslibs/np_datetime.pyx", line 121, in pandas._libs.tslibs.np_datetime.check_dts_bounds
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: -1-12-31 23:59:59

This appears to only happen on dates between 2014-01-04 and 2014-01-31
This doesn't happen for other dates

Extend OMNI, F10.7, and Kp routines to download forecasted values as well as observed or measured values.

OMNI HRO data is not loading when "with" statement is present.

This has happened on both my Mac OSX Sierra machines. The error I get is:

In [172]: omni.load(date=dtime)

AttributeError Traceback (most recent call last)
in ()
----> 1 omni.load(date=dtime)

/Users/ab763/Programs/Git/pysat/pysat/_instrument.py in load(self, yr, doy, date, fname, fid, verifyPad)
742 # if self.pad is False, load single day
743 else:
--> 744 self.data, meta = self._load_data(date=self.date, fid=self._fid)
745 if not self.data.empty:
746 self.meta = meta

/Users/ab763/Programs/Git/pysat/pysat/_instrument.py in _load_data(self, date, fid)
506 load_fname = [os.path.join(self.files.data_path, f) for f in fname]
507 data, mdata = self._load_rtn(load_fname, tag=self.tag,
--> 508 sat_id=self.sat_id, **self.kwargs)
509 else:
510 data = DataFrame(None)

/Users/ab763/Programs/Git/pysat/pysat/instruments/omni_hro.py in load(fnames, tag, sat_id)
120 fname = fnames[0][0:-11]
121 date = pysat.datetime.strptime(fnames[0][-10:], '%Y-%m-%d')
--> 122 with pysatCDF.CDF(fname) as cdf:
123 data, meta = cdf.to_pysat()
124 # pick out data for date

AttributeError: exit

And this went away when I removed the "with" statement.

Improve metadata docstring to include examples for setting custom metadata

Pandas get_duplicates is being dropped. It needs to be rewritten as stated in the testing output log when pandas brings up the deprecation.

One of the parameters for the Orbit class is the time period of the orbit. This is used to help filter the determined orbit breaks such that only the real breaks are kept. However, over a long data run, the satellite orbit period changes and can change enough that the orbit determination no longer works. One fix is to only have satellite data runs limited to a couple years, but that is avoiding the issue.

Plan:

• Modify the orbits class to store a running series of time differences between orbits. e.g. the difference between the start of each orbit

• Cleaning routines or other reasons can produce data sets that always produce incomplete orbit data. This requires thought.

• Stored periods need to be heavily filtered themselves.

• Changes in applied orbital period need to be slow

• Alternative: support accepting a list of dates and periods, and the class will use the closest past value when applying the orbits

• This is simpler and has less chance of doing funny things like the auto determination method

• The cost of the simpler method is more work by the user

• Ideal situation requires no input period whatsoever

The data access used by pysat:

ivm = pysat.Instrument('cnofs', 'ivm')
ivm.load(2009,1)
ivm[idx, 'glon'] = stuff

uses pandas.ix internally. Pysat.Instrument's getitem and setitem need updating to avoid .ix but retain functionality.

Currently pysat prints a bunch of different statements to the stdoutput. It would be much better if all of those statements were replaced with logging. Users could select how much feedback they get to terminal and to file.

Plan:

• Replace all of the print statements with logging statements

Currently users are required to go to the documentation or code to determine which tags are supported by each instrument. Instruments can be discovered at the command line, though this can be improved.

We need a routine that lists out the available instruments, what the tags are, etc.

Each instrument already has a dictionary with text descriptions that can be used. Routine needs to go through and determine what instruments are in pysat.instruments, import them, then print out salient info.

The list_files routine doesn't show up as a class function for ivm data:

cindi = pysat.Instrument("CNOFS", "IVM", clean_level="none")
cindi.list_files()

AttributeError Traceback (most recent call last)
in ()
----> 1 cindi.list_files()

AttributeError: 'Instrument' object has no attribute 'list_files'

Pandas Series warning for future deprecation:

Series.rolling_mean will be replaced with Series.rolling(kwargs).mean()

Functions 'collect_inst_model_pairs' and 'extract_modelled_observations' have the same docstring first line. Both are described as "Extracts instrument-aligned data from a modelled data set"

xarray has built in support for netCDF4 files. pysat manages both data and metadata, so pysat's to_netCDF4 and from_netcdf4 need to be modified to include xarray as the data, while maintaining pysat's metadata support.

To close:

• Incorporate xarray / netcdf4 interactions into unit tests
• Update to_netCDF4

I'm trying to test my shiny new custom functions for OMNI, but I'm running into an error using them. Fortunately (?) I get the same problem when I use existing custom functions. All of the custom functions perform fine on their own, it's a kernal issue. But I don't know anything significant about kernals. I had hoped the recent updates would solve the problem but they haven't:

import pysat
import pandas as pds
import numpy as np
import datetime as dt

Set the time

ttime = dt.datetime(2000,9,1)
inst = pysat.Instrument(platform="omni", name='hro', tag='1min', pad=pds.DateOffset(1))

Download the data, if you don't have it already

inst.download(ttime-dt.timedelta(days=1), ttime+dt.timedelta(days=1))

Add the Stoneback-written custom function

inst.custom.add(pysat.instruments.omni_hro.time_shift_to_magnetic_poles, kind='add')

Load the data!

inst.load(date=ttime)

Possibly get my error:

Initializing three day/file window
No omni hro 1min data for 08/31/00
Returning omni hro 1min data for 09/01/00
Returning omni hro 1min data for 09/02/00

ValueError Traceback (most recent call last)
in ()
----> 1 inst.load(date=ttime)

/Users/ab763/Programs/Git/pysat/pysat/_instrument.pyc in load(self, yr, doy, date, fname, fid, verifyPad)
758 # apply custom functions
759 if not self.data.empty:
--> 760 self.custom._apply_all(self)
761
762 # remove the excess padding, if any applied

/Users/ab763/Programs/Git/pysat/pysat/_custom.pyc in _apply_all(self, sat)
188 sat[name] = data
189 else:
--> 190 raise ValueError("kernel doesn't know what to do with returned data.")
191
192 # modifying loaded data

ValueError: kernel doesn't know what to do with returned data.

After initial xarray support is pulled into develop, use the included pysat testing instrument based on xarray and perform a number of typical pysat operations. Ensure custom functions work. Check existing functions written for pandas based Instrument object and determine how they behave with an xarray object.

Need to add 'random' variables to pysat testing, with known properties, so that the seasonal occurrence probability routines can be tested.

Plan:

• Add a signal to pysat_testing and pysat_testing2D that goes above a gate value some known percentage of the time. The percentage of times above the gate needs to vary spatially over longitude and latitude
• Another signal that varies over 3D bins
• Create tests that verifies the operation of the test signal as well as the seasonal occurrence routines.

pysat currently only works with local filesystems, particularly datasets with fixed-width filenames. pysat needs support for both local and remote systems, fixed width and delimited.

Features:

• Determine what files are available for download
• Keep data current with one command

Plan:

• Each Instrument needs to be upgraded with a remote_filelist method, that goes and grabs the list of remote files.
• create pysat files routine to grab a list of files from the os (in Files.from_os)
• Provide a pysat filelist parser to extract relevant information, date, version, revision, etc., based on fixed width names. (in from_os as well)
• Provide new filelist parser for delimited filenames, optional entry support can be tricky
• create pysat update routine that compares remote file information to local file information, downloads the difference

C/NOFS IVM cleaning routine uses one level of cleanliness for all data, but reliability criteria for drifts is more stringent than density values. We should update accordingly.

The scope of utils is getting too big. I suggest we break it up into themed submodules. For example, make utils a directory and have time, coords, stats, et cetera.

Some older instrument routines still include a '/' in the date portion of the filename. This can produce problems on other systems when the associated load routine parses the filename. Update to using a '-'
instead of the '/' and be sure to remove the leading path when grabbing date information.

Ensure that pysat pandas objects can be mapped intelligently to xarray and back, either natively or, more likely, create Instrument methods to transition from one form to the other. Some of this functionality is currently embedded in the to_netCDF4 and load_netCDF4.

Datetime inputs to pysat are presumed to only cover year, month, and day, though users may input others. A filter function has been added to the Instrument object to cover the load and download methods. A review of the code should be made to see if there are any other inputs that need to be filtered. See pull #95 that originated from enhacement request #83.

Couple renaming at Instrument level to changes at both the data and MetaData layer.

The current date format mm/dd/yyyy is confusing to anyone who is not based in the US. I recommend we switch to dd Month yyyy. This would make 2/4/2018 4 February 2018, and ensure it is not mistaken for 2 April 2018.

Look into the web services data serving and check if suitable for pysat.

Improve docstrings and other documentation in older instrument routines.

Replace all 'Loaded Orbit:%i' printouts in orbits class with calls to a single function that provides feedback. Note, do not try and fix the 'hack' to be zero indexed.

This work not only cleans up it the code, it also sets the stage for a 'trigger callback' design that will provide a foundation for a gui powered by the Instrument object itself.

Some missions use multiple satellites and require an ID to differentiate. Currently, there is no documentation on how to do this properly.

Support on windows need to be completed and verified.

The current from_os Files constructor is nominally a fixed width file name parser. That being said, it employs wildcards '' when searching the system os, allowing non-compliant filenames into the system, which then breaks. Switch from '' to using '?', one per expected character, in glob.

The current seasonal average supports a wiede variety of data in the pandas format, and all of it gets aligned as needed. With the basics of xarray in the xarray-init branch, support for this data format needs to be added to seasonal averaging.

"Documentation" is misspelled in Readme link

Remove all references to pandas objects in orbit code. Required by xarray-init branch. Use null_data attached to Instrument object.

The dates being read in for the JRO ISR data are wrong. The date assigned is the first day of month, rather than the actual day.

The traceback after asking for dates without instrument data is so long that the original error message is not visible. This can most likely be improved.

Code snippet and output:

dmsp[tag].load(yr=2000, doy=248)
Initializing three day/file window
Returning dmsp ivm f12 data for 09/03/00
Returning dmsp ivm f12 data for 09/04/00
Returning dmsp ivm f12 data for 09/05/00
Initializing three day/file window
No dmsp ivm_ephem f12 data for 09/03/00
No dmsp ivm_ephem f12 data for 09/04/00
No dmsp ivm_ephem f12 data for 09/05/00

IndexError Traceback (most recent call last)
in ()
----> 1 dmsp[tag].load(yr=2000, doy=248)

/net/ddngs.hpc/petastore/ion/lg/bin/pysat/pysat/_instrument.pyc in load(self, yr, doy, date, fname, fid, verifyPad)
757 # apply custom functions
758 if not self.data.empty:
--> 759 self.custom._apply_all(self)
760
761 # remove the excess padding, if any applied

/net/ddngs.hpc/petastore/ion/lg/bin/pysat/pysat/_custom.pyc in _apply_all(self, sat)
192 # modifying loaded data
193 if kind == 'modify':
--> 194 t = func(sat,*arg,**kwarg)
195 if t is not None:
196 raise ValueError('Modify functions should not return any information via return. '+

/net/ddngs.hpc/petastore/ion/lg/bin/pydineof/pydineof/dmsp.pyc in update_DMSP_ephemeris(dmsp, ephem)
87 date = dmsp.date
88 if ephem.date != date:
---> 89 ephem.load(date=date, verifyPad=True)
90 ephem.data = ephem.data.reindex(index = dmsp.data.index, method='pad')
91 ephem.data = ephem.data.interpolate('time')

/net/ddngs.hpc/petastore/ion/lg/bin/pysat/pysat/_instrument.pyc in load(self, yr, doy, date, fname, fid, verifyPad)
724 self.data = self.data.ix[first_pad : last_pad]
725 # want exclusive end slicing behavior from above
--> 726 if (self.data.index[-1] == last_pad) & (not want_last_pad):
727 self.data = self.data.iloc[:-1, :]
728

/net/ion/sc/1/agb073000/.local/lib/python2.7/site-packages/pandas/core/indexes/base.pyc in getitem(self, key)
1687
1688 if is_scalar(key):
-> 1689 return getitem(key)
1690
1691 if isinstance(key, slice):

IndexError: index -1 is out of bounds for axis 0 with size 0

Collect the coordinate transformations in pysatMagVect, along with the transformations to be added by A. Burrell in utils.py, into a pysat.coords submodule. Include outside packages into pysat's setup.py and make them available from within pysat.coords, presuming they are pip installable.