ceholden / tstools Goto Github PK

Passing around reference to the currently opened timeseries is stupid. Move this into a module which keeps track of storing reference. This would:

• enable us to keep more than one open at a time
• remove a lot of unrelated code from places (e.g. finding classes that inherit from AbstractTimeSeries)
• change some stupid design decisions which passed ts through several classes
• have more flexible init and re-init for when things change
• test more easily

Add an option to draw a point or box in/around the pixel selected by plot tool

Stop using print statements and use logging module for debug and for things not shown in QgsMessageBar

Create series of tabs for each series of configuration options?

1. Location
   • Stack location
   • Stack folder name pattern (LND*)
   • Stack file name pattern (*stack)
2. Imagery
   • Default bands
   • Default stretch
3. CCDC
   • What kind of fit, how many parameters, thresholds, etc

Abstract two things from controller to the time series drivers:

1. Adding of images
   • Ask to return a list of filenames for some index
   • If len(files) > 1, then add all but group them from controller
   • This way we can have tiled images as one product
2. Fetching of data - don't ask for a pixel X/Y, ask for a coordinate XY and let driver handle the conversion to pixel
   • the biggest thing this enables are stacks that aren't perfectly aligned to each other
   • ask for the coordinate in the projection of the timeseries driver's imagery (at least do that conversion)

Add them...

Issue

CCDC v11 doesn't always work with the DOY plot for some of the simpler fits. Root cause of problem is TS driver which can feed empty np.array objects.

Empty predictions for first fit

ipdb> ts.get_prediction(4)[0][0]
array([], dtype=float64)
ipdb> ts.get_prediction(4)[1][0]
array([], dtype=float64)

TS result record

ipdb> ts.result[0]
{'category': 3, 'rmse': array([0, 0, 0, 0, 0, 0, 0], dtype=uint8), 'magnitude': array([ -340.09516338,  -275.7719292 ,  -253.40984124,  -329.00342285,
        -423.18663196,  -250.63092108, -2455.        ]), 'pos': 48804583, 't_start': 728020, 'change_prob': -0.16666666666666666, 't_break': 0, 'num_obs': 1, 't_end': 728020, 'coefs': array([[ 768,  910,  866, 2623, 2421, 1351, 2455],
       [   0,    0,    0,    0,    0,    0,    0],
       [   0,    0,    0,    0,    0,    0,    0],
       [   0,    0,    0,    0,    0,    0,    0],
       [   0,    0,    0,    0,    0,    0,    0],
       [   0,    0,    0,    0,    0,    0,    0],
       [   0,    0,    0,    0,    0,    0,    0],
       [   0,    0,    0,    0,    0,    0,    0]], dtype=uint16)}

Suggestions:

plot_doy.py

Check to make sure the result has nonzero length

timeseries_ccdc.py

We can actually fit a value for the timeseries example given above - just predict that one date. Problem comes from this line of code:

                    _mx = np.linspace(rec['t_start'],
                                      rec['t_end'],
                                      rec['t_end'] - rec['t_start'])

We need to check if t_end == t_start and specify "num" parameter as 1.

Add ability to include additional metadata for images in time series

• Metadata in the GDAL image domain (at dataset level)
• Add ability to attach metadata via CSV
  • Image ID as key in CSV

When time series data is asked to be fetched for a given pixel, do so in a separate thread so the main GUI window doesn't freeze up.

Reference:
http://svn.reprojected.com/qgisplugins/trunk/threading_demo/threading_demo.py

Select between the original data and residuals (observations - model prediction)

So since we don't have the usual documentation for this yet, yank out the help folder entirely. This isn't some kind of API so a Sphinx based documentation style that came with the Plugin Builder defaults isn't useful. Would prefer to just write documentation that is accessible via Github (gh-pages too?) as markdown, which gets built into HTML if anyone wants it within the deployment. So:

• Rip out existing help folder and fix Makefile
• Write documentation for how to use the plugin
• Write documentation for how to write a timeseries driver
• Re-create Makefile to use pandoc to go from markdown to HTML
• gh-pages site? https://help.github.com/categories/github-pages-basics/
• Link to repository and help pages from plugin

Allow user to click (with some x/y tolerance) a plotted value in the time series and add the corresponding image to the QGIS list of layers.

Reference:
http://matplotlib.org/users/event_handling.html

When you click a value on the plot:

1. If image hasn't been added, add the image
2. If else we have added the image
   • Turn the layer on if off
   • Make the layer the top active layer displayed

Make that checkbox work so that the observation causing a break is highlighted in some manner

Create timeseries driver for BFAST monitor.

Plan:

1. Create proof of concept dataset to validate algorithm implementation against
2. @ceholden: Implement Python script to run BFAST monitor in R from Python using rpy2
3. Figure out what parameters, settings, etc. we'll want to be able to change while running BFAST monitor
4. Use code from #2 to create a timeseries driver for running BFAST monitor
5. Decide what results beyond the model fits and break points should be plotted and how we should visualize them

Time series plot should have an option to automatically scale the Y axis according to the pixel max/min +/- some wiggle room for each band...

This option will disable manual configuration of min/max, but will update the QLineEdits to the appropriate value.

Refactor CCDCTimeSeries into more generic "LayerStackTimeSeries" which only does a "pass" to the results methods, or doesn't implement them at all (and thus wouldn't be an option in the QComboBox of drivers).

• AbstractTimeSeries
  • LayerStackTimeSeries
    • CCDCTimeSeries
    • YATSMTimeSeries
      • MODYATSMTimeSeries
  • TiledStackedTimeSeries
  • UnstackedTimeSeries (something like how LandTrendr datasets are stored)
    • LandTrendrTimeSeries

Culprits:

• str (why?!)
• configurable
• configurable__str
• custom_controls_title
• custom_controls
• metadata
• metadata__str

Simple fix for any contributed drivers, but should fix asap

Would be helpful to have more control over what images are used in timeseries

X axis: day of year
Y axis: SR
Color: all points from each year same color; years proceed along color ramp (e.g., red to blue)

Tabbed options:

1. All time segments
   2 - N: by individual time segments

See attachment:

E.g. self.symbology_hint = [3, 2, 1]

On init of timeseries, if hasattr(ts, symbology_hint), then settings.p_symbol will change accordingly.

Example:
I load an 8 band dataset and then change to one that has 11 bands. The plot band QComboBox still shows 8 bands available but is referencing the new dataset

TODO:
Tear down plot options before switching to new timeseries

Use Python datetime & matplotlib's time module to show the year on the x-axis of the plot.

Reference:
http://matplotlib.org/examples/api/date_demo.html
http://stackoverflow.com/questions/13965740/converting-matlabs-datenum-format-to-python

Maybe put this in the options tab? See #14 as a feature that may accompany this within a pop-up

Cache requested pixel time series objects...

Can we use functools.lru_cache?

Reference:
http://docs.python.org/3.4/library/functools.html

Ask TS for attributes to add to table. Examples:

• Cloud cover (ACCA, if available)
• Sensor
• etc

It is possible to deselect an active layer in QGIS and have it removed from self.mapCanvas().currentLayer(). If so, then we don't do a plot because the current layer is None (ccdctools.py line 106).

Suggestions:

1. If no layers are selected, work off a temporary QgsRasterLayer extent from the time series stack variable

Make this repository citable by adding a Digital Object Identifier (DOI)

https://guides.github.com/activities/citable-code/

Add some sort of mouse driven information to the time series plot i.e. a tooltip, side corner display, etc.

Reference:
http://stackoverflow.com/questions/4652439/is-there-a-matplotlib-equivalent-of-matlabs-datacursormode

Have in some QComboBox a list of a variety of different strftime formats with example.

The question is, where do we want to put this? Toolbar configuration option? Popup configuration at the top of this tab?

Include number of bands in unique filename:

n500_x1_y1.npy --> n500_b8_x1_y1.npy

Create new tab on controls that includes various utilities like a zoom to row/col

IOError on existing file

Path doesn't save if you cancel from file dialog

IndexError or TypeError if user has a file extension in specified file name

etc...

When dealing with large datasets over the network, it may take a while to fetch the data. Implement a fix such that the mouse click event is disabled or ignored until the data collecting method has returned.

Optional?: add a spinning wheel, notice, etc.

Create a set of API tests for classes that inherit from AbstractTimeSeries so that people can more quickly develop new drivers.

This should basically serve as a more exact set of documentation for what the class methods ought to return/yield/etc.

Restructure code according to guidelines in milestone 0.6

Line 623 in controller.plot_add_layer

AttributeErrpr: 'PathCollection' has no attribute 'get_data'

The "images" tab on the control panel widget should display the following for all images in time series:

cbox on/off | sensor type | date

where:

• "cbox on/off" pushes event to slot adding/removing image to/from layers
• "sensor type" lists LT4, LT5, LE7, etc.
• "date" shows image data in YEAR-DOY format

i.e.
[X] | LT5 | 2010-031

Reference:
http://stackoverflow.com/questions/12366521/pyqt-checkbox-in-qtablewidget

Qt & QGIS can help save plugin preferences, so we should take advantage of it. For example, we store the last opened image stack directory, last used plot and image display options, etc.

Reference:
http://www.qgis.org/pyqgis-cookbook/settings.html

It would be nice to be able to change the plot symbols to represent categorical information about each image in the time series. For example, you could use stars for LE7 imagery and squares for LT4/5.

Univariate categorical information

Add dialog box in the plot tab to set symbology. Parse available metadata for each image in time series and add them as radio buttons. When one is selected, parse all unique values into a table that allows user to select the symbol type (circle, dot, square, etc) and the symbol color via combobox.

Multivariate categorical information

Much harder to accomplish - parse metadata and also include the image bands. Allow user to add categories with symbol and color. Categories are defined via expression system that allows logical operators to work on the variables and bands (e.g., sensor type == LT5 AND B5 < 3000). Big TODO since user has so much freedom with this.

Currently, QTableWidget stretches last column only by taking advantage of the option horizontalHeaderStretchLastSection.

Write a subclass of QTableWidget or something similar to enable auto-stretching of all columns, not just the last.

Submit request for TSTools to be added to QGIS repo:

https://plugins.qgis.org/

Some requirements:

• license
• example dataset
• documentation
• wiki page!

Eli suggested applying the marker symbology choice to the DOY plot so we retain color information ~ year.

We should be able to plot multiple timeseries on the same graph and be able to select which Y axis (btw we also now have multiple Y axes) to plot the data.

E.g.,

Bands menu is a pulldown, but we use tri-state checkboxes to indicate OFF, left Y axis, or right Y axis

We can also put QCheckBox into QComboBox (SO reference]

Long overdue...

Icons for configuration and plot tool

Move MapTool clicker from widget checkbox to item in toolbar and add in a button to toolbar that brings up configuration menu.

Issue: Once custom symbology has been applied to time series visualization, switching to new scene causes an error that prevents the new scene data/time series from being loaded.
Need a way to clear previous symbology file (manually or automatically) so new scene/symbology can be applied.

Process that generates the error:
Load time series for first scene (e.g. p012r031)
Click to display time series for a pixel
Plot > Symboloy > Attach metadata > Browse to csv (e.g. /projectnb/landsat/projects/IDS/p012r031/p012r031_attributes.csv) > Load > Attach metadata
Change colors/marker shapes and Apply (Note: attribute field for visualization in these files is "Score")

At this point, everything has been working fine.

Now, when I try to switch to a new scene, the following error message is generated 3-10 times:

An error has occured while executing Python code:
Traceback (most recent call last):
File "/project/earth/packages/CCDCTools/tstools/controller.py", line 174, in update_display
self.ctrl.symbology_controls.parse_metadata_symbology()
File "/project/earth/packages/CCDCTools/tstools/controls_symbology.py", line 321, in parse_metadata_symbology
self.md[row - 1] = getattr(tsm.ts, tsm.ts.metadata[row - 1])
AttributeError: 'YATSM_LIVE' object has no attribute 'SCORE

Then another error:
An error has occured while executing Python code:
Traceback (most recent call last):
File "/project/earth/packages/CCDCTools/tstools/tstools.py", line 137, in config_accepted
custom_options)
File "/project/earth/packages/CCDCTools/tstools/controller.py", line 151, in get_time_series
self.ctrl.init_plot_options()
File "/project/earth/packages/CCDCTools/tstools/controls.py", line 181, in init_plot_options
self.symbology_controls.setup_gui()
File "/project/earth/packages/CCDCTools/tstools/controls_symbology.py", line 57, in setup_gui
self.setup_tables()
File "/project/earth/packages/CCDCTools/tstools/controls_symbology.py", line 82, in setup_tables
self.md = [getattr(tsm.ts, _md) for _md in md]
AttributeError: 'YATSM_LIVE' object has no attribute 'DATE'

Then, there is a red message in the load bar area that "Time series failed to load". I need to close and re-open so I can load and work with next scene--very frustrating!
I'm hoping that there is a way to clear the custom symbology from the first scene before loading the next, either automatically or through an option in the symbology window.

Please let me know if you have any questions or need more info to recreate this sequence of errors.
Val

If you initialize a new timeseries that has a different projection AFTER the "query" layer has already been created, the "query" layer will be moved but the projection is not updated.