ssec / sift Goto Github PK

In GitLab by @rayg-ssec on Oct 19, 2015, 13:04

In GitLab by @djhoese on Oct 14, 2015, 17:35

Right now the animation loops without giving any indication (except for the slider on the bottom) of what frame it is currently on. It would be best for users if the time of the current layer being displayed is shown somewhere.

In GitLab by @djhoese on Sep 30, 2015, 11:03

The pieces aren't connected in the way they will be, but my current testing version does not update tile data while an animation is happening. So this is less of a current bug and more of a "make sure this works when things are more finalized".

When opening products SATPY readers should be used. Satpy is well supported and and have a number of product readers so should be used to open files and have SIFT should have no independent product readers.

In GitLab by @rayg-ssec on Oct 20, 2015, 11:33

Animation controls state should always represent what the user is seeing, even if they've taken a different route to displaying (e.g. toggling visibility).
Layer list can be locally inconsistent while animation is playing, but should settle to be representative when animation stops.
feature-animation-consistency branch

In GitLab by @rayg-ssec on Sep 6, 2015, 15:40

Right now we're generating too many tiles with small textures when displaying large areas. This is a performance hit, especially with larger numbers of layers.

common.MercatorTileCalc is used by tile image LayerReps to handle this currently. It provides the basic calculations. Consider turning this into a TileGeometryManager and make it more stateful? Still maintain the possibility of using it across different tile types using composition instead of inheritance.

• optimize so that we're creating fewer tiles with 512x512 textures of strided science data, rather than lots of tiles of 256x256, 128x128, etc.
• only generate tiles we know we need, plus extras around the side
• tile identifiers should be more than just (y,x) index pairs, they should include level of detail / stride
• eventually: LayerReps should have a setting for how many pixels of data they're allowed in the GPU, and LRU-cache flush tiles that are not needed rather than generate/flush/generate.

Data probing a pixel returns inaccurate values. Example given in the attached images.

capture 1: correct probing of the light grey pixel with the value of 0.279
capture 2: probing the pixel to the left still returns the same value of 0.279
capture 3: probing the pixel to upper left again returns the same value of 0.279
capture 4: correct probing of upper left pixel with the value of 0.082

In GitLab by @djhoese on Oct 14, 2015, 17:34

Right now the UI has 3 buttons to choose between Pan/Zoom, Point Probe, and Polygon/Region Probe functionality. Under the hood this is switching vispy cameras to handle mouse events. At the recent meeting the point probe should also be changed to show a marker where it is recording values. At the meeting we also discussed additional probe features. It would be nice to have the Point/Marker probe and the Region probe, but having a "hover" probe might not be the biggest priority as far as probing goes. There is also the idea of probe A vs probe B that needs to be figured out on the UI side, thoughts @rayg-ssec?

I think this could be done by having one camera based off of the vispy pan zoom camera that only uses the left mouse button (right now vispy uses both left and right). The mouse events can then be passed on to event callbacks that are part of the scene graph itself or wherever else they seem to fit best.

Experienced/Power users will use the mouse buttons to enables Image probing or selecting a region of interest so there is no need for these buttons to use up the screen real estate.
David did mention that new/inexperienced users may wish to retain these buttons so maybe we can enable/disable them via a config file

In GitLab by @rayg-ssec on Oct 1, 2015, 21:50

• Create a button subclass that shows a color map, with the color map name overlaid
• Make a list or other selector which allows color map to be changed
• add color map indicator to document layer list window
• bring up color map change utility

There should only be a need for the Profile window to appear , and levels to select, for products that contain this type of data.

Maybe some meta data withing the scene could be useful here

In GitLab by @rayg-ssec on Sep 2, 2015, 12:31

We need to load a coarse overview tile immediately and then progressively bring in data at resolution closely matching the screen. Demonstrate how this is done. The application needs a Task object class with a pool and a queue (like Activity Monitor in Mail.app, or Tasks in Aperture, or...). User must always know what the app is up to and when things will be ready if they're not instantaneous.

In GitLab by @rayg-ssec on Oct 19, 2015, 12:59

User should be allowed to drag & drop files onto the map window, or the layer list.

In GitLab by @rayg-ssec on Oct 16, 2015, 14:02

https://en.wikibooks.org/wiki/GLSL_Programming/Unity/Order-Independent_Transparency
If the depth test is turned off in the GL context it appears that additive blending is legal and productive, i.e. gl_FragColor = gl_FragColor + _____ .

We want to demonstrate RGB composites by setting a R, G, or B colormap on three layers, and toggling the blending mode from "normal" to "add". This would ultimately be done by the user in the layer dialog in a fashion similar to Photoshop.

In GitLab by @rayg-ssec on Sep 6, 2015, 11:59

Layer widget is currently a simple list view. This needs to include

• visibility toggle (visibility when not animating)
• animation order (1..n, what order it's cycled when animating)
• color map summary, if it has one
• selection (selected / deselected)

An intuitive way of marking the animation order is also needed. For instance, option click the layers you want to see animated in the order you want to see them. We'll doubtless revisit after user feedback. Think of animation as an alternate form of visibility for now.

Layer widget will be connected to a document model facet; the document will then present it to the MapWidget as a LayerDrawingPlan.

Starting with imageio 2.5.0 there is a separate dependency called imageio-ffmpeg that needs to be installed. See imageio/imageio-ffmpeg#9 and conda-forge/staged-recipes#7730.

We'll need to add this package to the meta.yaml for sift.

In GitLab by @rayg-ssec on Oct 20, 2015, 11:34

feature-document-io should include save and open of documents as
Documents should reference the files that were loaded, and prompt the user to locate datasets that have changed locations. Those user-entered locations then become part of a local data search path used for document opens during that session.

In GitLab by @rayg-ssec on Oct 8, 2015, 12:23

We're getting an unclean exit in Windows - this could be thread cleanup, application state cleanup, or even just a simple result code.

In GitLab by @djhoese on Oct 14, 2015, 17:41

Add a longitude and latitude value somewhere on the UI next to the probe value.

When i open an MSG file , eg, i would like to see all of the RGB's available for that Product and be able to select/visualize this RGB.

In GitLab by @rayg-ssec on Sep 6, 2015, 12:03

An animation self-test can be run with
TIFF_GLOB='/path/to/2015_07_14_195/0?00/HS*B03*merc.tif' VERBOSITY=3 python -m cspov

Toggle animation with 'a' key. It currently just loops through the loaded layers at 10fps.

Animation rate should be controllable
Animation frame number should be shown
User should be able to scrub through animation frames with a control while animating is on

Right now mouse activity goes through Animating activity class in MapWidget when animation is turned on. Panning and zooming during animation should be permitted, so this activity class may either not be needed, or it may imitate Idling in terms of pushing Pan/Zoom activities onto the activity stack.

In GitLab by @rayg-ssec on Sep 6, 2015, 12:05

A guidebook (eventually backed by a config file) should match file identifying marks to things like default color bars, ranges, enhancements, when creating new layers in the document. This should be propagated into configuring the LayerRep used, including tile_class.

In GitLab by @djhoese on Oct 14, 2015, 17:36

Right now next frame animation is attached to the 'n' key. A previous frame method should be added and connected to the left arrow key. The next frame should be connected to the right arrow key.

It would be useful to have a way to get/view some basic statistics (min, max, mean, stddev, ...) of the loaded layers. Full disk would be the default, but if the user has selected a ROI, the statistics would apply to the selected region.

I have no strong opinion on how to represent and show this in the GUI, but it could be, for example, a new tab next to "Layer Details".

In GitLab by @rayg-ssec on Oct 1, 2015, 21:46

We need scene graph to support area selection. For now, 1 area, eventually 2 or more.

Scene graph / SceneGraphManager responsibility:

• User presses selection tool button or command (command goes to SGM to switch from pan/zoom tool to select tool)
• User clicks and drags around an area of the map window
• Rubber-band area is created and shown on the map by SGM
• When user lets up on button, signal goes out
• Signal contains X/Y coordinates (meters) of the selected area in standard counterclockwise direction, later this will be smoothed and shown as smoothed on the display
• Signal is re-emitted when the selection area is re-done (eventually when it's changed)

Second half

Probe responsibilities, given signal of updated selection area coordinate list (polygon)

• looks up which dataset is selected in the document
• converts X/Y coordinates of selection polygon into lat/lon if needed
• generates localized mask against highest LOD version of selected dataset
• performs an operation on the masked area of the dataset, e.g. scatter plot
• sends output to probe pane in the GUI

In GitLab by @rayg-ssec on Sep 2, 2015, 12:33

• convert lat/lon map boundary via PROJ4 into a projected coordinate VBO backed by a gloo.Program, such that political and geographical boundaries can be plotted with selectable line color/weight/alpha, as a Layer subclass e.g. PoliticalBoundariesVectorLayer. Use data from Matplotlib or elsewhere.

It would be useful to have a config file so that the user can have an environment suitable for their needs eg The list of available readers could be limited to eumetsat files only.

Also the screen display could be set for new/power users as mentioned in Issue #5

In GitLab by @djhoese on Oct 19, 2015, 11:32

The camera should be restricted to a reasonable zoom level and shouldn't be able to pan much further away than the map (maybe go as far as requiring that at least 1 tile is visible).

In GitLab by @djhoese on Oct 19, 2015, 13:41

I tested Band 16 from 20150714_0000 and it shows an almost all white higher level image for an overview (it also has some saturation), but the data is mostly a light gray.

In GitLab by @rayg-ssec on Oct 19, 2015, 13:08

Request from Tim Schmit:
GFS or equivalent forecast wind overlay layer.

In GitLab by @rayg-ssec on Sep 6, 2015, 12:31

Right now Program.py only includes two color maps - grey and hot. The current setup allows for up to 16 color maps to be loaded into the GPU and easily selected among.

Eva and Nick have color maps for a variety of purposes that could be included in the default set. Adding those to Program.py (as hardcoded data in the short term, as data files in the longer term?) would be useful until we have a colorbar editor.

Change the screen cursor for Probe or Region selector to a cross hair as its currently a large disc, making accurate selection or probing difficult.

Add "Fused" prefix (or similar) to ABI L1B data that comes from the ABI fusion products; bands created from other bands when GOES-17 is experiencing instrument heating issues.

The products, so far, are just like the original L1B files but start with "FR" in the filename and have a global attribute called "fusion_args":

A new attribute, "fusion_args", contains the fusion parameters:

    :fusion_args = "2019_02_05_036 0900 0915 13" ;

This is probably best implemented as a new metadata magic handling when producing the display name. Something like "name_prefix" that gets set to "Fused" and if present gets prepended to the DISPLAY_NAME.

In GitLab by @djhoese on Oct 16, 2015, 14:10

We need to gather and process the data that will be used during the training. Below are copies of the emails along with a checklist to keep track.

Checklist

• Jordan Gerth
  • Japanese Volcan Eruption
    • Every 10 minutes between 2015/05/29 00:30 UTC and 04:00 UTC (22 frames) (10N to 40N and 120E to 150E)
  • Twin typhoons with Guam/CNMI
    • Every hour between 2015/08/17 12:00 UTC and 2015/08/18 12:00 UTC (25 frames)
    • Every 10 minutes between 2015/08/17 22:00 UTC and 2015/08/18 01:00 UTC (19 frames)
  • Extratropical transition east of Japan
    • Every hour between 2015/08/24 15:00 UTC and 2015/08/25 21:00 UTC (31 frames)
    • Every 10 minutes 2015/08/25 02:00 UTC and 05:00 (19 frames)
• Tim Schmit
  • Every 10 minutes between 2015/09/19 18:00 UTC and 2015/09/20 18:00 UTC
• Kathy Strabala
  • Every hour between 2015/07/17 21:00 UTC and 2015/07/18 20:00 UTC
  • Every 10 minutes between 2015/07/18 01:10 UTC and 2015/07/18 03:20 UTC
  • Every 10 minutes between 2015/07/18 14:00 UTC and 2015/07/18 16:00 UTC
• Scott Lindstrom
  • Every 30 minutes between 2015/10/07 00:00 UTC and 2015/10/08 00:00 UTC

Jordan Gerth

I have come up with my favorite cases. This should be 110 frames total. Ideally each case would contain all bands when available, so that is 1760 different files. I am a bit concerned that this is going to exceed the disk space on the workstations even before Kathy submits her cases.

If that is the situation, my priority would be to include high-resolution visible band for the second case only. If we can subset the data, I would clear off everything south of 30 S, north of 60 N, and west of 100 E.

Japanese volcano eruption: If we have the data (this was pre-operations), every ten minutes between 00:30 UTC on 29 May 2015 and 04:00 UTC on 29 May 2015 (22 frames). For this case, we really only need between 10 N and 40 N, and 120 E and 150 E.

Twin typhoons with Guam/CNMI in the middle: Every hour between 12 UTC on 17 August 2015 and 12 UTC on 18 August 2015 (25 frames), and every ten minutes between 22 UTC on 17 August 2015 and 01 UTC on 18 August 2015 (19 frames).

Extratropical transition east of Japan: Every hour between 15 UTC on 24 August 2015 and 21 UTC on 25 August 2015 (31 frames), and every ten minutes between 02 UTC on 25 August 2015 and 05 UTC on 25 August 2015 (19 frames).

Tim Schmit

Can you also load AHI from September 19 to September 20? (2015262 - 2015263)

If you have room, 18 UTC to 18 UTC. If not, 18 UTC to 09 UTC.

Kathy Strabala

I have a request for one day of data, at a rate of hourly all bands going from 21 UTC 17 July 2015 to 20 UTC 18 July 2015.

I would also like to have a subset of times where the data has full temporal resolution (still all bands) between 01:10 and 03:20 UTC 18 July 2015 (daytime) and 14:00-16:00 UTC 18 July 2015 (nighttime).

Scott Lindstrom

I'd also like one day of data: 00z 7 October 2015 to 00z 8 October 2015.

All bands, half-hourly.

@sjoro and I have been talking about this on the PyTroll slack. The idea is that it would be nice for certain settings to be configurable by the user and stored in a configuration file. This is extremely important for easy collaboration between SSEC and EUMETSAT on SIFT. We could have one profile as the default and another that defaults to settings preferred by the eumetsat crowd. This could be expanded to a "training" profile where certain things are disabled or simplified for forecaster training sessions.

I've thought about this a little and here is how I see this going:

1. Add a command line flag --profile that accepts either the name of a profile or a path to an existing .yaml file.
2. Use the donfig package (see documentation here) to create a config object in sift/__init__.py by doing config = donfig.Config('sift').
3. When profile is specified in the command line argument then we can do config.update_defaults(yaml_file_content) to update the configuration values used throughout SIFT.
4. In the future, we would add a File -> Preferences or something similar that brings up a UI window. This window would have a profile management section (similar to PyCharm probably) where you can choose a different profile, remove existing ones, import profile from an outside file, or create a new one based on the currently selected one. We can probably take advantage of donfig's ensure_file functionality to copy imported yaml files to the default configuration location.

In GitLab by @djhoese on Oct 8, 2015, 08:47

Problem

Currently we are testing CSPOV with mercator images that cover the whole earth from -180 to 180 longitude. We also display this image twice for a total "earth view" of -180 to 540 degrees longitude to create one complete and continuous image. These input images are of AHI data that mostly covers the Pacific ocean on top of the antimeridian (-180/180). For this data having an image that shows the entire earth is a waste of disk space, processing time, and memory since a large portion of the image is just black (empty, 0).

To accept projected mercator images of just the data requires some care as some/most of our calculations are dependent on the idea that the image covers the entire earth. The main concern is the MercatorTileCalc.visible_tiles method (originally written by @rayg-ssec, modified by @djhoese). This method returns a box object defining the tile index at the bottom, top, left, and right of the viewable area and can be iterated over with a simple 2-level for loop to get the tiles to be shown. With an image that is non-global most of this math will work, but get's a little questionable when considering the "wrapping" around the 180 longitude antimeridian.

Right now to handle wrapping the tile numbers keep increasing and are then modulo'd when used to get the actual image tile. For example, if the original -180/180 image is 10 tiles wide but the current view needs to see past 180 then the produced tile box will contain tiles with an X index of 11, 12, and so on. When these numbers are used to fill the GL Texture the index modded with the total number of tiles, so 11 % 10 => 1. This makes sense for global images that are continuous across the antimeridian boundary, but for non-global images there is an empty space so producing a "tile box" to iterate over is not sufficient. For example something at longitude 0 that only uses 1 tile being seen on both maps would currently result in a tile index of 32 (or some high number), but indexes 2-31 don't actually mean anything on the image.

Questions

1. Should the visible_tiles method return a list of image tile coordinates instead of a tile box?
2. If the user is viewing the entire -180 to 540 space do they see one or two repeats of the original image? This might seem weird if zooming out and suddenly the thing you were looking at disappears.
3. Does the current math done in visible_tiles handle these cases and I'm just overthinking it?
4. Is using modded tile indexes the best way to define "tile locations"? If not, what's another solution?

In GitLab by @rayg-ssec on Oct 19, 2015, 12:58

Open file dialog currently doesn't remember the last place you opened files from.

In GitLab by @djhoese on Sep 25, 2015, 13:23

An overview is a very low resolution copy of the image being displayed that fits in to one tile. This image should always be shown when the image layer as a whole is visible. This way fast panning and zooming shows something to the user.

In GitLab by @djhoese on Oct 8, 2015, 17:23

In combination with #18 we should create our own channel for Anaconda to control the versions of CSPOV and its dependencies that are made available to users. The common way to do this use to be binstar, but that has been absorbed in to anaconda.org. I've found these instructions (link below), but Coda Phillips has experience creating his own custom channel on larch.ssec.wisc.edu.

http://conda.pydata.org/docs/custom-channels.html

In GitLab by @djhoese on Sep 25, 2015, 12:25

Users should be able to view an entire data across the globe without having to deal with cutoff lines at the anti-meridian (-180/180). This is especially important for V1.0 since Guam users will be looking at Pacific data the most.

In GitLab by @djhoese on Oct 14, 2015, 17:23

A new layer should be added by default to the scene graph (or the document adding it by default). It should use a line visual similar to the borders/shapefile layer.

For an initial version this could/should just be a static resolution (5-10 degrees). If possible (or if time permits) try to make the resolution vary based on Z level. Another optional feature would be to have actual degree numbers next to the lines (this will be very difficult for a good user experience).

In GitLab by @rayg-ssec on Oct 20, 2015, 11:31

Anecdotally: Also it's really slow (seconds) on Windows while it's instant on Mac. Suspect Windows box was doing something else at the same time?

In GitLab by @rayg-ssec on Oct 20, 2015, 11:34

In the satpy reader we have set some values to nan to remove them from the final image.

Here we can see some areas , that contain nans , display heavily pixelated data on the screen. In particular look at the 2 versions of Madagascar.
Probing these pixels shows nan as the data values.
Only nans show this behaviour,
Viewing the data with satpy also doesnt exhibit this behaviour.

In GitLab by @rayg-ssec on Oct 19, 2015, 13:09

Request from TimS, radar imagery loading. Possibly as geotiff with proper color map??

In GitLab by @djhoese on Oct 19, 2015, 20:03

Jordan gave us the proper acceptable limits of the bands. The "clim" in the dataset info should have this data.

In GitLab by @rayg-ssec on Sep 6, 2015, 12:08

A mouse probe should recover the value of current selected layer under the cursor initially; where transparency is involved we ultimately want to "see through" transparency to the topmost layer being displayed. The data value should be put onto a widget, with document information on units (which probably comes from the guidebook).

This should be active during Idling activity.

In GitLab by @rayg-ssec on Oct 1, 2015, 21:53

First order implementation

• User selects three layers
• User brings up combinations / enhancements panel
• User selects RGB enhancement from available enhancements
• User assigns each channel to R,G,B with UI controls
• SGM is told to replace image layers with a single image layer combining three datasets

In GitLab by @djhoese on Oct 8, 2015, 17:18

It seems that we can create a script/executable that automatically installs and creates an anaconda environment to run CSPOV. I have not tested this, but have found this:

http://docs.continuum.io/anaconda-server/command-line/cas-installer#cas-installer

In GitLab by @djhoese on Oct 16, 2015, 12:35

Right now if a layer is added while we are zoomed in it does not schedule a retile. You can see this by zooming in and adding a layer. The image will be at the overview resolution until you pan or zoom.