pgbrodrick / bfg-nets Goto Github PK

In the reporter class and functions, we currently check for softmax output activations to determine whether we're working with continuous or categorical data. We can now just reference this directly in the config.raw_files section, from *_data_type attributes.

Alright, so use_multiprocessing is causing an issue:

keras-team/keras#11101

This might be fixed by switching from keras to tf.keras in tensorflow 2.0, so it's worth running a model and seeing if an error appears with use_multiprocessing set to True.

The commit in which use_multiprocessing was removed is:

e3ee428

and it should be trivial to add it back in with those changes highlighted.

Directly surface errors and warnings from the log, rather than expecting the user to review

✔️ combine input and output plots to avoid redundancy

• only show predictions when available, automatically

✔️ prediction plot labels run into one another excessively

• turn header labels 45 degrees

✔️ printing "sample x" in each row is excessive

• print once in header or once for central row or just rotate

✔️ weights coloring

• use white on outside, viridis on inside

✔️ history

• change histogram to line plot for cumulative epochs completed vs minutes elapsed

✔️ for categorical

• correct category plot needs colors lightened
• combine responses and predictions into single plots for each, respectively, warn if more than 20 classes!
• don't show transforms

README BUT ALSO add disclaimer that this is not being maintained for everyone to use, this is basically an internal tool that is a living codebase for our own purposes, we're making it available for others to kick around, be sure to use tags if you need certain functionality, etc

Note that pytest-cov is not playing well with tox right now using the following configuration option in tox.ini:

[pytest]
-addopts = --cov --no-cov-on-fail

Add flexible capacity at some point.

sample base stats on dataset in both performance and comparison reports
total area in original
total area in existing plots
category names
band names
response names
number of samples
fold splits and fold identity for verification

Per conversation:

• need better guidance on histogram plots
• legend
• constant y-axis across plots
• differentiate bins for zero from non-zero bins
• misc improvements as they come up

e.g., softmax for categorical and softplus for continuous

For loading/transforming data prior to fit/test. Been rolling our own but is it worth switching over? Additional functionality or performance?

Phil has code for this, might even be included already, but pinging just in case

Be better data scientists

Seems like data_management is for building data and handling samples, primarily?

If so, maybe an application module for taking a completed model and applying it to any input raster?

Confusion matrices give information on correct/incorrect predictions relative to the predicted and actual classes. We want to know about the spatial context in those predictions.

e.g., trying to predict land cover classes: are predictions for class a more likely to be correct when class a is found next to class b or class c?

@pgbrodrick has an idea of how to do this easily, so talk to him about implementation

https://github.com/semantic-release/semantic-release

Determine if we want to have a network-specific output directory, and what exactly we want to go in there.

Logging continues to go to the data log after build finishes, e.g., when modeling or reporting log messages are generated

This sounds easy, but we'd like this to be relatively comprehensive:

• update environment files
• update imports and API references
• in the process, ensure that the references to tensorflow and keras are compartmentalized such that we could easily switch to a new wrapper / backend if we wanted -- this is unlikely, but would be good coding practice and ensure that we're keeping our focus on what's important about our contribution... namely that we have automated tools for handling remote sensing data, building remote sensing -relevant models, and generating remote sensing -relevant reports... the actual network building is not a core contribution
• confirm things are working correctly

add in response_nodata_fraction and mask_nodata_fraction to mirror feature_nodata_fraction (right now feature_* is used for both response and mask).

Use case: user starts analysis pipeline for ten jobs with different model parameters but same raw file and data build configuration. Currently, all jobs will attempt to build the data and cause chaos in single directory. This can be handled manually by having the user start a single job, wait until data build is complete, and then start the remaining jobs. However, one solution could be to create a file lock in the data build directory, any jobs that find that file lock would not start data build, but would continually poll to determine when data build is complete. Apologies if terminology is off here.

Some validity checks were written already as a sample for functionality, but we need to flesh out the remainder.

Convert relative paths to absolute paths for when configs are loaded, so that models and histories (and other) can be found regardless of the Python working directory

Currently just raw spatial error values, if my quick reading is correct. Should be relatively easy to both show unweighted and weighted side-by-side in report.

Currently, the following doesn't pass our checks in training_data.check_projections:

Feature/Response projection mismatch at site 0
Feature proj: WGS_1984_UTM_Zone_10N
Response proj: WGS 84 / UTM zone 10N

Find a way to make sure that these different types of proj strings don't kill us, maybe straight to EPSG code.

i.e., the CPUs per job and not CPUs per node

IF SO: we need to add multiprocessing to keras model:

...
workers=psutil.cpu_count(logical=True),
use_multiprocessing=True,
...

Reporter only needs access to config to understand what the data_container and experiment objects have done, can reference those directly so that you're working with validated configs and making errors less likely

comparison report sample content
table with all models
print configs in table
highlight rows that are different
plus some other things
network summaries
layers
coefficients
time training
features
how many features
list of built data filepaths
samples
total samples
total training samples
total validation samples
image size
loss window
training time graph
include loss and validation loss as well

Use case: a model is being run and exits due to error or preemption, and we pick up where we left off with fitting. The history object is handled appropriately, but it would be wonderful if the tensorboard callback would simply append to the logdir so that it shows up as one run in the graphs, not multiple runs that overlap with one another.

Right now, the reporter class grabs a random set of samples from the built data. It would be really nice if we were able to grab samples proportionate to the observed features/responses, such that there's a variety of samples to review.

Responses as input vector files are currently assumed to be single, binary vector responses (E.G. each vector is its own response category). Consider enabling multi-response vectors.

Example:

  File "/home/nfabina/rsCNN/rsCNN/data_management/data_core.py", line 93, in build_or_load_rawfile_data
    self.config.raw_files.boundary_files)
  File "/home/nfabina/rsCNN/rsCNN/configuration/sections.py", line 517, in check_input_file_validity
    num_r_bands_per_file = [gdal.Open(x, gdal.GA_ReadOnly).RasterCount for x in r_file_list[0]]
  File "/home/nfabina/rsCNN/rsCNN/configuration/sections.py", line 517, in <listcomp>   
    num_r_bands_per_file = [gdal.Open(x, gdal.GA_ReadOnly).RasterCount for x in r_file_list[0]]
AttributeError: 'NoneType' object has no attribute 'RasterCount'

Code that caused this error:

apply_model_to_data.apply_model_to_raster(
    experiment.model, data_container, raster_for_application, basename_out)

apply_model_to_data.maximum_likelihood_classification(
    basename_out + '.tif', any_filepath_out)

Traceback:

Traceback (most recent call last):
  File "run_classification.py", line 106, in <module>
    run_classification(**args)
  File "run_classification.py", line 97, in run_classification
    filepath_out_base + 'apply.tif', filepath_out_base + 'class.tif')
  File "/home/nfabina/rsCNN/rsCNN/data_management/apply_model_to_data.py", line 182, in maximum_likelihood_classification
    prob = dataset.ReadAsArray(0, line, dataset.RasterXSize, 1)
  File "/home/nfabina/miniconda3/envs/asu/lib/python3.7/site-packages/osgeo/gdal.py", line 2089, in ReadAsArray
    callback_data = callback_data )
  File "/home/nfabina/miniconda3/envs/asu/lib/python3.7/site-packages/osgeo/gdal_array.py", line 304, in DatasetReadAsArray
    buf_obj, buf_type, resample_alg, callback, callback_data ) != 0:
  File "/home/nfabina/miniconda3/envs/asu/lib/python3.7/site-packages/osgeo/gdal_array.py", line 147, in DatasetIONumPy
    return _gdal_array.DatasetIONumPy(ds, bWrite, xoff, yoff, xsize, ysize, psArray, buf_type, resample_alg, callback, callback_data)
TypeError: in method 'DatasetIONumPy', argument 4 of type 'int'

Scaling is currently static, set it to link to the specified band types through data core and possibly be a list of multiple scalings.

Increase documentation around boundaries and boundary bad values, and auto-detect shapefiles and force the boundary_bad_value to be 0.

Please update the documentation so that it explains how to install the package via pip!