aiddata / monte-carlo-rasters Goto Github PK

System for dynamically creating jobs for processing

Includes:

• estimating walltimes
• estimating number of iterations needed
• generating filter settings for various data subsets

• possibly use spatial indexes (rtree) to speed up finding which adm to assign to points
• may not be necessary since init is under 2 minutes for nepal dataset but other datasets may have more locations being aggregated to adm levels (vs points, buffers, country)

Update how mean surf data is loaded and saved so that it requires minimal options to load existing mean surf for current data/script version and pixel size or save new mean surf for future use

Requires to feature changes/additions

1- Slightly rework of how results are output

• all options are stored in object which is converted to JSON
• JSON file from each run is stored in a central directory

2- External process (most likely on front end server) adds each JSON file as document to a mongo database

• runs at set intervals
• moves JSON files it processes to backup location

Integrate modeling and uncertainty analysis into jobs

Add robust support for filtering datasets

Run requests are submitted to central directory as JSON files which holds detailed options (filters, etc.)

bash script scans queue directory, creates jobscripts and submits them

Check error percentage at intervals to determine if error threshold has been met

Reproject geometries to calculate proper buffer sizes. Reproject back at end.

Add better error handling between master and worker processes.