dhardestylewis / dem2basin Goto Github PK

Consider the following back-ends for performance:

In time-expensive functions (see pycallgraph.png), consider replacing Shapely-dependent objects with GDAL/OGR objects
This would also mean using either:

• NetCDF, or
• PostgreSQL

for I/O

Restore functionality for using dem2basin as a script

• File name of DEM as found on Stampede2
• Actual file type of each DEM (.dem,.img,.tif)
• CRS
• Geographic CRS
• EPSG
• Bounding box
• Accuracy calculations for reprojections from source CRS to NAD83(2011) / UTM zone 14N & NAD83(2011) Texas Centric Albers Equal Area
• Polygonizing the rasters
• File size

Catch catchments not assigned to any HUC12 by algorithm. An example of such missing catchments is present in this screenshot

Include timing wrapper function:

def timethis(f):
    start_time = time.time()
    ret = f
    end_time = time.time()
    elapsed_time = end_time - start_time
    return((ret,elapsed_time))

Current approach: Merging TNRIS Lidar data modelled GDFs and mosaicking on highest resolution

Check area calculation difference before and after reprojection to an equal area CRS

Prioritize by functions appearing in Jupyter notebooks

Add docstrings and other documentation for functions grad students are using

Check to make sure prepare_and_output_geoflood_gis_inputs_by_huc works by default against National WBD

Run pycallgraph against prepare_and_output_geoflood with/with-out new coverage file to determine why the new coverage file is not yet being created

Break up code to speed up each individual import

Adopt the following organization:

• General GeoPandas extensions
• Pandas extensions
• Function wrappers
• Misc extensions of other Python libraries
• Hydro vector processing functions
• Hydro raster processing functions

Verify geometric association algorithm at scale across Texas

• later consider verifying nationally

Recheck file extensions in TNRIS Lidar database

Known extensions:

• .tif
• .dem
• .img
• .jp2

Check under both */dem/* and */tiles/*

Significantly revise get_bounding_boxes_by_project to center around project-level GDF objects

As soon as possible, revise the following code to operate on project_coverage GeoDataFrames, with a lambda function to find the files.

Also, dynamically compile a list of separate GeoDataFrames for each project, each row within these GDFs would represent each different DEM tile within that project's GDF, use geopandas.GeoDataFrame.to_crs to reproject these all to the same CRS, before concatenating them to a single master coverage file, which is the final output

    dem_tilenames = []
    filetypes = ('.img', '.dem', '.tif', '.jp2')
    for filetype in filetypes:
#        print(filetype)
        for project in projects:
#            print(project)
            for root, dirs, filenames in os.walk(str(project)):
#                print(root,dirs,filenames)
                for filename in filenames:
#                    print(filename)
                    if filename.endswith(filetype):
#                        print(True)
                        dem_tilenames.append(os.path.join(root,filename))

#    print(dem_tilenames)

    dem_tile_bounds = []
    for dem_tilename in dem_tilenames:
        dem_tile_bounds.append((
            dem_tilename,
            rasterio.open(dem_tilename).bounds
        ))

#    print(dem_tile_bounds)

    dem_tiles = gpd.GeoDataFrame(
        dem_tile_bounds,
        columns = ['lidar_file','bounds']
    )

#    print(dem_tiles)

Make Python libraries' developers aware of developed extensions

Such libraries include:

• Geopandas
• Pandas
• GDAL/OGR
• Fiona
• Shutil
• etc

Validate general mosaicking algo from each pair of 2 adjacent TNRIS Lidar projects

Slightly modify _get_mosaic_and_output_raster_simple for more general use-case

• make dependence on having a coverage file optional
• include option to explicitly specify a mask object (whether a polygon or a file)

Consider adding support for point-cloud to DEMs, but:

• record timings
• compare against DEM reprojection and mosaicking technique
• estimate large-scale node-hours usages

Set up Pegasus workflow for GeoFlood & Cahaba

Add function to fill in 0-value in mosaicked rasters for the coast

• double-check the datum

Revise documentation & Jupyter notebook to reflect new directory structure

Troubleshoot minor GDAL issues that arise during mosaicking at scale

Manually re-write 2 remaining missing dirnames in coverage file

For remaining 2 mis-written dirnames in Lidar availability file, consider manually rewriting just these 2

Run regular scrape of latest Lidar data using tools developed in tnris-lidardata-update

Upload latest version to PyPi successfully

Build basic functions to convert these datasets to existing TNRIS Lidar availability data model

Revise README.md for use as a library of functions

Determine accuracy of general reprojection & mosaicking workflow

Especially whether inaccuracies introduced through so many reprojections are significant enough to matter

Include rough measures of completion of each aspect, basic features, links to various repositories, etc

Check for bugs associated with not using select_utm option in various functions

Consider using crosswalk table to associate NHD MR and WBD:
https://www.sciencebase.gov/catalog/item/5c86a747e4b09388244b3da1

Gather boundary data for TNRIS Lidar point-clouds

potentially could use some of the same methodologies developed in DEM2basin for DEMs (just a derivation of those functions)

Develop coverage file from NOAA digital coast elevation meta-information at Louisiana/Texas border:

https://coast.noaa.gov/dataviewer/#/lidar/search/

Basic use-cases include:

• GeoFlood workflow preparation
• Non-HUC-oriented workflow preparation
• HEC-RAS preparation excluding road network / bridge segment aspects
• 10m
• 3m
• 1m
• Backfill 1m with 3m or 10m
• QA/QC existing DEMs for common issues including nodata gaps, missing source data, and wrong/bad projections
• Determine accuracy of resulting products

Provide HUC-by-HUC selection of TNRIS Lidar tiles, in a similar manner to the following web-portal:

https://www.coast.noaa.gov/inventory/

Get GeoFlood Fathom3m timings back to Mark Wang & Paola Passalacqua

Time large-scale use of pathlib vs os.path

Time for UnparsedPath in the following callgraph of DEM2basin indicates that the pathlib library may be too heavy to use at scale:
https://github.com/Khushbooagarwalhub/DEM2basin/blob/dev-kag/references/pycallgraph.png