deltares / delft-fiat Goto Github PK

Delft-FIAT version checks

• I have checked that this issue has not already been reported.

• I have checked that this bug exists on the latest version of Delft-FIAT.

• I have checked that this bug exists on the main branch of Delft-FIAT.

Reproducible Example

No response

Issue Description

No response

Installed Versions

No response

Resolve Description

No response

Additional Context

No response

Feature Type

• Adding new functionality

• Improving existing code

• Removing deprecated code/ clutter

Improvement Description

Take the code from the separate scripts and turn them into tests.

Implementation Description

No response

Additional Context

No response

Feature Type

• Adding new functionality

• Improving existing code

• Removing deprecated code/ clutter

Improvement Description

No response

Implementation Description

No response

Additional Context

No response

Feature type

Adding new functionality

Improvement Description

Right now FIAT will run, even if inputs are missing for some objects (e.g., depth damage function).

It is desirable to provide some warning in case inputs is missing.

Implementation Description

• Save object IDs of object that miss inputs in a text file.
• Provide a warning in the logger that points to that file.

Additional Context

No response

Delft-FIAT version checks

• I have checked that the issue still exists on the latest versions of the docs on master here

Location of the documentation

No response

Documentation problem

No response

Suggested fix for documentation

No response

Delft-FIAT version checks

• I have checked that the issue still exists on the latest versions of the docs on master here

Location of the documentation

No response

Documentation problem

No response

Suggested fix for documentation

• @B-Dalmijn sets up website
• Kathryn fills the general texts
• @B-Dalmijn creates a few examples
• @B-Dalmijn writes setup guide, installation
• @srpeiter writes about collaboration

Delft-FIAT version checks

• I have checked that the issue still exists on the latest versions of the docs on master here

Kind of issue

None

Location of the documentation

No response

Documentation problem

No response

Suggested fix for documentation

No response

Delft-FIAT version checks

• I have checked that the issue still exists on the latest versions of the docs on master here

Kind of issue

Docs are missing

Location of the documentation

Not yet online.

Documentation problem

Needs to be written.

Suggested fix for documentation

Create a structure for the webpages. Fill them out.

Fill out the docstrings (used for api reference of the website)

Feature Type

Make sure everything is synced.

Improvement Description

[global]
crs = "EPSG:4326" [OPTIONAL] # CRS that fiat can work with
gdal_cache = 10 #MB [OPTIONAL] # Set the GDAL cache in megabytes to restrict the footprint of GDAL
keep_temp_files = false [OPTIONAL] # whether or not to keep the temporary files spit out during calculation
thread = 4 [OPTIONAL] # Set the number of threads of the calucations

[global.grid]
chunk = [1024, 1014] [OPTIONAL] # Set the size at which to chunk grids

[output]
path = "output" # directory to save the results in, directory is relative to the settings.toml file or absolute

[output.csv]
name = "output.csv" # File name of the output csv

[output.geom]
name1 = "spatial.gpkg" # File name of the output GPKG
crs = "EPSG:4326" [OPTIONAL] # Projection for output data, DOESNT WORK YET

[output.grid]
name = "output.nc" # File name for the output for the grid model; WARN: not fully implemented yet

[hazard]
file = 'hazard/event_map.nc' # Path to netCDF relative to the setting.toml file or absolute, can contain one (for events) or multiple (for return periods) bands
crs = 'EPSG:32617' # Not compulsory but if the CRS is unknown in the data this is compulsory (to check)
elevation_reference = 'dem' # DEM or Datum for a water depth or water elevation map
risk = true # true or false, if true, risk (EAD) is calculated
return_periods = [2, 5, 10, 25] [OPTIONAL] # return periods for hazard calculation when they cannot be inferred from the data itself

[hazard.settings]
chunk = [1024, 1014] [OPTIONAL] # Set the size at which to chunk (Advise to use the setting under global.grid)
subset = 'zs' [OPTIONAL] # Specific layer from a multilayered netCDF file (GDAL cant read those properly)
var_as_band = false [OPTIONAL] # true or false # Read netCDF subdatasets as raster bands

[exposure.csv]
file = 'exposure/exposure.csv' # All additional info for the vector data
index = "Object ID" [OPTIONAL] specify the index column of the data

[exposure.geom]
crs = 'EPSG:32617' # Not compulsory but if the CRS is unknown in the data this is compulsory (to check)
file1 = 'exposure/points.gpkg' # Kijkt vanuit het 1e veld van de gpkg naar welke Object ID de geometrie linkt.

[exposure.grid]
crs = 'EPSG:32617'
file = 'exposure/raster.nc'

[exposure.grid.settings]
chunk = [1024, 1014] [OPTIONAL] # Set the size at which to chunk (Advise to use the setting under global.grid)
var_as_band = true [OPTIONAL] # true or false # Read netCDF subdatasets as raster bands

[vulnerability]
file = 'vulnerability/vulnerability_curves.csv'
step_size = 0.01 [OPTIONAL] # Interpolation scale for the vulnerability curves

Implementation Description

No response

Additional Context

No response

Feature type

Adding new functionality

Improvement Description

Calculate the risk in regards to hazards for certain return periods.

Implementation Description

While the concurrency part is there, the risk calculation is not.
Write the logic of the risk calculation as a function and implement that in the _patch_up function of the GeomModel.

Additional Context

No response

Feature type

Adding new functionality

Improvement Description

We need to brainstorm about somehow summing up and writing out the total damage of vector and raster data.

Implementation Description

This still needs to be designed.

Additional Context

No response

• FIATBase?
• Vulnerability
• HazardBase

Lazy reading

• Hazard
• Exposure

Feature type

Adding new functionality

Improvement Description

Write function to check e.g. alignment of data, mandatory columns etc.

Implementation Description

No response

Additional Context

Needs refinement.

Feature Type

• Adding new functionality

• Improving existing code

• Removing deprecated code/ clutter

Improvement Description

Brendan checkt de logic branch

Implementation Description

No response

Additional Context

No response

• Exposure
• Hazard

Feature Type

• Adding new functionality

• Improving existing code

• Removing deprecated code/ clutter

Improvement Description

No response

Implementation Description

No response

Additional Context

No response

Delft-FIAT version checks

• I have checked that this issue has not already been reported.

• I have checked that this bug exists on the latest version of Delft-FIAT.

Reproducible Example

No response

Current behaviour

No response

Desired behaviour

No response

Additional Context

No response

Delft-FIAT version checks

• I have checked that this issue has not already been reported.

• I have checked that this bug exists on the latest version of Delft-FIAT.

Reproducible Example

Any run where "datum" is used as reference in the hazard.

Current behaviour

, the ground elevation of the object is not provided, so a default of zero is used.

Desired behaviour

The actual ground elevation of the object should be used before calculating the indundation depth.

Additional Context

No response

Feature Type

• Adding new functionality

• Improving existing code

• Removing deprecated code/ clutter

Improvement Description

No response

Implementation Description

No response

Additional Context

No response

Delft-FIAT version checks

• I have checked that this issue has not already been reported.

• I have checked that this bug exists on the latest version of Delft-FIAT.

Reproducible Example

If a water level maps is used, where negative values exist, they will be assumed as zero.

For example if water level is -1 and an object has elevation of -2, while in reality it will be inundated be 1 m water depth, FIAT will assume it is not.

Current behaviour

If a water level maps are used, where negative values exist, they will be assumed as zero.

Desired behaviour

First translate to water depth and then do the <0 check.

Additional Context

No response

Feature type

Adding new functionality

Improvement Description

Write the docstrings of functions and methods

Implementation Description

Write docstrings...

Additional Context

No response

Column name in exposure CSV:

"Damage Function: {naam van schade functie}", bijv. "Damage Function: Structure" en deze linkt dan aan de "Max Potential Damage: {naam van schade functie}" bijv. "Max Potential Damage: Structure"

Feature type

Adding new functionality

Improvement Description

The ability to view the missed/ skipped objects from the calculations

Implementation Description

Log those missing/ skipped objects to a txt file.
Log a message telling the user if there are missing values.

Additional Context

No response

Feature Type

• Adding new functionality

• Improving existing code

• Removing deprecated code/ clutter

Improvement Description

Test if this works with all assets in one CSV (some with method = centroid en some with method = area). Collect all results from different methods.

Implementation Description

No response

Additional Context

No response

In de exposure CSV:

• Join ID en naam uniek per gpkg (2 kolommen)
• Gpkg verwijst Join ID en naam verwijst naar kolom in exposure CSV
• "First Floor Elevation” column should be named “Ground Floor Height”
• Maak een verschil in kolommen die in de exposure CSV optioneel of vereist zijn (als ze optioneel zijn hoeven ze er niet in te staan)
• NetCDF hazard flood map (has to be GDAL compliant) @Mares2022
• Vulnerability (CSV file): The first column name contains the unit name (e.g. water depth, in meters), with the value in the rows (flood depths for example). See example:
  damage_functions.csv

Delft-FIAT version checks

• I have checked that this issue has not already been reported.

• I have checked that this bug exists on the latest version of Delft-FIAT.

Reproducible Example

No response

Current behaviour

No response

Desired behaviour

No response

Additional Context

No response

Feature type

Changing existing functionality

Improvement Description

A real way of settings the srs of the hazard file from the settings toml.

Implementation Description

Two options:

• Use the CRS as a string variable throughout the model

  • Function that use the srs from the hazard file, build a srs object from this string and use that (Clipping etc).
  • Gives problems when warping the hazard file

• Write a new file with the srs in it and use that.

Settings the srs in an existing file produces some irregular behaviour.

Additional Context

No response

Feature Type

• Adding new functionality

• Improving existing code

• Removing deprecated code/ clutter

Improvement Description

Proper 2D functionality for both Small and Large CSV structs.

Implementation Description

Write it as to resemble a tabular structure.
Ensure same speed as the old structs.

Additional Context

No response

Feature type

Adding new functionality

Improvement Description

Multiprocess multiple bands
Manager thread to write data

Implementation Description

Write it into the geom-/ grid models

• find a way to circumvent the awfully slow queue...
• Alternative to 'stitching' back together data from child processes

Additional Context

Causes headaches...

Feature Type

• Adding new functionality

• Improving existing code

• Removing deprecated code/ clutter

Improvement Description

Save results as:

• csv (incl. all input data from the exposure csv)
• geopackage incl all data that is also saved in the exposure csv (for event flood maps one extra column 'damage [unit]', and for return period flood maps multiple extra columns: one for each return period and one for the risk)

Implementation Description

No response

Additional Context

No response

Feature Type

• Adding new functionality

• Improving existing code

• Removing deprecated code/ clutter

Improvement Description

No response

Implementation Description

No response

Additional Context

No response

Feature Type

• Adding new functionality

• Improving existing code

• Removing deprecated code/ clutter

Improvement Description

No response

Implementation Description

No response

Additional Context

No response

Feature Type

• Adding new functionality

• Improving existing code

• Removing deprecated code/ clutter

Improvement Description

• Optie om centroid en area beide te gebruiken: binnen 1 exposure csv kunnen rijen (assets) staan met centroid of area als extraction method. Behoud de optie om de centroid van een polygon als extractie locatie te gebruiken maar de data wel op als polygon.
• Dem / Datum opties qua overstromingskaart
• Meerdere processes mogelijk voor de return period flood maps

Implementation Description

No response

Additional Context

No response

Feature Type

• Adding new functionality

• Improving existing code

• Removing deprecated code/ clutter

Improvement Description

Copy (partialy) from FIAT-Objects.

Implementation Description

No response

Additional Context

No response

Feature Type

• Adding new functionality

• Improving existing code

• Removing deprecated code/ clutter

Improvement Description

Implement a simple integration test that does a full FIAT run. This does not need to output the aggregation excel yet.

Implementation Description

No response

Additional Context

No response

Feature type

Adding new functionality

Improvement Description

Add the raster model into the current codebase.
Mock-up is already present.

Implementation Description

Write it like the vector model in terms of structure.

Additional Context

No response

Feature Type

• Adding new functionality

• Improving existing code

• Removing deprecated code/ clutter

Improvement Description

No response

Implementation Description

No response

Additional Context

No response

Feature Type

• Adding new functionality

• Improving existing code

• Removing deprecated code/ clutter

Improvement Description

Implementing the code that is already there to calculate the risk over different return period damages. Use workers to use different processes for multiple hazard maps with different return periods.

Als er meerdere bands in de netCDF staan dan wordt er altijd over geloopt.
In settings.toml "risk = True" of "risk = False", als deze true is dan wordt de EAD berekend (risico)

Implementation Description

No response

Additional Context

No response

Feature type

Adding new functionality

Improvement Description

• Improve the coverage of the testing of Delft-FIAT.
• Different data sources
• Usage (Python vs CLI)

Implementation Description

Write more ACCURATE tests.
Add a list below of testing that should be done (IN PROGRESS):

• ConfigReader
• All of the I/O
• Logic of Delft-FIAT

Additional Context

No response