aifdr / riab Goto Github PK

We need to stop relying on www.aifdr.org:8080/geoserver and provide a script to add our own data by default.

So far only point data is used.

Look at Shapely.
Think about good numpy data structure for this

GeoServer does not recognise WKT projection information unless it has an EPSG code. Projections from e.g. ESRI do not have this to ESRI data uploaded to GeoServer are recorded as having no native spatial reference system. The result is that such layers downloaded from GeoServer in e.g. gml or geojson format come without any projection information. The work around is to force declared SRS in the web interface, but this functionality is not available programmatically and therefore not useful to GeoNode applications.

Need to resolve this issue.

Based on Abbys table, data from Padang and Ariel's project

The Risiko application is written in Bahasa Indonesia.
However, as it is applicable more widely it should make use of a proper internationalisation mapping.

Running the test suite (and code quality audit) creates a lot of unnecessary output. When all tests pass and code quality is good (as per pylint and pep8) the output should be minimal.
Currently, it is very hard to pick out potential problems from the output produced!

In particular, we always get huge amounts of logging output from GeoNode's upload function and also the logging from Risiko's calculate function.

I propose suppressing GeoNode's logging to screen completely (except for errors) and suppress risiko's log when running testing only.

This one is for everyone:
Write the tests and continually make sure they all pass.

Need to make sure it runs with Internet Explorer

Use the evacuation zone information from Badan Geologi/BNPB to calculate how many people needed to be evacuated
Version 1 - use gridded population data (2010)
Version 2 - use BNPB's exposure data polygons (calculated from landuse and BPS population data)

This depends on successfully implementing Issue #63 (#63)

In v0.1, when layers are added automatically (hazard,exposure, impact) they don't have the same options as when they are added through the widget.

The Risiko demo application (demo.riskinabox.org) crashes regularly when trying to upload a calculated result to the internal GeoNode/GeoServer. The error message is typically a None condition in the belly of GeoNetwork (inside GeoNode) but sometimes an Internal Server Error (500) from GeoServer (#47).

Typically the stack trace is

Environment:

Request Method: POST
Request URL: http://demo.riskinabox.org/api/v1/calculate/
Django Version: 1.2
Python Version: 2.6.6
Installed Applications:
['django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.admin',
'staticfiles',
'django_extensions',
'registration',
'profiles',
'avatar',
'geonode.core',
'geonode.maps',
'geonode.proxy',
'impact',
'django_nose']
Installed Middleware:
('django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware')

Traceback:
File "/home/software/riab_env/lib/python2.6/site-packages/django/core/handlers/base.py" in get_response

1.                 response = callback(request, _callback_args, *_callback_kwargs)
   

   File "/home/software/riab/impact/views.py" in calculate

2.               user=request.user,
   

   File "/home/software/riab/risiko/utilities.py" in save_to_geonode

3. layer = file_upload(filename, user=user, title=title, overwrite=False)
   

   File "/home/software/geonode/src/GeoNodePy/geonode/maps/utils.py" in file_upload

4. new_layer = save(layer, filename, theuser, overwrite)
   

   File "/home/software/geonode/src/GeoNodePy/geonode/maps/utils.py" in save

5.                              owner=user,
   

   File "/home/software/riab_env/lib/python2.6/site-packages/django/db/models/manager.py" in get_or_create

6.     return self.get_query_set().get_or_create(**kwargs)
   

   File "/home/software/riab_env/lib/python2.6/site-packages/django/db/models/query.py" in get_or_create

7.             obj.save(force_insert=True, using=self.db)
   

   File "/home/software/riab_env/lib/python2.6/site-packages/django/db/models/base.py" in save

8.     self.save_base(using=using, force_insert=force_insert, force_update=force_update)
   

   File "/home/software/riab_env/lib/python2.6/site-packages/django/db/models/base.py" in save_base

9.             created=(not record_exists), raw=raw)
   

   File "/home/software/riab_env/lib/python2.6/site-packages/django/dispatch/dispatcher.py" in send

10.         response = receiver(signal=self, sender=sender, **named)
    

    File "/home/software/geonode/src/GeoNodePy/geonode/maps/models.py" in post_save_layer

11. instance.save_to_geonetwork()
    

    File "/home/software/geonode/src/GeoNodePy/geonode/maps/models.py" in save_to_geonetwork

12.         md_link = gn.create_from_layer(self)
    

    File "/home/software/geonode/src/GeoNodePy/geonode/geonetwork.py" in create_from_layer

13.     self.set_metadata_privs(layer.uuid, {"all":  {"view": True}})
    

    File "/home/software/geonode/src/GeoNodePy/geonode/geonetwork.py" in set_metadata_privs

14.     data_dbid = doc.find('metadata/{http://www.fao.org/geonetwork}info/id').text
    

Exception Type: AttributeError at /api/v1/calculate/
Exception Value: 'NoneType' object has no attribute 'text'

Occasionally we also see an error 500 - Internal Server Error

Numbers in legend must be derived from extrema (range) of data, not hard coded.

Use get_extrema() method of layer objects.

Look at always displaying the legend on the front page rather than hidden behind the layer selection tab.

Currently the list of hazard and exposure layers is hardcoded, we need to to be able to read the stores of the viewer and then populate the hazard and exposure stores from that one.

The impact modelling formula is written in the rst file using LaTeX. However, it doesn't render as math. What is going wrong?

One test - which is failing at the moment - reveals that the extrema provided by GDAL through the function ComputeRasterMinMax(1) do not agree the with the actual extrema found in the data.

This functionality and the test has been removed because it is not really needed in
this project. The last version which is dated 6 March 2011 (GMT+7) that shows the problem is:
https://github.com/AIFDR/riab_core/commit/950e222380a08403016c359bda4cee279fc68

Text from code
FIXME (Ole): Gdal rounds everything to single precision.
Wrote to the mailing list with a test example 6th August 2010
There are large errors in extrema computed by ComputeRasterMinMax().
This has not been reported.
See also TRAC pages: http://www.aifdr.org/projects/riab

This is most likely a bug in GeoNode proper when running GeoServer 2.1

Currently layers come separated into hazard and exposure.
I will be more general to just supply one list of layer objects and then have helper functions to filter, manipulate and do the interpolation.

This cannot be fully done until keywords are firmly embedded in the layer metadata.

The idea is to be able to select all or some of the exposure levels and have one calculation be run for each exposure level.

This to collect priorities for GeoNode/GeoServer further development:

This has moved to Google Docs at: https://docs.google.com/document/d/1MnmRxYx_vWs62zJQto2eVFRiCVKkG7h6banvaZUFo5U/edit?hl=en_GB&authkey=CI6b4MAH

• Ability to serve Desktop GIS (especially ESRI), possibly through SDE. This way, datasets can be hosted on GeoServer/GeoNode but still be used with existing workflows.
• Expose all necessary (for RIAB) functionality as REST. This issue should be rephrased as follows: Provide a Python interface that will allow direct (i.e. not by downloading files) reading and writing of layers in geoserver in addition to necessary auxiliary functionality (workspaces, deletion, queries etc). Such an API would be essential to any kind of analysis tool that use spatial data.
• Ease of installing development version (?). This is still challenge - even for experts. One way forward is to refactor GeoNode up into the geonode library and geonode application. David has written this up: LINK?
• Tight integration with OSM needed. It should be as simple as pointing GeoNode to the url of an OSM server.
• Internationalisation is done differently in RIAB and TSUDAT. A common standard would be preferable.
• Ability to contour raster layers
• GeoServer does not recognise WKT projection information unless it has an EPSG code. Projections from e.g. ESRI do not have this. The result is that layers downloaded from GeoServer in e.g. gml or geojson format come without projection information.
• If a vector layer already exists uploading it with REST fails with the message 'Error occurred unzipping file:error in opening zip file'. Rasters work fine in that they can be uploaded multiple times.
• Synchronisation between layers in GeoServer and GeoNode: If layers in GeoServer are deleted they still appear in GeoNode.
• Data into PostGIS when uploaded to GeoServer
• Keywords seem to exist independently in GeoServer, GeoNode and GeoNetwork. This leads to confusion and inconsistencies. Also it doesn't seem to be possible to upload keywords to Geoserver through the API.
• GeoNode must have have a test suite.
• Every single attempt to write code to the GeoServer/GeoNode API result in silent errors and unveils a cascade of work arounds, untested code and inconsistencies.

Sometimes absolute numbers are important to emergency manager.
However, sometimes a proportional view would be useful too. However might want to include in risk modeling sequence aggregation by jurisdictional boundaries.

Could be two output layers - but will have to be defined by the plugin.

See https://docs.google.com/document/d/1MnmRxYx_vWs62zJQto2eVFRiCVKkG7h6banvaZUFo5U/edit?hl=en_GB&authkey=CI6b4MAH

Plugins that are not compatible with input data keywords must be excluded and there must be a test for this condition.

The HazardCombo and ExposureCombo list layers differently: Raster types, just by name (even though they do have a workspace) and Vector type with their workspace preceding the layer name.
For example

Earthquake_Ground_Shaking
Population_2010

but

exposure:lembang_schools

This has ramifications further down in the applications as these layer names are are used to download data.

Currently keys in dictionaries - which is what defines the fields in attributes in vector data - are truncated to width of 10. This should be increased substantially or even better be dynamic, i.e. any field width should be allowed.

Attempted this in https://github.com/AIFDR/riab_core/commit/779bf62d1e7a7f80a6548cfef8b138dd61bf5403

However, it didn't work. Need to understand what OGR does.

It says 401 - Full authentication required. The demo server is running directly off shiva, but the same issue happens when run from one of the virtual machines.

Implement functionality to inquire about values and features for raster and vector layers in map view.
This functionality exists in the standard GeoServer preview screen.

As part of this feature, please take a good hard look at the Javascript code and make sure the right libraries are used and refactor where needed.

Current demo seems to work only with Firefox.
IE and Chrome gives an empty frame without the OpenLayers component.

Implement flood use case from Marco Haartman

This was flagged in the original road map. We need a use case. How important is it?

Keywords seem to exist independently in GeoServer, GeoNode and GeoNetwork. This leads to confusion and inconsistencies. Also it doesn't seem to be possible to upload keywords to Geoserver through the API.

This is critical for Risk in a Box to determine the nature of input layers and which calculation is admissible for each combination.

Implement and test 2D interpolation where some numbers are NaN (IEEE Not-a-Number).
Apply this in the context of ASCII grids where NODATA of -9999 gets replaced by NaN.

Show bounding box used in calculations with result. That will highlight the domain and avoid misinterpretation of having no hazard outside that box.

Re-establish Readthedoc post commit hook on new repository

The current Risiko javascript is a mono-lithic code. Refactor into smaller javascript libraries.

Show only the exposure layers that are compatible with the selected hazard

Todo after keywords are done

The test case for this issue is the schools example that currently appears in red

This will be useful for python-FALL3D hazard outputs. The levels are

Load (kg/m^2): Impact

0.1 - 90 Loss of crops and livestock
90 - 150: Cosmetic Damage
150 - 300: Partial building collapse
300 - inf: Complete building collapse

Button to reset the web page

Once a calculation is complete - to avoid confusion, the final impact layer should be the only layer shown. The user might need to be reminded they can turn back on layers using the right tab.

Create system to trace quality of hazard level, exposure data and impact function plugin; and assign quality to impact map. Four stars will be used with (1) being the poorest and ***(4) the best. The resulting quality will be the minimum of the qualities of all 3 input types assigned to the output

On the main Riab map screen there are two single select combo buttons to add hazard and exposure layers. When these are changed a new layer is added and however any previously choosen layer is not removed. This is confusing and unexpected behaviour for new users.

Using tsunami inundation for Manado (from ARF DiREX) calculate number of people in greater than 1 m of water. Test data needs to be created and then pulled into RIAB.
This simple approach matches the approach of the Indonesian tsunami risk assessment guidelines.

The demo (as of 22 March) showed some strange outliers in the school damage dataset.
These are not present in the unit test so it should be investigated.

Data stored by OGR with the 'ESRI Shapefile' works fine. However, if the driver is changed to GML the geospatial reference information is not stored. Hence GML cannot be used at the moment.

im for at least 80% coverage per module