jmigueldelgado / buhayra Goto Github PK

the extraction of reservoir polygons involves a intermediate step where a small raster with the backscatter coefficient is stored as int32 type. In order to plot the tiff and visually validate individual reservoirs it would be helpful to have as script that transforms the and georeferences the raster as it is done in polygonize.raster2shapely. The affine parameters can be obtained with:

affParam=rasterio.Affine.from_gdal(metadata[0],metadata[1],metadata[2],metadata[3],metadata[4],metadata[5])

where the metadata is given by the json file stored next to the tiff.

possibly with https://joblib.readthedocs.io/en/latest/index.html or http://www.lpsm.paris/pageperso/sangnier/pbspython.html

https://forum.step.esa.int/t/can-idepix-be-called-in-snappy/6433
http://old.esaconferencebureau.com/docs/default-source/12c04_docs2/poster2_39_brockmann.pdf?sfvrsn=2

see for example http://step.esa.int/docs/tutorials/S1TBX%20Offset%20Tracking%20Tutorial.pdf

in 4_st_simplify.job

• download all scenes again
• repeat 2_sar2watermask_year_month.sh from January to June
• repeat 2_sar2watermask_year_month.sh from July to December
• repeat thresholding and insert from January to June
• repeat thresholding and insert from July to December

options:

1. mapserver + javascript
2. folium/bokeh only
3. mapserver + markers with folium/bokeh

I tend to prefer number 3. Arguments are the power of postgres and my total ignorance of javascript... If there is a solution for folium/bokeh that can serve geojson on the range of 50 MB, then number 2 would also be fine.

rr_db = 10*log10(rr)

Values should be in the range of about [-50,10]. Keep at least 2 decimals before converting to integer.

Traceback (most recent call last):
File "/users/stud09/martinsd/local/miniconda2/envs/mongodb/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/users/stud09/martinsd/local/miniconda2/envs/mongodb/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/users/stud09/martinsd/proj/buhayra/main.py", line 75, in
main()
File "/users/stud09/martinsd/proj/buhayra/main.py", line 41, in main
ipol.insertPolygons()
File "/users/stud09/martinsd/proj/buhayra/buhayra/insertPolygons.py", line 54, in insertPolygons
feat_id = s2w.update_one({'properties.id_funceme':feat["properties"]["id_funceme"] , 'properties.ingestion_time' :feat["properties"]["ingestion_time"] },feat,upsert=True).upserted_id
File "/users/stud09/martinsd/local/miniconda2/envs/mongodb/lib/python2.7/site-packages/pymongo/collection.py", line 831, in update_one
common.validate_ok_for_update(update)
File "/users/stud09/martinsd/local/miniconda2/envs/mongodb/lib/python2.7/site-packages/pymongo/common.py", line 405, in validate_ok_for_update
raise ValueError('update only works with $ operators')
ValueError: update only works with $ operators

geojson from march and april in mongodb have invalid syntax. They are lacking one [] in the coordinates. This probably originates an error in ogr2ogr and they cannot be displayed in the webmap.

possibly with 'node.js'

after subsetting with the reservoir extend + buffer, subset with 200x200 m

There is a branch for that here I would
probably use R to do that, since it is not clear to me how to
intersect a raster with a line in python.
I have been working with hav.py , but you could
start by extending this one https://github.com/jmigueldelgado/buhayra/blob/hav/buhayra/filter_vct.R

running script number 3 on branch rm-old-polygons:

ImportError: /home/delgado/local/miniconda3/envs/dask/lib/python3.7/site-packages/fiona/../../../libgdal.so.20: undefined symbol: _ZNK6libdap5Error17g
et_error_messageB5cxx11Ev

/users/stud09/martinsd/proj/buhayra/buhayra/jobs/5_insert_polys_mongo.sh: Zeile 3: activate: Datei oder Verzeichnis nicht gefunden
inserting into mongodb

Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/users/stud09/martinsd/proj/buhayra/__main__.py", line 52, in <module>
    print(main())
  File "/users/stud09/martinsd/proj/buhayra/__main__.py", line 29, in main
    import buhayra.insertPolygons as ipol
  File "/users/stud09/martinsd/proj/buhayra/buhayra/insertPolygons.py", line 1, in <module>
    from pymongo import MongoClient
ImportError: No module named pymongo

in particular, primary key in sib or neb tables should be of type serial:
ALTER TABLE sib ADD COLUMN id SERIAL PRIMARY KEY;

Picked up JAVA_TOOL_OPTIONS: -Xmx16g
Picked up _JAVA_OPTIONS: -Xmx18g -Xss2m
1 begin imports

Traceback (most recent call last):
File "/users/stud09/martinsd/proj/buhayra/sar2watermask/sar2watermask_cluster.py", line 21, in
from buhayra.getpaths import *
ImportError: No module named buhayra.getpaths

• [x ] add column centroid to jrc_neb table
• [x ] reduce marker size and type

obtain a dataset from past data to simulate a data assimilation algorithm.

1. Model: simple rainfall-runoff model

• training dataset without assimilation for training statistical model and a hindcast dataset with assimilation
• use funceme ID 174 or jrc id 62958

2. Measurement: SAR estimation of volume

Traceback (most recent call last):
File "/home/delgado/local/miniconda3/envs/concaveman/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/delgado/local/miniconda3/envs/concaveman/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/delgado/proj/buhayra_manuscript/main.py", line 212, in
main()
File "/home/delgado/proj/buhayra_manuscript/main.py", line 91, in main
import buhayra.loops as loops
File "/home/delgado/proj/buhayra_manuscript/buhayra/loops.py", line 124
def concaveman_insert(tiffs):
^
SyntaxError: import * only allowed at module level

and add as attribute in postgresdb table

File "/users/stud09/martinsd/proj/buhayra/sar2watermask/sar.py", line 41, in sar2sigma
product = ProductIO.readProduct(sarIn+"/"+f)
RuntimeError: java.io.IOException: org.esa.s1tbx.io.sentinel1.Sentinel1ProductReader
[input=/mnt/scratch/martinsd/s1a_scenes/in/S1A_IW_GRDH_1SDV_20180528T081725_20180528T081750_022102_0263B3_A38E.zip]:
Failed to create directory within 10000 attempts (tried 1542907003351-0 to 1542907003351-9999)

reinstall rasterio or simply fiorio3.6.yml

Traceback (most recent call last):
File "rasterio/_base.pyx", line 199, in rasterio._base.DatasetBase.init
File "rasterio/_shim.pyx", line 64, in rasterio._shim.open_dataset
File "rasterio/_err.pyx", line 194, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_NotSupportedError: Scanline/tile/strip size bigger than 2GB.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/users/stud09/martinsd/local/miniconda2/envs/buhayra/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/users/stud09/martinsd/local/miniconda2/envs/buhayra/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/users/stud09/martinsd/proj/buhayra/main.py", line 125, in
main()
File "/users/stud09/martinsd/proj/buhayra/main.py", line 66, in main
thresh.threshold_loop(scenes)
File "/users/stud09/martinsd/proj/buhayra/buhayra/thresholding.py", line 20, in threshold_loop
with rasterio.open(sarOut+'/'+f,'r') as ds:
File "/users/stud09/martinsd/local/miniconda2/envs/buhayra/lib/python3.6/site-packages/rasterio/env.py", line 398, in wrapper
return f(*args, **kwds)
File "/users/stud09/martinsd/local/miniconda2/envs/buhayra/lib/python3.6/site-packages/rasterio/init.py", line 217, in open
s = DatasetReader(path, driver=driver, **kwargs)
File "rasterio/_base.pyx", line 201, in rasterio._base.DatasetBase.init
rasterio.errors.RasterioIOError: Scanline/tile/strip size bigger than 2GB.

older polygons, eg older that 2 months should be deleted from the data base. The remaining information should be kept though!

java.lang.NullPointerException
Traceback (most recent call last):
File "/users/stud09/martinsd/local/miniconda2/envs/buhayra/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/users/stud09/martinsd/local/miniconda2/envs/buhayra/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/users/stud09/martinsd/proj/buhayra/main.py", line 101, in
main()
File "/users/stud09/martinsd/proj/buhayra/main.py", line 48, in main
sar.sar2sigma(f)
File "/users/stud09/martinsd/proj/buhayra/sar2watermask/sar.py", line 296, in sar2sigma
product_oc_tn=thermal_noise_removal(product_oc)
File "/users/stud09/martinsd/proj/buhayra/sar2watermask/sar.py", line 201, in thermal_noise_removal
result = GPF.createProduct('ThermalNoiseRemoval',params,product)
RuntimeError: org.esa.snap.core.gpf.OperatorException: java.lang.NullPointerException

maybe from here https://step.esa.int/docs/v2.0/apidoc/engine/org/esa/snap/core/datamodel/GeoCoding.html

Following error, possibly related to Mask is empty! warning

File "/users/stud09/martinsd/proj/buhayra/ndwi2watermask/ndwi.py", line 95, i\
n ndwi2watermask
    "width": ndwi_int.shape[2],
IndexError: tuple index out of range

something like this:

db.sar2watermask.createIndex(
    {
        properties.id_cogerh : 1 , properties.ingestion_date : -1
    }
)

the volume calculated using approach 1c of Pereira et al has some problems:
reservoirs <5000m³ must be calculated with the older method.
besides, in our case of max reservoir extents of Ceará, 48 of 22960 reservoirs are NA because K or alpha <0, 23 reservoirs have a negative volume because K and alpha < 0.

a query of the sort

SELECT * FROM my_table WHERE ST_Distance_Sphere(the_geom, ST_MakePoint(your_lon,your_lat)) <= radius

ERROR 1: ERROR: column s.consrc does not exist
LINE 1: ...nrelid = c.oid AND a.attnum = ANY (s.conkey) AND (s.consrc L...
^

This may be caused by incompatibility of gdal<=3.0.1 and PG12

https://gis.stackexchange.com/questions/342246/ogr2ogr-from-file-to-postgresql-12-table-fails

• query Eduardo regardin WB proposal in case GRACE option does not work
• query pedro medeiros about large scale validation
• responder ao email da ARH Alentejo (Alice Fialho)

transfer data from orson2 to webserver without authenticating every time.

I would like to obtain the watermasks from orson and copy them to the webserver. Using RSA or private credentials within the webserver would be great. The idea is that we do not have to insert a password so it can run from cron

create new tables scene_neb and scene_sib with two columns: ingestion_time and scene. Populate with all processed scenes.

Use http://initd.org/psycopg/

1. orbit correction
2. calculate GLCM texture with window 3x3 pixels for whole scene. check grey level, if larger than 16 bit, scale to 8 bit. Calculate GLCM for both cases and compare.
3. apply PCA on GLCM, check absolute loading of PC1 and the loading of GLCM variance in PC1.
4. subset GLCM by 300x300m
5. calculate the most distributed number and the SD of the reservoir subset
6. based on results of 5, determine the threshold.