rizac / egsim Goto Github PK

With OpenQuake>=3.10.0 the GSIM string representation might not match the class name, i.e. selecting e.g. "BindiEtAl2014Rjb" for trellis computation, the relative response data contains the GSIM key as "BindiEtAl2014Rjb(adjustment factor=0.0)". This is performed here:

https://github.com/GEMScienceTools/gmpe-smtk/blob/e2e537dada1cb120040b08ed12f1db2109c9fd0d/smtk/trellis/trellis_plots.py#L117

and it was probably added in some PR in the last year. The problem is just how should eGSIM behave now in general. Two possible scenarios:

a) Leave everything as it is (proposed solution in previous email, I know, but needs a fix, see below)

CONS: The returned GSIM names might not be the same as the user input, which is misleading and must be changed in smtk (see linked code above. I can deal with that if we choose this strategy)

PROS: Easiest and cleaner solution, as long as GSIMs needs very seldom to be given with input different than the defaults.

b) Enable GSIM parameters as input (i.e. "BindiEtAl2014Rjb(adjustment factor=0.0)"), providing a string "parsing" in eGSIM.

CONS: parsing might be prone to errors and hard to maintain (how much can we rely OpenQuake to not change their string representation in the future? Is this parsing easy, or will turn into a mess due to extremely specific and different GSIM behaviours?)

PROS: This is the most flexible solution also considering that several GMPE suites are now represented by a single GMPE with user-specified inputs.

(all in all, probably the real turning point is to understand and assess how much is important in a typical eGSIM use cases to be able to customise GSIMs)

The KothaEtAl2020ESHM20 and KothaEtAl2020EHSM20SlopeGeology GMPEs require region as a site attribute. Need to add the region parameter into the configuration for the Model-to-Model comparisons. This takes an integer value in the range [0, 5], and should have a default of 0 if not specified.

when I do a selection from the ESM flatfile only with magnitude > 4.

Then I do another selection of (magnitude > 4.0) & (rrup < 300), which should still return most of the records in the database, but now returns very few records and the plot suggests that the smallest magnitude is M 4.9

The documentation pages for the API don't appear to be visible on the web (as of 28/08/2023)

Tests of Trellis vs magnitude do not produce the same results as in old smtk. Reason is that for each magnitude we initialize now a new Gsimrupture, whereas in the old code it seem that we initialized only one rupture for all magnitudes:

https://github.com/GEMScienceTools/gmpe-smtk/blob/7948682b32ef804a52d266dfcc94512da01149ab/smtk/trellis/trellis_plots.py#L434C9-L439C61

The behavior looks right in the new version and a bug in the old one, but I need confirmation for this
(@g-weatherill no need to reply here, we'll discuss in the next days)

Some of the residuals are odd. For example I compared the BindiEtAl2014Rjb GMPE against the selection

(magnitude > 4.0) & (rrup < 300) & (station_country == "IT")

which should correspond to more or less the data set used by D.B. and usually provides a very good fit to the model, but now I seem to get significant trends in the residuals (skewed toward negative values).

Currently I am struggling little bit to implement file upload.

I am talking here about the API, i.e. users coding in their notebook (form upload is another story I can deal with).

To upload a file with a Python request, I thought we just needed some adjustments to the already provided code snippets. It turns out it's not the case (a starting discussion here: https://stackoverflow.com/a/27050525. Links here are just for ma as reminder). We cannot expect users to code 30-40 lines of code for each file upload. Possible workarounds are:

1. Document that users should install the requests module. installation overhead, but apparently then it's 2-3 lines of code: https://stackoverflow.com/a/22567429

2. Cleanup as much as possible existing examples (as said, 30-40 lines of code. See links below) and create a code snippet to be copied/pasted by users. No installation overhead but honestly the least attractive option: it's harder to maintain for us and scary/impractical for notebook users
   - https://gist.github.com/rhoit/9573c40feaeb3cf44b4a8544dc0ae2a1
   - https://stackoverflow.com/a/681182
   - https://gist.github.com/rhoit/9573c40feaeb3cf44b4a8544dc0ae2a1 (simplified version of
   - https://www.matteomattei.com/http-https-get-and-post-requests-with-python-including-file-upload/

3. create an egsim_client.py module that wraps the code in 2 and exposes simple functions as in 1. The module needs only yo be downloaded in the user project or folder and not necessarily installed. Still harder to maintain for us

@g-weatherill what is your opinion here? Also, I am googling a little bit it on uploaded file size. we can think of a maximum size of 1.5-2Mb. Still very heuristic and not tested, but let's stick to these numbers: it would be between 1/10 and 1/5 of the ESM flatfile i.e. roughly between 2000 and 4000 records. Is that covering most use cases, nobody will ever use so "few" rows and we are in troubles? what else?

Thanks

For reference:

pandas-dev/pandas#11424

The URL above describes a non-documented feature that allows flexible depths in indexes. E.g.:

   "PGA"       "mag"
   "model"            # <- empty string under "mag"
0  0.52357   0.295221
1  0.108467  0.772837
2  0.29988   0.547574

>>> type(df['PGA'])
<class 'pandas.core.frame.DataFrame'>

>>> type(df['mag'])
<class 'pandas.core.series.Series'>

Turns out the csfr token is needed when POST ing a API request (not from the browser). More info here.

https://docs.djangoproject.com/en/3.2/ref/csrf/

(not urgent, must be done before the 2.0 release)

In the new ongoing ContextDB implementation, for each event /records tuple, we have our Ctx object with all Site/Rupture/Distance parameters. Because now we might have new parameters, I added a last part of code where I query the egsim db for all registered flatfile columns and relative Ctx attributes in order to add missing attributes to Ctx, in case.

The code works, perfect. But I noticed that in the legacy code the station coordinates were set as Ctx.lats, Cts.lons (plural). Hence, now we add (redundantly?) Ctx.lat and Ctx.lon, because these are the registered attribute names in eGSIM and OpenQuake.

The question here is not only what to do, but first of all how the code actually works (this is why is maybe better to have a zoom) because logically, it should work with lat and lon (no plural, as defined in OpenQuake). Probably there is some part in smtk where the conversion tooks place, in case we might see how to simplify the code, because this might be related to another topic I would like to discuss (i.e., stick to OpenQuake names whenever possible, there are in eGSIM some patches throughout the code for normalizing almost equal names, that might be avoided for clarity)

When considering distances that are small (< 5 km) the distance calculations appear to behave unexpectedly and give extremely large values for some cases, resulting in incorrect spectra. This occurs with the following configuration when the dip is set to 60.

{
    "aspect": "1.5",
    "dip": "60",
    "distance": "1,3,5",
    "magnitude": "6.9,7.4",
    "model": [
        "AkkarBommer2010",
        "ChiouYoungs2008",
        "CauzziFaccioli2008",
        "ZhaoEtAl2006Asc"
    ],
    "plot": "s",
    "stdev": true
}

If the dip is set to 90 then the distances are correctly calculated and the spectra look normal.

This could be potentially a bug in the SMTK code, so investigation needed.