oda-hub / dispatcher-plugin-integral Goto Github PK

View Code? Open in Web Editor NEW

1.0 3.0 1.0 531 KB

License: Other

Python 80.04% Jupyter Notebook 19.96%

dispatcher-plugin integral

dispatcher-plugin-integral's Introduction

cdci_osa_plugin

OSA INTEGRAL plugin for cdci_data_analysis

What's the license?

cdci_osa_plugin is distributed under the terms of The MIT License.

Who's responsible?

Andrea Tramacere, Volodymyr Savchenko

ISDC Data Centre for Astrophysics, Astronomy Department of the University of Geneva, Chemin d'Ecogia 16, CH-1290 Versoix, Switzerland

test

set the environment variable with the dispatcher url, e.g.
- export DISP_URL='cdcicn01.isdc.unige.ch:32003/dispatch-data'
if need set the token environment variable, e.g.
- export ODA_API_TOKEN=$HOME/.oda-api-token
pytest ../cdci_osa_plugin/tests/test_osa.py::test_jemx_image -sv

dispatcher-plugin-integral's People

Contributors

Stargazers

Watchers

Forkers

burnout87

dispatcher-plugin-integral's Issues

inconsistency in ISGRI and JEM-X product show

from oda_api.api import DispatcherAPI
disp=DispatcherAPI(url='http://dispatcher.staging.internal.odahub.io', instrument='mock')
par_dict={
"src_name": "4U 1700-377",
"RA": "270.80",
"DEC": "-29.80",
"T1": "2021-05-01",
"T2": "2021-05-05",
"T_format": "isot",
"instrument": "jemx",
"osa_version": "OSA11.0",
"radius": "4",
"max_pointings": "50",
"integral_data_rights": "all-private",
"jemx_num": "1",
"E1_keV": "3",
"E2_keV": "20",
"product_type": "Real",
"detection_threshold": "5",
"product": "jemx_lc",
"time_bin": "4",
"time_bin_format": "sec",
"catalog_selected_objects": "1,2,3",
"selected_catalog" :'{"cat_frame": "fk5", "cat_coord_units": "deg", "cat_column_list": [[0, 1, 2], ["GX 5-1", "MAXI SRC", "H 1820-303"], [96.1907958984375, 74.80066680908203, 66.31670379638672], [270.2771301269531, 270.7560729980469, 275.914794921875], [-25.088342666625977, -29.84027099609375, -30.366628646850586], [0, 1, 0], [0.05000000074505806, 0.05000000074505806, 0.05000000074505806]], "cat_column_names": ["meta_ID", "src_names", "significance", "ra", "dec", "FLAG", "ERR_RAD"], "cat_column_descr": [["meta_ID", "<i8"], ["src_names", "<U10"], ["significance", "<f8"], ["ra", "<f8"], ["dec", "<f8"], ["FLAG", "<i8"], ["ERR_RAD", "<f8"]], "cat_lat_name": "dec", "cat_lon_name": "ra"}',
"token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJlbWFpbCI6IkNhcmxvLkZlcnJpZ25vQHVuaWdlLmNoIiwibmFtZSI6ImNmZXJyaWdubyIsInJvbGVzIjoiYXV0aGVudGljYXRlZCB1c2VyLCBhZG1pbmlzdHJhdG9yLCBjb250ZW50IG1hbmFnZXIsIGdlbmVyYWwsIGludGVncmFsLXByaXZhdGUtcWxhLCBtYWdpYywgdW5pZ2UtaHBjLWZ1bGwsIHB1YmxpYy1wb29sLWhwYywgYW50YXJlcywgc2RzcyIsImV4cCI6MTYyMzEzNDg5M30.UVaNwQBGdcUly3bsQDH17X0Wa9fPz8nTWyphHP6yoBY"
}

data_collection_lc.show()

ID=0 prod_name=jemx_lc_0_H1820303 meta_data: {'src_name': 'H 1820 303', 'time_bin': 4.62963e-05, 'time': 'TIME', 'rate': 'RATE', 'rate_err': 'ERROR'}

ID=1 prod_name=jemx_lc_1_MAXISRC meta_data: {'src_name': 'MAXI SRC', 'time_bin': 4.62963e-05, 'time': 'TIME', 'rate': 'RATE', 'rate_err': 'ERROR'}

The "-" in the source name of "H 1820-303" disappears in the "show()" method.

test fails in some cases?

https://github.com/oda-hub/dispatcher-app/runs/4368086679?check_suite_focus=true

maybe something is not set

in jemx products: IndexError: list index out of range

Traceback (most recent call last):
  File "/src/cdci-data-analysis/cdci_data_analysis/analysis/queries.py", line 581, in process_query_product
    process_products_query_out=self.process_product(instrument,self.query_prod_list,api=api,config=config,**kwargs)
  File "/src/cdci-data-analysis/cdci_data_analysis/analysis/queries.py", line 554, in process_product
    query_out= self.process_product_method(instrument, query_prod_list, api=api, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/cdci_osa_plugin/osa_image_query.py", line 132, in process_product_method
    html_fig = query_image.get_html_draw(
  File "/src/cdci-data-analysis/cdci_data_analysis/analysis/products.py", line 386, in get_html_draw
    _du=self.data.get_data_unit(ID=data_ID)
  File "/usr/local/lib/python3.8/site-packages/oda_api/data_products.py", line 419, in get_data_unit
    return self.data_unit[ID]
IndexError: list index out of range

hide dev osa versions

clarify how ro run peruser runners

make reasonable defaulf without sparql access

Meta data of JEM-X light curves contain spaces instead of '+' or '-'

The code that gets the source names from ddosa seem the one around this line, but it does not perform the string replacement that we search.

dispatcher-plugin-integral/cdci_osa_plugin/osa_lightcurve_query.py

Line 161 in fe2ccb7

src_name_list = [n.replace('lc_','') for n in src_name_list]

isgri sum lc breaks on lesta?

make another kind of dummy products, to be used before post-processing?

to estimate if feasible
could be stored in the very same dummy directory, just called differently
might be easy

Insert warning if in JEM-X light curve a user asks for <0.1s time bins

j_ima_iros minimum time bin is 0.1 s with default to 4 s
the same is reflected in jemx_science_analysis

I have made a request from the API with 0.02 s bin job_id: 2bbfe99e
and one from the web interface with 0.02 s time bin job_id: e2c6aace

They both returned a light curve with 4 s time bin.

It is a good safety feature, but one should warn the user that the query parameters have been adapted to reflect limits.

[I do not know if this is even possible in automatic fashion, but we discussed something on weird inputs for ISGRI energy range]

enable a choice between ii_light and ii_lc_extract

https://redmine.astro.unige.ch/issues/14675

test how default parameters are set in job id

check that those parameters which are always set in frontend, are always set in URLs, even if not provided in api request.

include request suggestions

automatically make per-orbit mosaics

`OSAVersion` missing units field

Related to oda-hub/dispatcher-app#250

Is it intended?

catalog as string becomes long unbreakable string

curl "https://www.astro.unige.ch/mmoda/dispatch-data/inspect-state?token=$(cat ~/.oda-token | xargs)&job_id=62840af4b40af6c8" | jq

very large requests (>500 scw)

change to 11.2

slow write

ile "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/flask_app/app.py", line 254, in run_analysis
    r = query.run_query(disp_conf=app.config['conf'])
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/flask_app/dispatcher_query.py", line 1433, in run_query
    query_out = self.instrument.run_query(product_type,
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/instrument.py", line 239, in run_query
    query_out = query_obj.run_query(self, out_dir, job, run_asynch,
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 663, in run_query
    query_out = self.process_query_product(instrument,
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 582, in process_query_product
    process_products_query_out=self.process_product(instrument,self.query_prod_list,api=api,config=config,**kwargs)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 555, in process_product
    query_out= self.process_product_method(instrument, query_prod_list, api=api, **kwargs)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_osa_plugin/osa_spectrum_query.py", line 330, in process_product_method
    query_spec.write()
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/products.py", line 338, in write
    self.data.write_fits_file(file_path, overwrite=overwrite)
  File "/src/oda-api/oda_api/data_products.py", line 532, in write_fits_file
    self.to_fits_hdu_list().writeto(filename,overwrite=overwrite)

isgri LC problem

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 581, in process_query_product
    process_products_query_out=self.process_product(instrument,self.query_prod_list,api=api,config=config,**kwargs)
  File "/usr/local/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 554, in process_product
    query_out= self.process_product_method(instrument, query_prod_list, api=api, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/cdci_osa_plugin/osa_lightcurve_query.py", line 413, in process_product_method
    _html_fig.append(query_lc.get_html_draw())
  File "/usr/local/lib/python3.8/site-packages/cdci_osa_plugin/osa_lightcurve_query.py", line 233, in get_html_draw
    dx = data['TIMEDEL']*0.5
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/fitsrec.py", line 507, in __getitem__
    return self.field(key)
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/fitsrec.py", line 694, in field
    column = self.columns[key]
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/column.py", line 1644, in __getitem__
    key = _get_index(self.names, key)
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/column.py", line 2046, in _get_index
    raise KeyError(f"Key '{key}' does not exist.")
KeyError: "Key 'TIMEDEL' does not exist."
ERROR:cdci_data_analysis.flask_app.dispatcher_query:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/column.py", line 2037, in _get_index
    indx = names.index(key.rstrip())
ValueError: 'TIMEDEL' is not in list

could we make a test for LC?

for single scw 066500220010.001 and E1 25, E2 80 ?

further customize OSA versions by dda component revision

make mosaic from sub-mosaics

for very large mosaic (>10k) making mosaic takes a while. Also every new added scw requires complete re-making of the mosaic.

Instead, we can make mosaics per-orbit, and then split merge them. With mimosa or varmosaic.
This will resolve both issues quoted above.

sort out osa11 version for all cases

enable very large time bins

isgri lightcurves do not contain XAX_E, and are occasionally empty

@burnout87 could we make a test to request ISGRI LC for a single pointing, 066500220010.001 ?

`osa_fake` is in the list of installed instruments

I guess it should not be, can this be confirmed?

jemx images are not shown

I suspect parsing issue

AttributeError: 'OSAVersion' object has no attribute '_units'

WARNING:[ InstrumentQueryBackEnd : instrument-not-set ]:after clear_temp_dir 2.7239949703216553 s
INFO:cdci_data_analysis.flask_app.dispatcher_query:constructed <class 'cdci_data_analysis.flask_app.dispatcher_query.InstrumentQueryBackEnd'>:[ InstrumentQueryBackEnd : isgri ] for data_server_call_back=False
ERROR:cdci_data_analysis.flask_app.app:Exception on /api/meta-data [GET]
Traceback (most recent call last):
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask_restx/api.py", line 672, in error_router
    return original_handler(e)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/flask_app/app.py", line 68, in run_api_meta_data
    return query.get_meta_data()
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/flask_app/dispatcher_query.py", line 656, in get_meta_data
    l.append(self.instrument.get_parameters_list_as_json(prod_name=prod_name))
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/instrument.py", line 396, in get_parameters_list_as_json
    l.append(_query.get_parameters_list_as_json(prod_dict=self.query_dictionary))
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 247, in get_parameters_list_as_json
    l.append(par.reprJSON())
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/parameters.py", line 305, in reprJSON
    return dict(name=self.name, units=self.units, value=self.value)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/parameters.py", line 218, in units
    return self._units
AttributeError: 'OSAVersion' object has no attribute '_units'

time range in INTEGRAL orbits somewhere?

we should discuss if/where it makes sense

trace scw index/db version for reproducible requests

for public/private data status
to consider

add realtime data

once released, send a GCN with explanations

i realized it would be nicer if we use an additional common parameter to request private data and other data subsets

it's better than just relying on the token.
like "integral_data_rights" with two possible values "public" and "all-private"
that is, this parameter should be only allowed to be set to "all-private" if such the role "integral-private" is set.

meaningfull error if timesystem is down

sub-scw extraction

Seems like no change to frontend, core dispatcher, or backend is needed.
but need to test if there are no hidden difficulties.

few extra keywords needed in lightcurves

HEASOFT lcurve needs specific keywords in the lightcurve output. dispatcher plugin should set them.

see there:

https://gitlab.astro.unige.ch/oda/api-clients/oda_api_wrapper/-/blob/master/oda_integral_wrapper/wrapper.py#L780
(lc is the object returned by oda_api)

and let's make a test for it!

if file exists in cache, re-use it

prevent nrt scw in public data

isgri_lc does not implement OSA11.2-beta

e.g.:

https://www.astro.unige.ch/mmoda/?DEC=-30.361277777778&E1_keV=20&E2_keV=40&RA=275.91904166667&T1=2017-03-06T13%3A26%3A48.0&T2=2017-03-06T15%3A32%3A27.0&T_format=isot&detection_threshold=5&instrument=isgri&integral_data_rights=all-private&max_pointings=50&osa_version=OSA11.1&product_type=isgri_lc&query_status=new&query_type=Real&radius=15&scw_list=242300320010.000%2C242300330010.000%2C242300420010.000%2C242300430010.000%2C242300520010.000&src_name=4U+1820-30&time_bin=20.5&time_bin_format=sec&use_scws=form_list

it was added in a rush, and perhaps this one was forgotten. Easy hotfix is warranted.

when uploading scw list file, list of science windows is limited to 50 for registered users, independently of roles

In general, the rule is

50 scw for unregistered user
500 scw for registered user
unlimited scw for users with HPC unige role

However if one uploads a list of scw, the number is limited to 50 even if you are registered. There should be a unique rule to be applied.

add energy limits

mosaic crashes with many sources

When there are too many sources to consider, ii_skyimage crashes with segfault. There is an automatic workaround (increasing sig threshold), but sometimes it still return an error. In such a case a pre-set catalog should be probably used.

added as a known issue

https://github.com/oda-hub/known-issues/blob/main/README.md#isgri