oda-hub / dispatcher-plugin-integral Goto Github PK

View Code? Open in Web Editor NEW

1.0 3.0 1.0 531 KB

License: Other

Python 80.04% Jupyter Notebook 19.96%

dispatcher-plugin integral

dispatcher-plugin-integral's Issues

download files for jemx dummy requests

the find search didn't yield any results

automatically make per-orbit mosaics

test how default parameters are set in job id

check that those parameters which are always set in frontend, are always set in URLs, even if not provided in api request.

slow write

ile "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/flask_app/app.py", line 254, in run_analysis
    r = query.run_query(disp_conf=app.config['conf'])
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/flask_app/dispatcher_query.py", line 1433, in run_query
    query_out = self.instrument.run_query(product_type,
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/instrument.py", line 239, in run_query
    query_out = query_obj.run_query(self, out_dir, job, run_asynch,
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 663, in run_query
    query_out = self.process_query_product(instrument,
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 582, in process_query_product
    process_products_query_out=self.process_product(instrument,self.query_prod_list,api=api,config=config,**kwargs)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 555, in process_product
    query_out= self.process_product_method(instrument, query_prod_list, api=api, **kwargs)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_osa_plugin/osa_spectrum_query.py", line 330, in process_product_method
    query_spec.write()
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/products.py", line 338, in write
    self.data.write_fits_file(file_path, overwrite=overwrite)
  File "/src/oda-api/oda_api/data_products.py", line 532, in write_fits_file
    self.to_fits_hdu_list().writeto(filename,overwrite=overwrite)

isgri lightcurves do not contain XAX_E, and are occasionally empty

@burnout87 could we make a test to request ISGRI LC for a single pointing, 066500220010.001 ?

catalog as string becomes long unbreakable string

curl "https://www.astro.unige.ch/mmoda/dispatch-data/inspect-state?token=$(cat ~/.oda-token | xargs)&job_id=62840af4b40af6c8" | jq

when job fails on backend, dispatcher may time out API

i realized it would be nicer if we use an additional common parameter to request private data and other data subsets

it's better than just relying on the token.
like "integral_data_rights" with two possible values "public" and "all-private"
that is, this parameter should be only allowed to be set to "all-private" if such the role "integral-private" is set.

make another kind of dummy products, to be used before post-processing?

to estimate if feasible
could be stored in the very same dummy directory, just called differently
might be easy

another place with osa versions

time range in INTEGRAL orbits somewhere?

we should discuss if/where it makes sense

add realtime data

once released, send a GCN with explanations

increase maximum allowed time bin to 10 ks

When extracting a long-term light curve of some sources, I noticed that occasionally, there are science windows with long duration. 10 ks should be safe value to enable one point per science window with ISGRI

make reasonable defaulf without sparql access

if file exists in cache, re-use it

add energy limits

aggregate image block

sort out osa11 version for all cases

trace scw index/db version for reproducible requests

for public/private data status
to consider

further customize OSA versions by dda component revision

add 11.2-beta

inconsistency in ISGRI and JEM-X product show

from oda_api.api import DispatcherAPI
disp=DispatcherAPI(url='http://dispatcher.staging.internal.odahub.io', instrument='mock')
par_dict={
"src_name": "4U 1700-377",
"RA": "270.80",
"DEC": "-29.80",
"T1": "2021-05-01",
"T2": "2021-05-05",
"T_format": "isot",
"instrument": "jemx",
"osa_version": "OSA11.0",
"radius": "4",
"max_pointings": "50",
"integral_data_rights": "all-private",
"jemx_num": "1",
"E1_keV": "3",
"E2_keV": "20",
"product_type": "Real",
"detection_threshold": "5",
"product": "jemx_lc",
"time_bin": "4",
"time_bin_format": "sec",
"catalog_selected_objects": "1,2,3",
"selected_catalog" :'{"cat_frame": "fk5", "cat_coord_units": "deg", "cat_column_list": [[0, 1, 2], ["GX 5-1", "MAXI SRC", "H 1820-303"], [96.1907958984375, 74.80066680908203, 66.31670379638672], [270.2771301269531, 270.7560729980469, 275.914794921875], [-25.088342666625977, -29.84027099609375, -30.366628646850586], [0, 1, 0], [0.05000000074505806, 0.05000000074505806, 0.05000000074505806]], "cat_column_names": ["meta_ID", "src_names", "significance", "ra", "dec", "FLAG", "ERR_RAD"], "cat_column_descr": [["meta_ID", "<i8"], ["src_names", "<U10"], ["significance", "<f8"], ["ra", "<f8"], ["dec", "<f8"], ["FLAG", "<i8"], ["ERR_RAD", "<f8"]], "cat_lat_name": "dec", "cat_lon_name": "ra"}',
"token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJlbWFpbCI6IkNhcmxvLkZlcnJpZ25vQHVuaWdlLmNoIiwibmFtZSI6ImNmZXJyaWdubyIsInJvbGVzIjoiYXV0aGVudGljYXRlZCB1c2VyLCBhZG1pbmlzdHJhdG9yLCBjb250ZW50IG1hbmFnZXIsIGdlbmVyYWwsIGludGVncmFsLXByaXZhdGUtcWxhLCBtYWdpYywgdW5pZ2UtaHBjLWZ1bGwsIHB1YmxpYy1wb29sLWhwYywgYW50YXJlcywgc2RzcyIsImV4cCI6MTYyMzEzNDg5M30.UVaNwQBGdcUly3bsQDH17X0Wa9fPz8nTWyphHP6yoBY"
}

data_collection_lc.show()

ID=0 prod_name=jemx_lc_0_H1820303 meta_data: {'src_name': 'H 1820 303', 'time_bin': 4.62963e-05, 'time': 'TIME', 'rate': 'RATE', 'rate_err': 'ERROR'}

ID=1 prod_name=jemx_lc_1_MAXISRC meta_data: {'src_name': 'MAXI SRC', 'time_bin': 4.62963e-05, 'time': 'TIME', 'rate': 'RATE', 'rate_err': 'ERROR'}

The "-" in the source name of "H 1820-303" disappears in the "show()" method.

clarify how ro run peruser runners

Insert warning if in JEM-X light curve a user asks for <0.1s time bins

j_ima_iros minimum time bin is 0.1 s with default to 4 s
the same is reflected in jemx_science_analysis

I have made a request from the API with 0.02 s bin job_id: 2bbfe99e
and one from the web interface with 0.02 s time bin job_id: e2c6aace

They both returned a light curve with 4 s time bin.

It is a good safety feature, but one should warn the user that the query parameters have been adapted to reflect limits.

[I do not know if this is even possible in automatic fashion, but we discussed something on weird inputs for ISGRI energy range]

isgri_lc does not implement OSA11.2-beta

e.g.:

https://www.astro.unige.ch/mmoda/?DEC=-30.361277777778&E1_keV=20&E2_keV=40&RA=275.91904166667&T1=2017-03-06T13%3A26%3A48.0&T2=2017-03-06T15%3A32%3A27.0&T_format=isot&detection_threshold=5&instrument=isgri&integral_data_rights=all-private&max_pointings=50&osa_version=OSA11.1&product_type=isgri_lc&query_status=new&query_type=Real&radius=15&scw_list=242300320010.000%2C242300330010.000%2C242300420010.000%2C242300430010.000%2C242300520010.000&src_name=4U+1820-30&time_bin=20.5&time_bin_format=sec&use_scws=form_list

it was added in a rush, and perhaps this one was forgotten. Easy hotfix is warranted.

`osa_fake` is in the list of installed instruments

I guess it should not be, can this be confirmed?

test fails in some cases?

https://github.com/oda-hub/dispatcher-app/runs/4368086679?check_suite_focus=true

maybe something is not set

enable a choice between ii_light and ii_lc_extract

https://redmine.astro.unige.ch/issues/14675

when uploading scw list file, list of science windows is limited to 50 for registered users, independently of roles

In general, the rule is

50 scw for unregistered user
500 scw for registered user
unlimited scw for users with HPC unige role

However if one uploads a list of scw, the number is limited to 50 even if you are registered. There should be a unique rule to be applied.

make mosaic from sub-mosaics

for very large mosaic (>10k) making mosaic takes a while. Also every new added scw requires complete re-making of the mosaic.

Instead, we can make mosaics per-orbit, and then split merge them. With mimosa or varmosaic.
This will resolve both issues quoted above.

`OSAVersion` missing units field

Related to oda-hub/dispatcher-app#250

Is it intended?

in jemx products: IndexError: list index out of range

Traceback (most recent call last):
  File "/src/cdci-data-analysis/cdci_data_analysis/analysis/queries.py", line 581, in process_query_product
    process_products_query_out=self.process_product(instrument,self.query_prod_list,api=api,config=config,**kwargs)
  File "/src/cdci-data-analysis/cdci_data_analysis/analysis/queries.py", line 554, in process_product
    query_out= self.process_product_method(instrument, query_prod_list, api=api, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/cdci_osa_plugin/osa_image_query.py", line 132, in process_product_method
    html_fig = query_image.get_html_draw(
  File "/src/cdci-data-analysis/cdci_data_analysis/analysis/products.py", line 386, in get_html_draw
    _du=self.data.get_data_unit(ID=data_ID)
  File "/usr/local/lib/python3.8/site-packages/oda_api/data_products.py", line 419, in get_data_unit
    return self.data_unit[ID]
IndexError: list index out of range

hide dev osa versions

meaningfull error if timesystem is down

make more clear message when the time bin is too large

prevent nrt scw in public data

isgri sum lc breaks on lesta?

mosaic crashes with many sources

When there are too many sources to consider, ii_skyimage crashes with segfault. There is an automatic workaround (increasing sig threshold), but sometimes it still return an error. In such a case a pre-set catalog should be probably used.

added as a known issue

https://github.com/oda-hub/known-issues/blob/main/README.md#isgri

can we test with token passed with oda_api?

I do not think this was done yet?

test would verify that condition set by the plugin is properly applied for the selected role.

isgri LC problem

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 581, in process_query_product
    process_products_query_out=self.process_product(instrument,self.query_prod_list,api=api,config=config,**kwargs)
  File "/usr/local/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 554, in process_product
    query_out= self.process_product_method(instrument, query_prod_list, api=api, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/cdci_osa_plugin/osa_lightcurve_query.py", line 413, in process_product_method
    _html_fig.append(query_lc.get_html_draw())
  File "/usr/local/lib/python3.8/site-packages/cdci_osa_plugin/osa_lightcurve_query.py", line 233, in get_html_draw
    dx = data['TIMEDEL']*0.5
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/fitsrec.py", line 507, in __getitem__
    return self.field(key)
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/fitsrec.py", line 694, in field
    column = self.columns[key]
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/column.py", line 1644, in __getitem__
    key = _get_index(self.names, key)
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/column.py", line 2046, in _get_index
    raise KeyError(f"Key '{key}' does not exist.")
KeyError: "Key 'TIMEDEL' does not exist."
ERROR:cdci_data_analysis.flask_app.dispatcher_query:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/column.py", line 2037, in _get_index
    indx = names.index(key.rstrip())
ValueError: 'TIMEDEL' is not in list

change to 11.2

AttributeError: 'OSAVersion' object has no attribute '_units'

WARNING:[ InstrumentQueryBackEnd : instrument-not-set ]:after clear_temp_dir 2.7239949703216553 s
INFO:cdci_data_analysis.flask_app.dispatcher_query:constructed <class 'cdci_data_analysis.flask_app.dispatcher_query.InstrumentQueryBackEnd'>:[ InstrumentQueryBackEnd : isgri ] for data_server_call_back=False
ERROR:cdci_data_analysis.flask_app.app:Exception on /api/meta-data [GET]
Traceback (most recent call last):
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask_restx/api.py", line 672, in error_router
    return original_handler(e)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/flask_app/app.py", line 68, in run_api_meta_data
    return query.get_meta_data()
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/flask_app/dispatcher_query.py", line 656, in get_meta_data
    l.append(self.instrument.get_parameters_list_as_json(prod_name=prod_name))
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/instrument.py", line 396, in get_parameters_list_as_json
    l.append(_query.get_parameters_list_as_json(prod_dict=self.query_dictionary))
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/queries.py", line 247, in get_parameters_list_as_json
    l.append(par.reprJSON())
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/parameters.py", line 305, in reprJSON
    return dict(name=self.name, units=self.units, value=self.value)
  File "/pyenv/versions/3.8.5/lib/python3.8/site-packages/cdci_data_analysis/analysis/parameters.py", line 218, in units
    return self._units
AttributeError: 'OSAVersion' object has no attribute '_units'

Line 161 in fe2ccb7

src_name_list = [n.replace('lc_','') for n in src_name_list]

https://gitlab.astro.unige.ch/oda/api-clients/oda_api_wrapper/-/blob/master/oda_integral_wrapper/wrapper.py#L780
(lc is the object returned by oda_api)

and let's make a test for it!

oda-hub / dispatcher-plugin-integral Goto Github PK

dispatcher-plugin-integral's Issues

Recommend Projects

Recommend Topics

Recommend Org