Add a limit on ndethist<20 for early SNe class. Thanks!

schema_client = client.schema()
schema_list = list(schema_client.columnNames())
d = {i: schema_client.type(i) for i in schema_list}

return jsonify({'ZTF fields': d})

or something along this

When I click on the Explore button in the navbar (top-right), the menu goes outside of the screen

The SNN scores are computed for each alert, hence using only partial information. In the science portal, we aggregate data, so the user should be able to re-run SNN on all aggregated data.

Once the coordinates are transformed to hours, m, s for alpha and degrees, arcmin, arcsec for delta, the number of digits after the decimal point can not be the same: one more significant digit for alpha is needed.
For example, 13h 09m 47.706s -23d 23' 01.79''

If the CDS xmatch service is down, we skip the initial xmatch step and we continue with the other modules. Each cdsxmatch field in alerts contain Fail, but it is not directly searchable.

Action item: Add an entry in the class dropdown button with the Fail category

If the user access the web portal via mobile phone there should be a warning saying that the mobile website is under construction -- or saying that we advise use in the desktop... just to demonstrate we are aware of the situation.

When opening an object page, one or several of the cutouts are not displayed correctly. i think the error comes from the fact that the task is multi-threaded (+different callbacks), and the client that retrieves the cutouts does not like it...
Action item: gather all db calls in the same callback.

See https://dash.plotly.com/datatable/editable #Export Data Table

Currently, users can only perform search along one axis (time, space, class or objectId). Ideally, we need to combine different axes (time & space, or space & class...). For a non-relational database, this is tricky, and we might have to create complex index tables based on pre-defined needs.

It would be nice to have the time axis in Julian Dates (or Modified) as a second x-axis on the top, probably.

New features: magnitude increase rate per day and colour evolution per day (e.g. g-r) for the first 14 days of a new detection. More specifically between the last two measurements and the first and last measurement available.
Corroborated by https://arxiv.org/pdf/2010.09724.pdf Figure 7!

calibration might be an issue...

Comment from Maria:

Is it possible to extend the representation of different coordinates formats? 
Astronomers not only like to use them in a form "hh:mm:ss and dd:mm:ss", 
but also "hh mm ss and dd mm ss" or "275d11m15.6954s+17d59m59.876s", etc. 
This can be useful if we are looking for the alerts on a place of some known objects 
(e.g., dwarf novae) and the coordinates of this object are copied from some astronomical 
catalog or database. Unfortunately, the coordinate format is not homogeneous among 
them. The more formats will be available, the easier it will be for a researcher.

• in the legend below the probability score plot which says Random Forest should be Early SN score (to be more homogeneous with the other 2 labels).

• it would be nice to add more vertical space between the plots and between plots and axis labels, to avoid lines on top of each other.

It should be possible to choose the field we want to download when performing bulk download (to reduce the bandwidth & reduce complexity for the user). Note that bulk download is not yet publicly available.

It would be better to have tooltips instead of a big blob of information.

We use a very simple algorithm to combine individual science module outputs and assign the final classification. we should at least follow what was published in the paper...! This will mostly affect SN & microlensing.

Action item: update utils.extract_fink_classification with paper criteria.

Cutouts should change when pressing to different points on the light curve. In addition, there should be information on filter and date shown.

There are some minor typos in background text on explorer entry page.

https://community.plotly.com/t/dynamically-add-columns-in-datatable-based-on-selected-dropdown-values/27947/2

We recently capitalised SIMBAD class names for display (#28 ). However, these capitalised names shouldn't be used in the client, as they do not exist in the database!

See #73

We should add color evolution. Two quantities to consider:

# assuming r and g measurement at epoch t
r_minus_g(t) = mag_r(t) - mag_g(t)

# at last epoch with 2 filters
r_minus_g(t=last) = mag_r(t=last) - mag_g(t=last)

# last epoch - first detection
delta(r_minus_g) = r_minus_g(t=last) - r_minus_g(t=first)

Idea: users would make a query, and then being able to upload a table to cross-match data against. See https://dash.plotly.com/datatable/editable + uploading data!

We should have a xmatch page based on the .pixel table. Users would upload a file, and we would gather all corresponding pixels and return results. Current perfs:

neighbour pixels (i.e. 1 object with increasing conesearch radius)

disconnected pixels (N objects with fix radius = 1.5 arcsec)

We should put limit at 1000.

It would be nice to have the Fink probabilities displayed in the right to change when the user clicks on different points in the light curve.

that would correspond to the probabilities of a specific alert (similar to what is already implemented with the cut outs).

gatspy computes internally a score (model.score(period)). Not sure how this can be translated into a chi^2/dof.
Not that we are using

https://github.com/astroML/gatspy/blob/a8f94082a3f27dfe9cb58165707b883bf28d9223/gatspy/periodic/lomb_scargle_multiband.py#L205-L209

def _score(self, periods):
    # Total score is the sum of powers weighted by chi2-normalization
    powers = np.array([model.score(periods) for model in self.models_])
    chi2_0 = np.array([np.sum(model.yw_ ** 2) for model in self.models_])
    return np.dot(chi2_0 / chi2_0.sum(), powers)

so we can easily derive the chi2 using np.sum(model.yw_ ** 2). It remains the number of ddof to determine (nterm_base + nterm_band + period?)

The class computation in the card ID is based on extract_fink_classification_single which calls extract_fink_classification. Unfortunately the call to the last one contains a typo: twice d:mulens_class_1... leading to wrong display!

The ML scores are for valid measurements. Hence, when the lightcurve is in units of Difference Mag, the x-axes of the two plots are not aligned... (it works fine in DC mag or DC flux that do not contain upper limits).

Action item: Force the x-axis to be always the same.

They are supposed to be sorted, but they are not all capitalised, hence they are not fully sorted.

We only show valid measurements for the lightcurve, but recorded non-detections could be useful to e.g. constrain the SN-subtyping and prioritisation for followup (thanks M. Smith!).

Sometimes the chi2/dof is extremely good, but the fitted parameters are not constrained... Need to inspect the formula used

It would be nice to access from the home page to a statistics page that would show the global footprint as in the paper with a summary of alerts processed.

We can easily perform cross-match with SDSS using

http://skyserver.sdss.org/dr13/en/tools/chart/navi.aspx?ra=190.82411&dec=29.68735

We usually have 2 bands for the fit (g & r). If there is only one band, the fit is still performed. For g only, that is fine, but for r only, there is an index mismatch when plotting data measurements (the second band becomes first!) and the fit is not done.

When one band is available, the display is not done with error:

File "/root/miniconda/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/root/miniconda/lib/python3.7/site-packages/dash/dash.py", line 1059, in dispatch
    response.set_data(func(*args, outputs_list=outputs_list))
  File "/root/miniconda/lib/python3.7/site-packages/dash/dash.py", line 994, in add_context
    output_value = func(*args, **kwargs)  # %% callback invoked %%
  File "/home/centos/fink-science-portal/apps/plotting.py", line 738, in plot_mulens
    'x': [convert_jd(t, to='iso') for t in normalised_lightcurves[1][:,0]],
IndexError: list index out of range

See e.g. http://134.158.75.151:24000/ZTF20aavtedl

The current fitter is wrong as it uses PSF mag (magpsf) instead of the DC mag...

The current GitHub link points to fink-broker instead of fink-science-portal!

Thanks @aflp91!

Develop necessary routines for users to access the data via an API

In the explorer page, whenever someone chooses one of the classes in the drop down menu the list should show alerts in decreasing order of probability.

In this case the user would see the most probable candidates first, and build confidence/intuition for subsequent searches.

In explorer, we wait for the user to click on the submit button to trigger an action. The code uses a trick to manage it:

# in explorer.py
# Trigger the query only if the submit button is pressed.
changed_id = [p['prop_id'] for p in dash.callback_context.triggered][0]
if 'submit_query' not in changed_id:
    raise PreventUpdate

But that's needed because we use Inputs. Instead, if we use States, the callback will be triggered only when the button will be pressed. Action:

-@app.callback(
-    Output("table", "children"),
-    [
-        Input("submit_query", "n_clicks"),
-        Input("objectid", "value"),
-        Input("conesearch", "value"),
-        Input('startdate', 'value'),
-        Input('window', 'value'),
-        Input('class-dropdown', 'value')
-    ]
-)
-def construct_table

+@app.callback(
+    Output("table", "children"),
+    [
+        Input("submit_query", "n_clicks"),
+    ],
+    [
+        State("objectid", "value"),
+        State("conesearch", "value"),
+        State('startdate', 'value'),
+        State('window', 'value'),
+        State('class-dropdown', 'value')
+    ]
+)
+def construct_table

and remove the piece of code above.

From time to time, the cutouts are not displayed because the client returns None (despite a valid request)... This problem is usually solved by re-sending the query.

We have a column family a: for annotation. We should, through the API (PUT) and via a secured channel, let users upload their comments. Example user toto would annotate an alert with a:toto -> str

• whenever a give search does not find anything there should be a message: "nothing found"

• It would be nice to add a reset bottom to the search page and a message so the user is informed on the need to re-initiate after a search returning no alerts.

https://github.com/astrocatalogs/OACAPI