phockett / tmo-dev Goto Github PK

For SACLA workflow:

• Pull run data from logfile (usually GSheet).
• Additional data, e.g. ToFs and config settings usually posted in working channel (Slack, Discord).
• Should configure for:
  • Config files per run/set of runs, and check for this when processing. (But may need to create manually.)
  • Labelling with additional info from logbook.

Basic logbook code example (from GSheet), working for 2023 runs:

# Source: https://stackoverflow.com/questions/19611729/getting-google-spreadsheet-csv-into-a-pandas-dataframe

# Tidy version
import pandas as pd

# Import Logbook
SHEET_ID = '<id number here>'
SHEET_NAME = 'Sheet1'
url = f'https://docs.google.com/spreadsheets/d/{SHEET_ID}/gviz/tq?tqx=out:csv&sheet={SHEET_NAME}'

logbook = pd.read_csv(url, header=0, index_col='Aq ')  # Get data from URL

# Clean up
logbook.index = logbook.index.astype(int)  # Force to int (otherwise float)
logbook.drop(logbook.columns[31:37], axis=1, inplace=True)  # Drop extra cols at the end
logbook.columns = logbook.columns.str.rstrip()   # Clean col name strings

# Display current
logbook.loc[runs]

Quick and dirty params setting in notebook via magic, e.g. %run -i 'aq189_params.py' where params file is just python vars from other sources.

To implement - see epsman code for method using JupyterRunner:

https://pypi.org/project/jupyter-runner/
https://github.com/omar-masmoudi/jupyter-runner
https://pythonhosted.org/jupyter_runner/

This will be particularly good for slow internet connection cases, where large figures can time-out in notebook.

In LW06 tests (runs 81 - 87) seeing a lot of hits with only one "good" dimension, and the other at max - is this a binning issue, or "real" artifact (likely reflection current)? To look into - probably need to read more on the instrument first.

• General 1D & 2D histograms
• Discrete histograms from another data axis
• Raw data plots for testing/checking
• Stacked plotting for multiple data types/dims
• Polar plots & image ROI analysis
• List data types & detectors at file IO

Currently missing some basics for image analysis:

• ROI setting & extraction.
• ROI finding.
• Polar coord conversion.
• Circle finding.

Some of this already in psana...?

Currently set for single filter set only in tmoDataBase class, but wrapped for multi-level filters in setFilter() method and VMI class. Should be unified architecture here (and back-propagate to early basic plotting functions), and more transparently applied.

May want a different branch for this, since it will likely break things.