phockett / autonotebook Goto Github PK

Now mostly implemented in 8385bd5

But a total mess.

• Poll for .proc files.
• Update settings from .proc file (NOTE: THIS CURRENTLY SKIPS PATH UPDATES)
• Arb arg passing to notebook via
  • ENV settings (limited to string type).
  • JSON settings file.

TODO:

• Fix path updates. See

  autonotebook/automaton/autoProc.py

  Line 252 in 8385bd5

  if updateFlag and not resetFlag:

• Subdir output broken for .proc file case? No idea why - might be Path() issue?
• Add demo notebooks with arb arg passing.
• Formalise JSON method as better option, and use this in general. Handy for keeping track of settings as notebook sidecar file, should also be flexible enough to use with, e.g., Jupyter Runner or Papermill.
• Tidy up - should remove repeated trigger path updates and related to a function. See also #2

For consideration...

• Pull data for current file from elog & include in processed notebook.
• Pull data from all files from elog, and auto process groups.
• Use elog to trigger builds? This might be useful if, e.g., data on a remote server. (Cf. psana methods.)
• Write processing info (data verification, file links etc.) to elog.

For SACLA data this is via Google Sheets, e.g. use gspread.

Currently using v2, slackclient, should move to v3 slack_sdk
https://slack.dev/python-slack-sdk/v3-migration/index.html

Sort out general path spaghetti, esp. with subdirs and subfunctions.

• Should rationalise to a single global get/set to avoid definition changes.
• Persist some file-based paths to allow for easy reuse.

See, e.g. https://github.com/prehensilecode/python-daemon-example

Currently also possible with various external methods, e.g., nohup, screen or Docker.

• Implement proper handling of triggered processes, including comms for generate file triggers (rather than requiring them to be picked up by dir watcher process).
• Implement queue for larger jobs/multiple dataset processing cases.

Testing in new branches:

• mergeTest, main + multiRunDev + brittonm
• mergeTest2, main + brittonm + multiRunDev

Both merged OK, not sure if some cross-functionality may be broken, and possibly will depend on ordering (?).

See https://github.com/phockett/autonotebook/pulls?q=is%3Apr+is%3Aclosed

Currently implement a basic dir poll, with Glob + dictionary comparisons (as sets).

Lots of options for more sophisticated methods. Should also integrate with proper event queue/trigger routines.

• fswatch (cross platform)
• Watchdog (cross platform) (lots of examples online, e.g. threaded queue example here, and basic demo)
• inotifyx (linux) Python binding for inotify. (Not lots of similar packages/project, e.g. pyinotify, although that seems to be not recommended - although this is old.)
• Qt FileSystemWatcher