phockett / epsman Goto Github PK

TODO:

• Decorators or general util methods for this.
• Cases for setting attribs from passed values with/without overwrite already in ._util.setAttribute() and setAttributesFromDict(), but crude. (See also ePSproc plotting libs - partially finished some similar stuff).

Needs doing! Much of this is now relatively mature (at least in terms of local use), but needs consolidating and documenting properly.

Basics working (tested on Bemo). To do:

• Fix git push verification.
• Nbsphinx maths issues - seems to be problem with eqnarray (although wasn't for ePSproc pages).
• Test on other machines.

General logging & print routines to implement. Mainly bespoke so far.

Now running OK on AntonJr, but issue with data save...? May be due to multiple HDF5 options?

See, e.g., https://gist.github.com/nfaggian/8644080

Running OK on Jake.

Testing for

• ABCO
• CH3I

To do:

• Better plotting for matrix elements.
• Header info: Ehv, main title.
• Handling for cases with extra E on list.

Basics now implemented in epsman.elecStructure.EShandler(), using CCLIB functionality on the backend.

Current status:

• Read Gamess log files OK.
• Files from writeMoldenFile2006() are working with ePS (tested for N2O test file).
• Files from reformatMoldenFile() are NOT working due to line-endings issues.

To do:

• Finish documentation.
• Finish demo notebook.
• Further testing.
• Better logging and error checking, esp. resetting E and coords for failed runs.
• Tidy up "extra" parameters. Working OK as of e11e7f9 (23/01/24), but currently have some params existing for self.paramName, self.params[paramName] and self.g.paramName, which is messy and unnecessary.
• Further tests for self.buildES() - OK from scratch, but may inconsistently reset only some params when rerunning.

In the current implementation, running and inspecting notebooks remotely, via Fabric commands, is done via shell scripts.

At least two issues with this currently:

• Requires Conda env. setting in shell script, which may fail if shell is not set correctly, and if env is missing (currently hard-coded in scripts). See shell/conda_note.txt for details.
• Scripts currently default to eps/scripts2019, not epsman/shell.

Also... Conda settings were working for notebook processing, but not working for basic script testing - currently unclear why. Has an update broken something?

Current REST API has 100MB file size limit. Implemented hacky multi-part zip file creation on source machine via system zip to deal with this (splitArchFiles()), but not ideal.

Better solutions:

• Split by files into reasonably groups.
• Avoid using python zip implementation
• Use different archive format? Tar?

At some point Zenodo will also be supporting larger files...

Currently packaging (to archive) all files matching glob/re expressions. This may, or may not, include additional post-processing files.

Should be consistent here: either include all post-processing files, or keep archive for ePS raw files only.

Quite a few legacy machine-specific files to fix:

• jobs/ePS_batch_nohup.sh
• scripts2019/machine.conf
• scripts2019/[template scripts]

Should consolidate to "local settings" and/or autogenerate for machine as required.

Fix issue with remote/local titles and consistency - at the moment pkgOverride() allows manual title correction, this is propagated to Zenodo but not notebook header. Ugh.

To add:

• Readme.txt with some basic instructions, details and info.
• .json file with full details.
• License info (also to add to HTML code gen).
• Fix remote run with Nohup - currently drops on errors? Adding file checks should fix this.

Some inconsistencies and things to fix/tidy-up

• File name schema currently set in shell scripts, should be passed for transparency.
• Consolidate scripts to repo.
• Propagate naming schema to other tools, e.g. tidyJobs() etc.
• Update job templating methods, currently python > conf file > shell script engine. This works, but is not very flexible. Maybe Jinja could simplify this?

Current implementation (tries) to run shell script via Fabric, but this has issues with detached/background processes sometimes (see, e.g., start a background process with nohup using fabric and ssh TTY behaviour notes.

Todo:

• Fix current implementations? (As per start a background process with nohup using fabric, see Fabric user notes for more discussion). Now fixed in c7e91e4.
• Better/more robust implementations using local python process as manager? (Then run locally with subprocess.run() and/or os.fork() or similar - see, e.g., autonotebook code, esp triggerNotebook().)
• Full job management routines on ePS machine? Python or via another toolset/manager/queue system, e.g. ts, CRON, supervisord, SLURM.... - see Fabric user notes for more discussion.
• Wrapper for arb commands via Fabric.
• Set & propagate custom run command for flexibility.
• Use Docker? See https://github.com/phockett/open-photoionization-docker-stacks

Also/to debug:

• Fix path setting for host=localhost only (i.e. running ePS on local machine). This currently may not be set, see

  epsman/epsman/epsJob.py

  Line 65 in 09782ca

  # TODO: sometimes have bugs due to presetting localhost here, should consolidate with other init codes.

Currently set in .json file per job and batch (directory).

Plan to process these globally and put in a DB for better job tracking.

Currently in progress, sort-of working for some dev/test cases, but still painful (as of 10/06/21, 04469f5).

Outstanding items to fix:

• Very ugly... need to finish merging EShandler and ESjob class, currently have to keep passing former to latter methods, even though it's already set to self.esData (may want to rename!).
• Symmetry code still needs manual fixing for generating full sym list with ePS syms (and also for general use).
• Still have some bug with writeGen() code, which throws file IO errors on first run... missing paths until self.createJobDirTree() is run?
• Manually setting up job each time, still, did this not already get fixed? It's a pain. symTest() method is working, so should just wrap single job in the same manner?

Improvements to implement:

• Wrapping for multiple ionization channels and/or electronic structure files cases. Currently sets one file and one channel per class object. May want to wrap in another class, or set items to dictionaries to allow multiple cases in the current class.
• More tests & debug for job creation ordering
  • as of 14/02/22 working for ESjob with files, e.g. job = ESjob(fileName = 'xe_SPKrATZP_rel.log', fileBase = testFilePath), OR job tree, but not mixed case (e.g. job = ESjob(mol = 'Xe', fileName = 'xe_SPKrATZP_rel.log', fileBase = testFilePath) fails)? Due to setJob and/or setPaths routines, but ends up a bit circular, cf. default paths and debugging.
  • After file import, job.buildePSjob(channel = 18) is OK, and sets paths.
• Master dictionary to collect all job settings. Currently set partly as class attributes (main epsJob settings) and/or sub-classes and/or dictionaries/lists. For easy back-compatibility should just push to dicts at creation, then point attribs to these, although this may get messy. Alternatively stick with attribs and just parse relevant properties to dict with a separate function?

Bugs:

• Orbital indexing currently inconsistent, with 0-indexing for labels. FIXED in 27f16bf

Some issues when reinit jobs in notebook after close/crash. Currently need to reset job settings locally before resuming, but should have method for chkpoint file or otherwise recovering previous status. (For packaging and web builds this is OK, done via json file, but not for initial job run & tidy-up.)

Currently requires (usually):

• Init connection
• genFile path
• Elist