phockett / epsman Goto Github PK
View Code? Open in Web Editor NEWManagement tools for ePolyScat job generation & execution.
Management tools for ePolyScat job generation & execution.
TODO:
._util.setAttribute()
and setAttributesFromDict()
, but crude. (See also ePSproc plotting libs - partially finished some similar stuff).Needs doing! Much of this is now relatively mature (at least in terms of local use), but needs consolidating and documenting properly.
Basics working (tested on Bemo). To do:
General logging & print routines to implement. Mainly bespoke so far.
Now running OK on AntonJr, but issue with data save...? May be due to multiple HDF5 options?
See, e.g., https://gist.github.com/nfaggian/8644080
Running OK on Jake.
Testing for
To do:
Basics now implemented in epsman.elecStructure.EShandler(), using CCLIB functionality on the backend.
Current status:
To do:
self.paramName
, self.params[paramName]
and self.g.paramName
, which is messy and unnecessary.self.buildES()
- OK from scratch, but may inconsistently reset only some params when rerunning.In the current implementation, running and inspecting notebooks remotely, via Fabric commands, is done via shell scripts.
At least two issues with this currently:
Also... Conda settings were working for notebook processing, but not working for basic script testing - currently unclear why. Has an update broken something?
Current REST API has 100MB file size limit. Implemented hacky multi-part zip file creation on source machine via system zip to deal with this (splitArchFiles()), but not ideal.
Better solutions:
At some point Zenodo will also be supporting larger files...
Currently packaging (to archive) all files matching glob/re expressions. This may, or may not, include additional post-processing files.
Should be consistent here: either include all post-processing files, or keep archive for ePS raw files only.
Quite a few legacy machine-specific files to fix:
Should consolidate to "local settings" and/or autogenerate for machine as required.
Fix issue with remote/local titles and consistency - at the moment pkgOverride() allows manual title correction, this is propagated to Zenodo but not notebook header. Ugh.
To add:
Some inconsistencies and things to fix/tidy-up
Current implementation (tries) to run shell script via Fabric, but this has issues with detached/background processes sometimes (see, e.g., start a background process with nohup using fabric and ssh TTY behaviour notes.
Todo:
Also/to debug:
Line 65 in 09782ca
Currently set in .json file per job and batch (directory).
Plan to process these globally and put in a DB for better job tracking.
Currently in progress, sort-of working for some dev/test cases, but still painful (as of 10/06/21, 04469f5).
Outstanding items to fix:
Improvements to implement:
job = ESjob(fileName = 'xe_SPKrATZP_rel.log', fileBase = testFilePath)
, OR job tree, but not mixed case (e.g. job = ESjob(mol = 'Xe', fileName = 'xe_SPKrATZP_rel.log', fileBase = testFilePath)
fails)? Due to setJob and/or setPaths routines, but ends up a bit circular, cf. default paths and debugging.job.buildePSjob(channel = 18)
is OK, and sets paths.Bugs:
Some issues when reinit jobs in notebook after close/crash. Currently need to reset job settings locally before resuming, but should have method for chkpoint file or otherwise recovering previous status. (For packaging and web builds this is OK, done via json file, but not for initial job run & tidy-up.)
Currently requires (usually):
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.