joestubbs / endofday Goto Github PK
View Code? Open in Web Editor NEWExecute pipelines and other workflows of docker containers.
License: Other
Execute pipelines and other workflows of docker containers.
License: Other
wrangler is an obvious target for our platform, and the wrangler folks have an independent desire to build docker support for their system.
at the moment, it looks like creating a hadoop requires submitting a slurm job:
https://portal.tacc.utexas.edu/user-guides/wrangler#reservations
Have started the conversation with Amit who plans to get back in touch within the next day or two.
there are currently some issues trying to run eod from within another docker container, since eod itself wants to be launched as a container and the paths are interpreted as host paths. in particular, it would be nice if users could launch eod workflows from their notebook servers launched from jupyterhub.
one approach:
The AgaveAsynResponse object uses the access token on the agave client that made the original request to check the status of the response. However, it uses the requests library directly since the history URL is returned as a hyperlink which the python sdk cannot convert into a proper client method. This means that there is no "refresh logic", and if the token has expired on a status check call, endofday fails. Ideally, the sdk would provide a way to "follow links", enabling client applications to leverage the refresh logic already in the sdk.
When running eod using the "--agave" flag, the entire workflow executes remotely as an agave job running on a single VM. However, it would be nice if that process could submit additional jobs for particularly intensive tasks. This won't help at the moment as we only have one VM at eod's disposal. But once we have a pool of VMs this feature could enable better performance.
If a workflow has a task with execution type specified as "agave", and if that workflow is executed with the "--agave" flag, then currently eod will attempt to submit a subsequent job for the task, but the job submission will fail because the remote instance of eod is not configured with agave credentials.
One option would be to create a single oauth client for the remote eod app and pass the access and refresh token into the job. Another option may be to generate a postit for submitting the task job on behalf of the user before submitting the initial eod job, and then pass the postit into the job.
inspiration and possible solution: https://github.com/jhamrick/nbflow
A very common pattern is to spawn a number of processes ("workers") dependent on the "size" of the output from a previous step (e.g., a dynamic version of the pi approximation wf). There are a number of ways this could work from the client/api perspective. Nextflow has several examples. We should first document what we could pull off using doit. There may be some limitations around using doit with multiprocessing and actions defined using closures.
Possible implementation could be writing a generic docker container to wrap the job submission process. Would still require changes to eod since an execution type of "agave" on this process would not mean the same thing as it would for a "normal" docker container.
Will likely also need the docker images.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.