joestubbs / endofday Goto Github PK

wrangler is an obvious target for our platform, and the wrangler folks have an independent desire to build docker support for their system.

at the moment, it looks like creating a hadoop requires submitting a slurm job:
https://portal.tacc.utexas.edu/user-guides/wrangler#reservations

Have started the conversation with Amit who plans to get back in touch within the next day or two.

there are currently some issues trying to run eod from within another docker container, since eod itself wants to be launched as a container and the paths are interpreted as host paths. in particular, it would be nice if users could launch eod workflows from their notebook servers launched from jupyterhub.

one approach:

• make an assumption (configurable) that paths in the eod workflow lived in a host mounted volume.
• change endofay.sh to submit workflow executions over a port or socket. a small flask server would be fine, but a zmq socket might be nice and provide queueing, fanout, etc.

The AgaveAsynResponse object uses the access token on the agave client that made the original request to check the status of the response. However, it uses the requests library directly since the history URL is returned as a hyperlink which the python sdk cannot convert into a proper client method. This means that there is no "refresh logic", and if the token has expired on a status check call, endofday fails. Ideally, the sdk would provide a way to "follow links", enabling client applications to leverage the refresh logic already in the sdk.

When running eod using the "--agave" flag, the entire workflow executes remotely as an agave job running on a single VM. However, it would be nice if that process could submit additional jobs for particularly intensive tasks. This won't help at the moment as we only have one VM at eod's disposal. But once we have a pool of VMs this feature could enable better performance.

If a workflow has a task with execution type specified as "agave", and if that workflow is executed with the "--agave" flag, then currently eod will attempt to submit a subsequent job for the task, but the job submission will fail because the remote instance of eod is not configured with agave credentials.

One option would be to create a single oauth client for the remote eod app and pass the access and refresh token into the job. Another option may be to generate a postit for submitting the task job on behalf of the user before submitting the initial eod job, and then pass the postit into the job.

inspiration and possible solution: https://github.com/jhamrick/nbflow

A very common pattern is to spawn a number of processes ("workers") dependent on the "size" of the output from a previous step (e.g., a dynamic version of the pi approximation wf). There are a number of ways this could work from the client/api perspective. Nextflow has several examples. We should first document what we could pull off using doit. There may be some limitations around using doit with multiprocessing and actions defined using closures.

Possible implementation could be writing a generic docker container to wrap the job submission process. Would still require changes to eod since an execution type of "agave" on this process would not mean the same thing as it would for a "normal" docker container.

Will likely also need the docker images.