vixus0 / awful-tools Goto Github PK

Being able to specify the directory in which a job should be run would definitely be helpful.

I thought I would start an issue on this so we have some sort of trace of what we talked about.

Basically, we would create a Daemon much like the one in http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/ , overloading run with something like:

def run():
    while True:
        cmd_string = pid.readline()
        parse_and_run(cmd_string)

Assuming the Daemon class runs well enough, I expect the only complicated part will be the parsing. A viable option might be to use the argparse module. The daemon could then be passed command strings that look much like shell commands, eg. qsub "sleep 20" --name="boring" --nprocs=2. Argparse would be able to parse that easily, and we could use the subparsers for the different commands.

Anyway, this seems like a good way to avoid the overhead of having to write our own parser.

Any thoughts?

I think it would be helpful to allow IO redirection. At the moment, stdout is pushed to /dev/null and stderr to a /tmp/mdrunr.log file.

These are good defaults, I think, but allowing the user to specify a file name / stream to write to might be useful?

I'm happy to code this up.

Need a way to delete jobs from the queue.

So, largely due to my own lack of foresight, at the moment you have to have Pyro even if you just want to run the queue interactively, without ever starting the server (the so-called "easy way" in "usage.md").

This doesn't make sense. For example, we don't have Pyro on our work computers in physics, and I doubt the COs would be particularly keen to install it. However, having a queuing system would still be useful, even if jobs needed to be submitted from within the same python shell that spawned the queue.

I think it would be easy enough to move the components that need Pyro out of mdrunr and mdclient into some separate module (say mdremote).

But beyond this, we could consider re-designing the module structure. We could have a top-level python module that provides an API to the rest of the code. I would like to be able to open a python interpreter and, say, type the following:

>>> import awful
>>> awful.start_queue(nprocs=2)
>>> awful.add_job("job-name", "job-commands", nprocs=2, ...)
>>> awful.list_jobs()
[ some readable format rather than the current dictionary mess ]

... somewhat like pylab provides an API to matplotlib.

If this awful module doesn't import Pyro (and relies on modules that don't import Pyro), we can effectively decouple the parts that use Pyro (exposed in the shell-interface) and the parts that don't (exposed in the awful python module).

Because this requires a bit of organisation, it might be good to have a small code-freeze during the implementation. Otherwise, the resultant merges might end up being horrible.

I'm happy coding that up.

It would be good to have a shell interface, instead of having to open a python prompt to define a job.

For example, assuming a queuing server is running:

$ awful qsub "rm -rf /" --name="i-am-a-moron" --nprocs=2
$ awful qlist