The htcondor-tools from danforthcenter

htcondor-tools's People

Contributors

Stargazers

Watchers

htcondor-tools's Issues

Create a flexible HTCondor job/DAG workflow templating system

Here's my current thinking on how to do this (feel free to suggest alternative solutions):

Instead of editing job files, users would put all their job/DAG workflow settings into a single JSON file, something like this for a single job:

{
    "jobs" : [
        {
            "name" : "job0",
            "universe" : "vanilla",
            "getenv" : "true",
            "accounting_group" : "$ENV(CONDOR_GROUP)",
            "notification" : "complete",
            "request_cpus" : "8",
            "request_memory" : "2G",
            "executable" : "bowtie2",
            "arguments" : "-foo -bar",
            "log" : "$(Cluster).$(Process).bowtie2.log",
            "output" : "$(Cluster).$(Process).bowtie2.out",
            "error" : "$(Cluster).$(Process).bowtie2.error"
        }
    ]
}

and maybe like this for a DAG workflow

{
    "dag" : [
        {
            "name" : "job0",
            "universe" : "vanilla",
            "getenv" : "true",
            "accounting_group" : "$ENV(CONDOR_GROUP)",
            "notification" : "complete",
            "request_cpus" : "8",
            "request_memory" : "2G",
            "executable" : "bowtie2",
            "arguments" : "-foo -bar",
            "log" : "$(Cluster).$(Process).bowtie2.log",
            "output" : "$(Cluster).$(Process).bowtie2.out",
            "error" : "$(Cluster).$(Process).bowtie2.error",
            "child" : "job1"
        },
        {
             "name" : "job1",
             "..." : "..."
        }
    ]
}

Then have a Python script read the JSON and create the necessary job and DAG files. @calizarr also added the idea of using jinja as a templating tool to insert the values from the JSON into job/DAG template files.

Recommend Projects