ioam / lancet Goto Github PK

It would be really helpful if jobs launched via GridEngine could list the job number in the job name first, such that it is viewable via qstat. Currently although it is included in the job name, qstat usually truncates it.

I'd really appreciate this change because I will often launch large parameter searches with jobs that can run up to 48 hours and hog 8 CPU's and realize relatively early that certain parameters won't work out. Rather than wasting the resources I then cancel these jobs selectively.

In the past I have had to work out which job-IDs correspond to which parameters by looking at the first or last job and counting from there. This is obviously insane (and error-prone) when there are >100 jobs and becomes even more difficult when some jobs have already completed. I'd therefore hugely appreciate if we could prepend the job number to the job name submitted to qsub, making identifying specific jobs much easier.

Currently the job name is built like this:

job_name = "%s_%s_tid_%d" % (self.batch_name, self.job_timestamp, tid)

Producing this qstat output:

job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
7817272 0.00941 divisive_g s1048519     r     07/22/2014 20:10:03 ecdf@eddie491                      2        
7817306 0.00941 divisive_g s1048519     t     07/22/2014 23:43:15 ecdf@eddie320                      2 
....

I suggest changing it to this:

job_name = "t%d_%s_%s" % (tid, self.batch_name, self.job_timestamp)

Output:

job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
7817272 0.00941 t0_divisiv s1048519     r     07/22/2014 20:10:03 ecdf@eddie491                      2        
7817306 0.00941 t1_divisiv s1048519     t     07/22/2014 23:43:15 ecdf@eddie320                      2
....

python setup.py develop does not work out of the box with Python-2.7.x and Lancet.

While reading your interesting paper and experimenting with lancet, I tried the following:

from lancet import Args, Range
params = Args(arg1=1.0) * Range('arg2', 1, 3, steps=3)
ap = params - Args(arg1=1.0, arg2=2.0)

But this doesn't work. I can imagine scenarios where this is just as natural as extending a parameter space with addition. Was wondering why this was not supported.

Launcher currently re-launches everything when called again. Instead Launcher should only run previous runs if the earlier invocation failed (with a non-zero exit status) or explicitly asked to force-rerun. This would also make it easy to change the parameters and add additional parameters ranges to explore without having to worry about re-running everything.

The current git HEAD seems to have trouble running the basic example given on the lancet homepage:

>>> import lancet
>>> integers = lancet.Range('integer', 100, 115)
RuntimeError: maximum recursion depth exceeded while calling a Python object

/Users/user/Code/lancet/lancet/core.pyc in __len__(self)
    431         return DataFrame(self.specs) if DataFrame else "Pandas not available"
    432 
--> 433     def __len__(self): return len(self.specs)
    434 
    435 

/usr/local/lib/python2.7/site-packages/param/parameterized.pyc in __get__(self, obj, objtype)
    372         # Parameterized class); objtype is never None
    373 
--> 374         if not obj:
    375             result = self.default
    376         else:

/Users/user/Code/lancet/lancet/core.pyc in __len__(self)
    431         return DataFrame(self.specs) if DataFrame else "Pandas not available"
    432 
--> 433     def __len__(self): return len(self.specs)
    434 
    435 

/usr/local/lib/python2.7/site-packages/param/parameterized.pyc in __get__(self, obj, objtype)
    372         # Parameterized class); objtype is never None
    373 
--> 374         if not obj:
    375             result = self.default
    376         else:

1. This would make it easy to get back the stdout/stderr along with any files generated without having to write a lot of code.
2. The returned information should essentially be a combination of what Log and FilePattern provide.

For example:

launcher = Launcher(...)
output = launcher()
results = [float(stdout.read()) for stdout in output.stdout] 

Ideally the output should contain enough information that it should be very easy to generate HoloMaps or pandas dataframes from this output without having to write a lot of code.

Now that noarch is generally well supported, we should make this pure-Python package be a noarch conda package to avoid ultra-confusing problems like holoviz/datashader#532 due to having no Lancet package for a particular Python version.