Logging of subprocess jobs to Logbook about ngi_pipeline HOT 14 CLOSED

maybe our @brainstorm has something to aport?... :-P

from ngi_pipeline.

I tried bothering @brainstorm earlier but I suspect he's on vacation or remaining silent for my own good so I learn to read the docs myself.

from ngi_pipeline.

http://adam.herokuapp.com/past/2011/4/1/logs_are_streams_not_files/

p = subprocess.Popen(['ls'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.stdout.readlines()
['Applications\n', 'Desktop\n', 'Documents\n', 'Downloads\n', 'Library\n', 'Mail\n', 'Movies\n', 'Music\n', 'Pictures\n', 'Public\n']


p = subprocess.Popen(['ls', 'blarh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
log.info(p.stderr.readlines())
[2014-08-15 14:06:41.337391] INFO: None: ['ls: blarh: No such file or directory\n'] 

from ngi_pipeline.

I can't recall precisely now but I think I was hoping for a non-blocking way to do this -- I believe readlines would wait until it got a StopIteration or whatever? I should have been more specific. Though now that I'm thinking about it the logs would get pretty confusing with multiple processes feeding them output simultaneously.

Also I love reading emails from Roman! I promise to have a key-signing party as the next GW talk, although we may have to lie about what's actually happening in the announcement

from ngi_pipeline.

This is were my comment on listening to @guillermo-carrasco and putting
redis in the mix through logbook might ease your concurrency concerns ;)

No need to lie, just have the party at the beggining or end of the talk
like unavoidable (in this case useful) advertisements.

from ngi_pipeline.

Aha I see,

I tried it and it worked :-). What about?:

p = subprocess.Popen(['ls', 'blarh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
if err:
    log.error(err)
else:
    log.info(out)

Maybe encapsulated in a log_command_line() method, so that we don't need to write these conditional again and again, of course. Also, If we use RedisHandler, this will go directly to Logstash.

from ngi_pipeline.

Yeah but p.communicate() waits until the process terminates I believe and so stalls code execution. Not really feasible for a four-day alignment/variant calling process. I'm thinking we'll need to detach these processes anyway so they don't die when the main python thread exits, and then I suppose we'd need something even more clever to hook them up to the RedisHandler.

from ngi_pipeline.

Also I'm all for RedisHandler and LogBook and all that. We haven't decided against that at all, we just haven't gotten around to it yet.

from ngi_pipeline.

okok, I'll talk about ELK on the next Dev meeting.

from ngi_pipeline.

Gotcha @mariogiov, thanks so much for making me search this one:

http://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python

Can I hire you? :)

from ngi_pipeline.

Oh no, you don't! ;-)
/Per K

2014-08-20 10:20 GMT+02:00 Roman Valls Guimerà [email protected]:

Gotcha @mariogiov https://github.com/mariogiov, thanks so much for
making me search this one:

http://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python

Can I hire you? :)

—
Reply to this email directly or view it on GitHub
#14 (comment)
.

Per Kraulis, Ph.D.
Systems Architect, National Genomics Infrastructure (NGI), SciLifeLab.
Dept Biochemistry and Biophysics, Stockholm University.
[email protected], +46 (0)8 5248 1465, http://www.scilifelab.se/
Visiting address: Tomtebodavägen 23A, Karolinska Institutet Science Park,
Solna
Mailing address: SciLifeLab Stockholm, Box 1031, 171 21 Solna, Sweden

from ngi_pipeline.

Hahaha @pekrau xD

Now that I think, isn't this a similar problem you had with genologics @alneberg ?

from ngi_pipeline.

How about this one:

http://www.electricmonk.nl/log/2011/08/14/redirect-stdout-and-stderr-to-a-logger-in-python/

It replaces stdout and stderr with logger instances, should ideally be done within a context manager I guess.

from ngi_pipeline.

Alright the implementation I coded up is here: this follows the general implementation as outlined in the handy stackoverflow post @brainstorm mentioned above.
Guillermo also pointed out that Logbook has built-in support for these kinds of threaded queues, so we'll probably move to that at some point down the line.

from ngi_pipeline.