Comments (14)
maybe our @brainstorm has something to aport?... :-P
from ngi_pipeline.
I tried bothering @brainstorm earlier but I suspect he's on vacation or remaining silent for my own good so I learn to read the docs myself.
from ngi_pipeline.
http://adam.herokuapp.com/past/2011/4/1/logs_are_streams_not_files/
p = subprocess.Popen(['ls'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.stdout.readlines()
['Applications\n', 'Desktop\n', 'Documents\n', 'Downloads\n', 'Library\n', 'Mail\n', 'Movies\n', 'Music\n', 'Pictures\n', 'Public\n']
p = subprocess.Popen(['ls', 'blarh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
log.info(p.stderr.readlines())
[2014-08-15 14:06:41.337391] INFO: None: ['ls: blarh: No such file or directory\n']
from ngi_pipeline.
I can't recall precisely now but I think I was hoping for a non-blocking way to do this -- I believe readlines would wait until it got a StopIteration or whatever? I should have been more specific. Though now that I'm thinking about it the logs would get pretty confusing with multiple processes feeding them output simultaneously.
Also I love reading emails from Roman! I promise to have a key-signing party as the next GW talk, although we may have to lie about what's actually happening in the announcement
from ngi_pipeline.
This is were my comment on listening to @guillermo-carrasco and putting
redis in the mix through logbook might ease your concurrency concerns ;)
No need to lie, just have the party at the beggining or end of the talk
like unavoidable (in this case useful) advertisements.
from ngi_pipeline.
Aha I see,
I tried it and it worked :-). What about?:
p = subprocess.Popen(['ls', 'blarh'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
if err:
log.error(err)
else:
log.info(out)
Maybe encapsulated in a log_command_line()
method, so that we don't need to write these conditional again and again, of course. Also, If we use RedisHandler, this will go directly to Logstash.
from ngi_pipeline.
Yeah but p.communicate()
waits until the process terminates I believe and so stalls code execution. Not really feasible for a four-day alignment/variant calling process. I'm thinking we'll need to detach these processes anyway so they don't die when the main python thread exits, and then I suppose we'd need something even more clever to hook them up to the RedisHandler.
from ngi_pipeline.
Also I'm all for RedisHandler and LogBook and all that. We haven't decided against that at all, we just haven't gotten around to it yet.
from ngi_pipeline.
okok, I'll talk about ELK on the next Dev meeting.
from ngi_pipeline.
Gotcha @mariogiov, thanks so much for making me search this one:
http://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python
Can I hire you? :)
from ngi_pipeline.
Oh no, you don't! ;-)
/Per K
2014-08-20 10:20 GMT+02:00 Roman Valls Guimerà [email protected]:
Gotcha @mariogiov https://github.com/mariogiov, thanks so much for
making me search this one:http://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python
Can I hire you? :)
—
Reply to this email directly or view it on GitHub
#14 (comment)
.
Per Kraulis, Ph.D.
Systems Architect, National Genomics Infrastructure (NGI), SciLifeLab.
Dept Biochemistry and Biophysics, Stockholm University.
[email protected], +46 (0)8 5248 1465, http://www.scilifelab.se/
Visiting address: Tomtebodavägen 23A, Karolinska Institutet Science Park,
Solna
Mailing address: SciLifeLab Stockholm, Box 1031, 171 21 Solna, Sweden
from ngi_pipeline.
Hahaha @pekrau xD
Now that I think, isn't this a similar problem you had with genologics @alneberg ?
from ngi_pipeline.
How about this one:
http://www.electricmonk.nl/log/2011/08/14/redirect-stdout-and-stderr-to-a-logger-in-python/
It replaces stdout and stderr with logger instances, should ideally be done within a context manager I guess.
from ngi_pipeline.
Alright the implementation I coded up is here: this follows the general implementation as outlined in the handy stackoverflow post @brainstorm mentioned above.
Guillermo also pointed out that Logbook has built-in support for these kinds of threaded queues, so we'll probably move to that at some point down the line.
from ngi_pipeline.
Related Issues (20)
- Decide how to track genotyping in the piper_ngi code HOT 5
- Track qc jobs in local jobs database
- Move CharonSession code to the charon repo
- Analysis deletion should reset Charon status values HOT 1
- AttributeError: 'exceptions.KeyError' object has no attribute 'msg' HOT 3
- Analysis files from wrong samples removed HOT 5
- flowcells in a2015179 not found HOT 3
- Specific versions in the requirements cause conflicts HOT 1
- Changing delivery status when reanalysing sample
- Running NGI pipeline with multiple flowcells HOT 1
- No files are synced when you start the NGI pipeline with the -k flag.
- Duplication metrics parsing ignores multiple library preps
- Fastq_screen and fastqc overwrites result if fastq files has same name
- Fastq files are not organized under the correct library prep
- Hire More Americans HOT 1
- Rewrite everything in Python3 HOT 1
- For GATK jarfile reliant scripts, set $GATK_HOME as default
- update bam2fastq.sh using latest libs
- Non-existant variable referenced
- If this is a joke about how Swedes pronounce "w"
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ngi_pipeline.