Comments (11)
Congrats to @emilio-r !
from bactpipe.
We might have to see if it works with path objects instead of file (i.e. reads[0].size())
We could rely on toFile
method for a java Path
object.
Though, I think it'd be best if we encapsulate this functionality into a function or a closure in Groovy.
The final solution would look something like
time computeTime()
from bactpipe.
from bactpipe.
It would be nice if we could decouple the environment specifications from the time specifications somehow. We would encounter the same issue in the gandalf config as well if we had such large input files, and it feels like it could quickly get messy if we need to keep multiple versions of each system profile.
from bactpipe.
Hello team,
I am not sure how the cluster environments work, but perhaps we could explore computing the time limit dynamically?
Something like (pseudocode)
time = 20.m * task.attempt
from bactpipe.
Yeah, that would be an alternative @abhi18av, if one can make some benchmark how much time it needs per 100 Mb input file or something. It should "only" be fastp, shovill and Kraken that will be affected by input file size...
from bactpipe.
I think it should be possible to implement it so it actually reads the size of the input file for the process and computes the time allocation based on that. I think that, perhaps in combination with an extension on failures, would make sense and make it more effective. The alternative that has been proposed already would unnecessarily spend core hours for users with mainly (too) large samples on failed attempts that would then be increased only after failing.
from bactpipe.
Hi team,
I came across this possible solution somewhere else and perhaps we could explore this here
time { 20.m * sample.size() }
from bactpipe.
Neat. I wasn't aware of that functionality!
I guess we would need to make some simple calculation using the sample size to come up with a good multiplier for the time, perhaps also modulo some value so we don't end up with weird allocation request. There should also be a minimum size allocation as well I think :). Can you guys help me come up with something that would make sense? I'm thinking a "normal" sample would result in a time allocation of 20.m
(i.e. the multiplier would be 1), but larger samples would increase in whole integer steps depending on the size of the sample file to 20.m * 2
(medium sized file), 20.m * 3
(large file) etc.
Not sure if sample.size()
would work in our context, as there is no object in the FASTP
process definition called sample
. We might have to see if it works with path
objects instead of file
(i.e. reads[0].size()
), or consider rewriting the process definition slightly to use file
instead of path
.
from bactpipe.
Would someone have time to prototype something around this?
from bactpipe.
I think that the function should look something like this, however I can't think of a way I can test this on my infra.
def computeTime (inputPathObject) {
fileObject = inputPathObject.toFile()
fileSize = fileObject.size()
factor = fileSize % 3
if(factor == 0) {
return 20.m
}
return 20.m * factor
}
This function might need to be adapted based on the test runs.
from bactpipe.
Related Issues (20)
- Update flowchart for new 3.0 options
- Update citation/publish pipeline HOT 25
- The documentation for 3.0 is not automatically updated by sphinx HOT 4
- Update changelog HOT 1
- BACTpipe workflow completed without errors HOT 1
- Shovill-Sendsketch skipping HOT 31
- Limit MultiQC search options HOT 7
- Running with the latest stable nextflow version (20.10.0) does not work HOT 2
- The docs still makes references to database paths HOT 2
- Warn user if signalp is missing if prokka_gram is true HOT 9
- SHOVILL process running with default threads 8 HOT 1
- Add automated testing via Github workflows HOT 1
- shovill.log output as standard HOT 6
- Replace SendSketch with Kraken2 HOT 2
- Invalid process definition -- Unknown keyword 'stub' @ line 20, column 5 HOT 5
- Changelog 3.1 HOT 2
- Prokka process gets mixed inputs HOT 5
- bioconda::prokka environment error HOT 1
- Prokka isn't running if Kraken fails
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bactpipe.