Tibanna Group Jobs Upload Files in Apparently Arbitrary Order about tibanna HOT 3 OPEN

Upon further investigation, this seems to be a result of Grouped jobs not being time stamped correctly when uploading to AWS. Tibanna wants to rerun the racon rule that produced "penny_1326.flye.racon3.fasta" because "penny_1326.flye.racon3.fasta" happened to get uploaded to aws before "penny_1326.flye.racon2.fasta" and "penny_1326.flye.racon2.fasta" is an input for the rule that produces "penny_1326.flye.racon3.fasta". This was only possible because I was executing all racon jobs as a single group in my earlier execution which causes all three racon files to apparently be uploaded in an arbitrary order.

The user level fix is to edit the time stamps in AWS, and clear out the ".snakemake" cache. Long term sollution is to make tibanna upload output files in the order defined by the DAG representing the group job being executed. Not being super familiar with the source code of snakemake or tibanna, I'm not certain this is an easy or even possible update to make

from tibanna.

@nhartwic Thank you for reporting this. The best fix would be to preserve time stamp for the output files but as far as I know AWS S3 does not provide that option. The output files from a given instance ('group') can be sorted before being uploaded to S3, but that still would not guarantee all the out files are uploaded in the correct order if there are multiple instances running concurrently (parallel independent group jobs). I'll see if I can at least get the files ordered within a group.

from tibanna.

Sounds good. As long as output files for each group is ordered correctly, that is probably sufficient as any dependencies of the group must have been uploaded prior to the groups execution and any downstream products must get uploaded after, just due to the way groups get spawned. The only potential errors would be if multiple partial runs were being performed in which the dag topology meaningfully changes, but I'd argue that in such cases, the rules themselves are the problem and Snakemake in general can't resolve the issue. As an example, imagine run 1 has structure "rule A -> Rule B" and run 2 has structure "rule B -> rule A -> rule C". This example should probably never happen and ought to be avoided by workflow writers.

from tibanna.