Put FastQ File Splitting in a Dedicated Process about hic HOT 8 OPEN

I'd like to ditto that request. 99% of the time taken up by the pipeline when I run it is taken up in this step.

from hic.

I agree that this could be a very nice feature indeed. Actually, it might be useful for several nf-core pipelines. Will try to discuss this point with the nf-core core members ;)

from hic.

To give some further context, when inputting very large FASTQ files, the .splitFastq implementation currently being used will in INPUT_CHECK will:

1. Read and decompress every imported FASTQ.gz
2. Parse the FASTQ reads and split into chunks of reads
3. Recompress the chunks
4. Write the chunks to scratch storage

Given these currently happen on the device where the engine itself is running, and the very large size of FASTQs for these type of data, this ends up taking a long time. Increasing the workflow engine instance resources doesn't help much since the actions are not parallelized regardless, so additional CPUs don't provide much help.

Again, very much appreciate the time and energy you've put into this pipeline, and hopeful it can be applied to larger-scale data more easily with this change. Thanks!

from hic.

Hi @ieres-amgen - Can you explain a little more how you currently implement this .splitFastq ?
I'm new to this workflow and tried setting "--split-fastq TRUE" in my yaml file but the workflow didn't move forward for me.

To give some further context, when inputting very large FASTQ files, the .splitFastq implementation currently being used will in INPUT_CHECK will:

1. Read and decompress every imported FASTQ.gz
2. Parse the FASTQ reads and split into chunks of reads
3. Recompress the chunks
4. Write the chunks to scratch storage

Given these currently happen on the device where the engine itself is running, and the very large size of FASTQs for these type of data, this ends up taking a long time. Increasing the workflow engine instance resources doesn't help much since the actions are not parallelized regardless, so additional CPUs don't provide much help.

Again, very much appreciate the time and energy you've put into this pipeline, and hopeful it can be applied to larger-scale data more easily with this change. Thanks!

from hic.

@Krithika-Bhuvan, this thread is about changing the general way split_fastq is implemented, not troubleshooting its current functionality.

Based on what you wrote, my guess is that you encounter issues because you're passing the wrong name for the parameter (it's "split_fastq" instead of "split-fastq"), but I can't say for certain without more information about the errors you see. I highly recommend heading over to the nf-core slack and seeking guidance there if you encounter further issues, that is the best place for troubleshooting.

from hic.

Thank you for the explanation @ieres-amgen. Is there anyway to check if the splitting process is working or not ? I could not locate any log files related to this during my test so I can't tell if that is working or not (I used the right tags in the yaml file). Any suggestions on where to look would be helpful. Thank you !

from hic.

No, to my knowledge, part of the disadvantage of not having this in a dedicated process is that there is no way to check on progress.

from hic.

I'm new to the pipeline so I've been wondering If I was doing something wrong. Thank you for confirming this ! Its just a waiting game now.

from hic.