MergePeakCoordinates takes long time！ about sierra HOT 9 CLOSED

Sorry for the slow response. I wouldn't expect three datasets to take a long time to merge. How many peaks are in each of the files?

from sierra.

Hi @rj-patrick. We are facing a similar issue. With a total of 37 samples, the merge step has been running for three weeks and not finished yet. There are > 2.8 million peaks across all samples. "ncores" is set to 50. Do you have any insights regarding our situation? Thanks.

from sierra.

Hi @alanlamsiu,
Thanks for that. Do you have output from the MergePeaks function? It would be helpful to know where it has gotten up to. Three weeks is too long, but this is a much larger dataset than what we've tested it on. There's a couple of potential issues, but output would be the best for diagnosing the issue.
Cheers,
Ralph

from sierra.

Thanks @rj-patrick for getting back.
I can see the size of the log file keeps increasing, which has now 27,482,128 lines, yet the output specified by "output.file" is not generated. Based on the log file, I guess that "internal peak merging" is done for all 37 samples, while the messages "Comparing peaks from to remaining data-sets" are already printed. For computing resources, the average memory usage is 4.5Gb and peak CPU usage is 39.05.

I recalled that when I ran a test using five samples, with a total of > 490 thousand peaks, it took less than a week to complete.

Please let me know if you need more details.

Thanks.

from sierra.

Thanks, to clarify, the message "Comparing peaks from [DATASET X] to remaining data-sets" is printed for how many of the 37 datasets?

from sierra.

The message has been printed for all 37 datasets.

from sierra.

Thanks. I think I know where the problem is. There is a final step where peaks are iteratively checked for merging, but with your dataset, my guess is it's getting stuck in a loop. Perhaps the best thing at this point would be to merge whatever is remaining, but for now I've set a limit on the number of iterations to go through. Pull the latest update and see if that fixes the issue.

from sierra.

Thanks @rj-patrick for the fix. I tried the updated version. The run was finished within a few days. I think I am good to go with it.

from sierra.

Thank you for solving the problem!

from sierra.