awsdocs / amazon-omics-developer-guide Goto Github PK

Hello! I really don't know if this is the right place to put it, but there are multiple things that do not work as expected with AWS Omics. I cannot find proper information in the documentation to debug and my logs in CloudWatch are of no help because no error is marked. FYI, I am using Nextflow workflows. Some of the main things I see are:

• If a task fails before starting, there is no way to see the error messages. I see the run as failed, but no information about why it failed appears in the CloudWatch logs. If I start the run from the command line, sometimes I do get a more explicit error message, but often I don't. It makes it very difficult to debug.
• Main thing... for some reason when I run a complex workflow with multiple samples running in parallel, I cannot find data for all my samples but only for one! My run had two samples with multiple steps (QC, alignment, etc) and ends up with a MultiQC step where I collect information from all samples. I see that every step is run properly for both samples and all tasks and the full run end successfully. However, when I look at my S3 bucket, only the results for one of the samples are present! (not always the same sample). I then downloaded the multiqc report. This report should contain information for both samples (if they ran properly) and it does! So, both samples were indeed processed successfully! However, only one of them gets copied to my S3 bucket. This behavior is not what one should expect. I would like to see these type of examples in the documentation (I mean a more complex workflow with 'branches' per sample.

I think it would also help to have a deeper explanation on what happens behind the scenes. I mean, I assume some EC2 instances running each of my tasks get started (with AWS Batch). However, how is the data transferred to my S3 bucket? Is the data transferred from the local ephemeral storage from the EC2 instance to S3? Or is there like an EBS attached?

I hope I was explicit enough. Thank you!