nf-core / marsseq Goto Github PK
View Code? Open in Web Editor NEWMARS-seq v2 pre-processing pipeline with velocity
Home Page: https://nf-co.re/marsseq
License: MIT License
marsseq's People
Contributors
Stargazers
Watchers
Forkers
brickmanlabmarsseq's Issues
AWS full testing ignores staging of s3 files
Description of the bug
Currently, when testing pipeline on AWS, only reference building is executed. The issues is that in the execution, the PREPARE_PIPELINE module can't access the s3 bucket files, they are not staged (dowloaded as FASTQ files, not sure why). Here is a link to tower execution.
You can see that as the docs state, the s3 link is converted to /nf-core/test-datasets/marsseq/testdata/SB26/, however, the files are not staged and therefore the pipeline fails.
I've checked that the files are present in the bucket.
$ aws --no-sign-request s3 ls s3://ngi-igenomes/test-data/marsseq/
2023-06-06 16:10:32 312138696 Undetermined_S0_L001_R1_001.fastq.gz
2023-06-06 16:10:32 133363364 Undetermined_S0_L001_R2_001.fastq.gz
2023-06-06 16:10:32 9516 amp_batches.xlsx
2023-06-06 16:10:32 567 design.csv
2023-06-06 16:10:32 8737 seq_batches.xlsx
2023-06-06 16:10:32 180347 wells_cells.xlsxI can't test this with internal dangpu cluster because I don't have AWS credentials and whenever I try to execute the pipeline it will timeout, probably because of the missing credentials. I think this could be associated with how the bucket is public/not public, see nextflow/issues/3281.
Possible solution
Check if the files are coming from s3 bucket, if yes, then convert the s3 to normal url format and download the files with WGET module.
http://test-bucket.s3.amazonaws.com/test-folder/test-file.txt
http://ngi-igenomes.s3.amazonaws.com/test-data/marsseq/wells_cells.xlsx
The question still remains, will the reference folder be staged correctly, or will it fail as well?
Command used and terminal output
$ nextflow run . -profile ku_sund_dangpu --genome mm10 --outdir ./results --input /home/fdb589/Brickman/projects/proks_et_al_2022/pipeline/SB26/design_remote.csv
```csv
batch,fastq_1,fastq_2,amp_batches,seq_batches,well_cells
SB26,s3://ngi-igenomes/test-data/marsseq/Undetermined_S0_L001_R1_001.fastq.gz,s3://ngi-igenomes/test-data/marsseq/Undetermined_S0_L001_R2_001.fastq.gz,s3://ngi-igenomes/test-data/marsseq/amp_batches.xlsx,s3://ngi-igenomes/test-data/marsseq/seq_batches.xlsx,s3://ngi-igenomes/test-data/marsseq/wells_cells.xlsx
### Relevant files
```log
Jun-21 22:00:52.235 [main] DEBUG nextflow.cli.Launcher - $> nextflow run . -profile ku_sund_dangpu --genome mm10 --outdir ./results --input /home/fdb589/Brickman/projects/proks_et_al_2022/pipeline/SB26/design_remote.csv
Jun-21 22:00:52.448 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 23.04.1
Jun-21 22:00:52.467 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/scratch/Brickman/pipelines/nfcore/plugins; core-plugins: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected]
Jun-21 22:00:52.477 [main] INFO org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jun-21 22:00:52.478 [main] INFO org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jun-21 22:00:52.481 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Jun-21 22:00:52.490 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Jun-21 22:00:53.367 [main] DEBUG nextflow.config.ConfigBuilder - Found config local: /scratch/Brickman/pipelines/nf-core-marsseq/nextflow.config
Jun-21 22:00:53.367 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /scratch/Brickman/pipelines/nf-core-marsseq/nextflow.config
Jun-21 22:00:53.377 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: `ku_sund_dangpu`
Jun-21 22:00:54.362 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [cfc_dev, ifb_core, denbi_qbic, alice, mjolnir_globe, uppmax, uge, rosalind_uge, lugh, unibe_ibu, vai, czbiohub_aws, jax, ccga_med, scw, tigem, tubingen_apg, google, ipop_up, googlels, eddie, medair, apptainer, bi, bigpurple, sbc_sharc, adcra, cedars, vsc_kul_uhasselt, pawsey_nimbus, ucl_myriad, utd_ganymede, charliecloud, icr_davros, ceres, munin, arm, rosalind, hasta, cfc, uzh, ebc, ku_sund_dangpu, ccga_dx, crick, marvin, biohpc_gen, shifter, mana, mamba, wehi, awsbatch, imperial, maestro, genotoul, abims, janelia, nihbiowulf, nu_genomics, googlebatch, oist, sahmri, mpcdf, leicester, vsc_ugent, sage, cambridge, podman, cheaha, xanadu, test, computerome, seg_globe, sanger, dkfz, pasteur, test_full, azurebatch, hki, crukmi, docker, engaging, gis, psmn, eva, fgcz, conda, crg, singularity, prince, utd_sysbio, debug, genouest, cbe, phoenix, gitpod, seawulf, uct_hpc, aws_tower, binac]
Jun-21 22:00:54.395 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declararion
Jun-21 22:00:54.411 [main] INFO nextflow.cli.CmdRun - Launching `./main.nf` [deadly_elion] DSL2 - revision: 691bff9081
Jun-21 22:00:54.412 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jun-21 22:00:54.412 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Jun-21 22:00:54.417 [main] DEBUG nextflow.secret.LocalSecretsProvider - Secrets store: /scratch/Brickman/pipelines/nfcore/secrets/store.json
Jun-21 22:00:54.420 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@29bd85db] - activable => nextflow.secret.LocalSecretsProvider@29bd85db
Jun-21 22:00:54.477 [main] DEBUG nextflow.Session - Session UUID: 9eeae9d5-2f33-4bcc-bc65-9a05757e3280
Jun-21 22:00:54.477 [main] DEBUG nextflow.Session - Run name: deadly_elion
Jun-21 22:00:54.478 [main] DEBUG nextflow.Session - Executor pool size: 224
Jun-21 22:00:54.488 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=672; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jun-21 22:00:54.514 [main] DEBUG nextflow.cli.CmdRun -
Version: 23.04.1 build 5866
Created: 15-04-2023 06:51 UTC (08:51 CEST)
System: Linux 4.18.0-425.3.1.el8.x86_64
Runtime: Groovy 3.0.16 on Java HotSpot(TM) 64-Bit Server VM 11.0.15+8-LTS-149
Encoding: UTF-8 (UTF-8)
Process: [email protected] [10.84.1.4]
CPUs: 224 - Mem: 3.9 TB (181 GB) - Swap: 4 GB (0)
Jun-21 22:00:54.539 [main] DEBUG nextflow.Session - Work-dir: /scratch/Brickman/pipelines/nf-core-marsseq/work [xfs]
Jun-21 22:00:54.558 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jun-21 22:00:54.568 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jun-21 22:00:54.618 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Jun-21 22:00:54.629 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 225; maxThreads: 1000
Jun-21 22:00:54.693 [main] DEBUG nextflow.Session - Session start
Jun-21 22:00:54.698 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /scratch/Brickman/pipelines/nf-core-marsseq/results/pipeline_info/execution_trace_2023-06-21_22-00-54.txt
Jun-21 22:00:54.707 [main] DEBUG nextflow.Session - Using default localLib path: /scratch/Brickman/pipelines/nf-core-marsseq/lib
Jun-21 22:00:54.710 [main] DEBUG nextflow.Session - Adding to the classpath library: /scratch/Brickman/pipelines/nf-core-marsseq/lib
Jun-21 22:00:54.710 [main] DEBUG nextflow.Session - Adding to the classpath library: /scratch/Brickman/pipelines/nf-core-marsseq/lib/nfcore_external_java_deps.jar
Jun-21 22:00:55.465 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jun-21 22:00:55.512 [main] INFO nextflow.Nextflow -
-�[2m----------------------------------------------------�[0m-
�[0;32m,--.�[0;30m/�[0;32m,-.�[0m
�[0;34m ___ __ __ __ ___ �[0;32m/,-._.--~'�[0m
�[0;34m |\ | |__ __ / ` / \ |__) |__ �[0;33m} {�[0m
�[0;34m | \| | \__, \__/ | \ |___ �[0;32m\`-._,-`-,�[0m
�[0;32m`._,._,'�[0m
�[0;35m nf-core/marsseq v1.0.0dev�[0m
-�[2m----------------------------------------------------�[0m-
�[1mCore Nextflow options�[0m
�[0;34mrunName : �[0;32mdeadly_elion�[0m
�[0;34mcontainerEngine : �[0;32msingularity�[0m
�[0;34mlaunchDir : �[0;32m/scratch/Brickman/pipelines/nf-core-marsseq�[0m
�[0;34mworkDir : �[0;32m/scratch/Brickman/pipelines/nf-core-marsseq/work�[0m
�[0;34mprojectDir : �[0;32m/scratch/Brickman/pipelines/nf-core-marsseq�[0m
�[0;34muserName : �[0;32mfdb589�[0m
�[0;34mprofile : �[0;32mku_sund_dangpu�[0m
�[0;34mconfigFiles : �[0;32m/scratch/Brickman/pipelines/nf-core-marsseq/nextflow.config�[0m
�[1mInput/output options�[0m
�[0;34minput : �[0;32m/home/fdb589/Brickman/projects/proks_et_al_2022/pipeline/SB26/design_remote.csv�[0m
�[0;34moutdir : �[0;32m./results�[0m
�[0;34mgenomes_base : �[0;32m./results/references�[0m
�[1mReference genome options�[0m
�[0;34mfasta : �[0;32m./results/references/mm10/GRCm39.fa�[0m
�[0;34mgtf : �[0;32m./results/references/mm10/GRCm39.gtf�[0m
�[0;34mbowtie2_index : �[0;32m./results/references/mm10/bowtie2/�[0m
�[0;34mstar_index : �[0;32m./results/references/mm10/star/�[0m
�[1mRNA velocity�[0m
�[0;34mread_length : �[0;32m75�[0m
�[1mInstitutional config options�[0m
�[0;34mconfig_profile_description: �[0;32mdangpufl01 configuration�[0m
�[0;34mconfig_profile_contact : �[0;32mAdrija Kalvisa <[email protected]>�[0m
�[1mMax job request options�[0m
�[0;34mmax_cpus : �[0;32m30�[0m
�[0;34mmax_memory : �[0;32m200 GB�[0m
�[0;34mmax_time : �[0;32m3d�[0m
!! Only displaying parameters that differ from the pipeline defaults !!
-�[2m----------------------------------------------------�[0m-
If you use nf-core/marsseq for your analysis please cite:
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
* Software dependencies
https://github.com/nf-core/marsseq/blob/master/CITATIONS.md
-�[2m----------------------------------------------------�[0m-
Jun-21 22:00:57.764 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withLabel:process_single` matches labels `process_single` for process with name NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK
Jun-21 22:00:57.765 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:SAMPLESHEET_CHECK` matches process NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK
Jun-21 22:00:57.770 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jun-21 22:00:57.770 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jun-21 22:00:57.779 [main] DEBUG nextflow.executor.Executor - [warm up] executor > slurm
Jun-21 22:00:57.783 [main] DEBUG n.processor.TaskPollingMonitor - Creating task monitor for executor 'slurm' > capacity: 5; pollInterval: 5s; dumpInterval: 5m
Jun-21 22:00:57.787 [main] DEBUG n.executor.AbstractGridExecutor - Creating executor 'slurm' > queue-stat-interval: 1m
Jun-21 22:00:57.891 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: FASTQC, ERCC_CREATE, BOWTIE2_BUILD, MULTIQC, CUTADAPT, VELOCITY_WHITELIST, DOWNLOAD_GTF, VELOCITY_STARSOLO, GUNZIP_GTF, MERGE_READS, DOWNLOAD_FASTA, CAT_CAT, QC_ALIGNED, WGET, DEMULTIPLEX, GUNZIP, SAMPLESHEET_CHECK, VELOCITY_TRIM, QC_REPORT, GUNZIP_FASTA, VELOCITY_CONVERT, FASTP_SPLIT, CAT_FASTA, NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK, QC_BATCH, PREPARE, STAR_GENOMEGENERATE, CUSTOM_DUMPSOFTWAREVERSIONS, BOWTIE2_ALIGN, CUT_SAM, STAR_ALIGN, EXTRACT_LABELS
Jun-21 22:00:57.896 [main] DEBUG nextflow.Session - Igniting dataflow network (11)
Jun-21 22:00:57.901 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK
Jun-21 22:00:57.902 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Jun-21 22:00:57.902 [main] DEBUG nextflow.Session - Session await
Jun-21 22:00:57.906 [PathVisitor-3] DEBUG nextflow.plugin.PluginUpdater - Installing plugin nf-amazon version: 1.16.2
Jun-21 22:00:57.914 [PathVisitor-3] INFO org.pf4j.AbstractPluginManager - Plugin '[email protected]' resolved
Jun-21 22:00:57.914 [PathVisitor-3] INFO org.pf4j.AbstractPluginManager - Start plugin '[email protected]'
Jun-21 22:00:57.932 [PathVisitor-3] DEBUG nextflow.plugin.BasePlugin - Plugin started [email protected]
Jun-21 22:00:57.947 [PathVisitor-3] DEBUG nextflow.file.FileHelper - > Added 'S3FileSystemProvider' to list of installed providers [s3]
Jun-21 22:00:57.947 [PathVisitor-3] DEBUG nextflow.file.FileHelper - Started plugin 'nf-amazon' required to handle file: s3://ngi-igenomes/test-data/marsseq/wells_cells.xlsx
Jun-21 22:00:57.954 [PathVisitor-3] DEBUG nextflow.file.FileHelper - Creating a file system instance for provider: S3FileSystemProvider
Jun-21 22:00:57.960 [PathVisitor-3] DEBUG nextflow.file.FileHelper - AWS S3 config details: {max_error_retry=5}
Jun-21 22:00:57.977 [Actor Thread 14] DEBUG nextflow.container.SingularityCache - Singularity found local store for image=https://depot.galaxyproject.org/singularity/python:3.8.3; path=/scratch/Brickman/singularity-images/depot.galaxyproject.org-singularity-python-3.8.3.img
Jun-21 22:00:58.119 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - [SLURM] submitted process NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv) > jobId: 61809; workDir: /scratch/Brickman/pipelines/nf-core-marsseq/work/e1/168e23842e46da72bb193df6d24e09
Jun-21 22:00:58.120 [Task submitter] INFO nextflow.Session - [e1/168e23] Submitted process > NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)
Jun-21 22:00:58.297 [PathVisitor-3] DEBUG com.upplication.s3fs.AmazonS3Client - Setting S3 glacierRetrievalTier=null
Jun-21 22:01:02.797 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 61809; id: 1; name: NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv); status: COMPLETED; exit: 0; error: -; workDir: /scratch/Brickman/pipelines/nf-core-marsseq/work/e1/168e23842e46da72bb193df6d24e09 started: 1687377662794; exited: 2023-06-21T20:01:02.132786Z; ]
Jun-21 22:01:02.816 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'PublishDir' minSize=10; maxSize=672; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Jun-21 22:01:02.820 [main] DEBUG nextflow.Session - Session await > all processes finished
Jun-21 22:01:02.835 [main] DEBUG nextflow.Session - Session await > all barriers passed
Jun-21 22:01:08.136 [Actor Thread 19] WARN c.a.i.InstanceMetadataServiceResourceFetcher - Fail to retrieve token
com.amazonaws.SdkClientException: Failed to connect to service endpoint:
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)
at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91)
at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69)
at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:60)
at com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:48)
at com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:124)
at com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:80)
at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:166)
at com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper.getCredentials(EC2ContainerCredentialsProviderWrapper.java:75)
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:118)
at com.amazonaws.services.s3.S3CredentialsProviderChain.getCredentials(S3CredentialsProviderChain.java:36)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1269)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1290)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5397)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:928)
at com.upplication.s3fs.AmazonS3Client.listObjects(AmazonS3Client.java:167)
at com.upplication.s3fs.util.S3ObjectSummaryLookup.lookup(S3ObjectSummaryLookup.java:112)
at com.upplication.s3fs.S3FileSystemProvider.checkAccess(S3FileSystemProvider.java:668)
at java.base/java.nio.file.Files.exists(Files.java:2440)
at java.base/jdk.internal.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.invoke(StaticMetaMethodSite.java:44)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:89)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:94)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139)
at nextflow.extension.FilesEx.exists(FilesEx.groovy:456)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:54)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1258)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
at groovy.runtime.metaclass.NextflowDelegatingMetaClass.invokeMethod(NextflowDelegatingMetaClass.java:64)
at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:44)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:130)
at Script_80cf5f94.create_fastq_channel(Script_80cf5f94:31)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:362)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:61)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
at Script_80cf5f94$_runScript_closure1$_closure2$_closure3.doCall(Script_80cf5f94:15)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:38)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139)
at nextflow.extension.MapOp$_apply_closure1.doCall(MapOp.groovy:56)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
at groovy.lang.Closure.call(Closure.java:412)
at groovyx.gpars.dataflow.operator.DataflowOperatorActor.startTask(DataflowOperatorActor.java:120)
at groovyx.gpars.dataflow.operator.DataflowOperatorActor.onMessage(DataflowOperatorActor.java:108)
at groovyx.gpars.actor.impl.SDAClosure$1.call(SDAClosure.java:43)
at groovyx.gpars.actor.AbstractLoopingActor.runEnhancedWithoutRepliesOnMessages(AbstractLoopingActor.java:293)
at groovyx.gpars.actor.AbstractLoopingActor.access$400(AbstractLoopingActor.java:30)
at groovyx.gpars.actor.AbstractLoopingActor$1.handleMessage(AbstractLoopingActor.java:93)
at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:132)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketException: Network is unreachable (connect failed)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
at java.base/java.net.Socket.connect(Socket.java:608)
at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341)
at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1232)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015)
at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:95)
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)
... 99 common frames omitted
Jun-21 22:01:24.646 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Jun-21 22:01:24.649 [main] INFO nextflow.Nextflow - -�[0;35m[nf-core/marsseq]�[0;32m Pipeline completed successfully�[0m-
Jun-21 22:01:24.649 [main] INFO nextflow.Nextflow - -�[0;35m[nf-core/marsseq]�[0;32m Pipeline completed successfully�[0m-
Jun-21 22:01:24.655 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=1s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=1; peakMemory=6 GB; ]
Jun-21 22:01:24.655 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file
Jun-21 22:01:24.657 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report
Jun-21 22:01:24.682 [main] DEBUG nextflow.trace.ReportObserver - Execution report summary data:
[{"cpuUsage":{"mean":65.4,"min":65.4,"q1":65.4,"q2":65.4,"q3":65.4,"max":65.4,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"},"process":"SAMPLESHEET_CHECK","mem":{"mean":4808704,"min":4808704,"q1":4808704,"q2":4808704,"q3":4808704,"max":4808704,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"},"memUsage":{"mean":0.07,"min":0.07,"q1":0.07,"q2":0.07,"q3":0.07,"max":0.07,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"},"timeUsage":{"mean":0.01,"min":0.01,"q1":0.01,"q2":0.01,"q3":0.01,"max":0.01,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"},"vmem":{"mean":31240192,"min":31240192,"q1":31240192,"q2":31240192,"q3":31240192,"max":31240192,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"},"reads":{"mean":2072025,"min":2072025,"q1":2072025,"q2":2072025,"q3":2072025,"max":2072025,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"},"cpu":{"mean":65.4,"min":65.4,"q1":65.4,"q2":65.4,"q3":65.4,"max":65.4,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"},"time":{"mean":1000,"min":1000,"q1":1000,"q2":1000,"q3":1000,"max":1000,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"},"writes":{"mean":774,"min":774,"q1":774,"q2":774,"q3":774,"max":774,"minLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","maxLabel":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q1Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q2Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)","q3Label":"NFCORE_MARSSEQ:MARSSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (design_remote.csv)"}}]
Jun-21 22:01:25.214 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline
Jun-21 22:01:25.318 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Jun-21 22:01:25.318 [main] INFO org.pf4j.AbstractPluginManager - Stop plugin '[email protected]'
Jun-21 22:01:25.318 [main] DEBUG nextflow.plugin.BasePlugin - Plugin stopped nf-amazon
Jun-21 22:01:25.350 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Jun-21 22:01:25.351 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
System information
DanGPU server
Merge downloading and execution in on workflow
Combine download-reference and main workflow in one, instead of having 2 step process.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.