Dear MINI-EX developers @SvitlanaLukicheva @gabimadrid,

I encountered a new issue, as shown below in the file .nextflow.log. The last error message have been repeating for several hours and it seems like it will be runing endlessly. I have also tried removing the parallel analysis, but the problem is still there. How can I solve this? Thanks.
Best,

.nextflow.log:

Jul-19 23:26:09.180 [main] DEBUG nextflow.cli.Launcher - $> nextflow -C miniex_run_slurm.config run miniex.nf
Jul-19 23:26:09.639 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 21.10.6
Jul-19 23:26:09.750 [main] INFO nextflow.cli.CmdRun - Launching miniex.nf [loving_mandelbrot] - revision: 7bb6de7c5c
Jul-19 23:26:09.830 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/miniex_run_slurm.config
Jul-19 23:26:09.939 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: standard
Jul-19 23:26:14.032 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; plugins-dir=/home/hguojian/.nextflow/plugins
Jul-19 23:26:14.047 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Jul-19 23:26:14.078 [main] INFO org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Jul-19 23:26:14.080 [main] INFO org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Jul-19 23:26:14.086 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Jul-19 23:26:14.153 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Jul-19 23:26:14.376 [main] DEBUG nextflow.Session - Session uuid: aa6306af-cf44-4dd6-955c-94a0b94e8126
Jul-19 23:26:14.376 [main] DEBUG nextflow.Session - Run name: loving_mandelbrot
Jul-19 23:26:14.377 [main] DEBUG nextflow.Session - Executor pool size: 2
Jul-19 23:26:14.412 [main] DEBUG nextflow.cli.CmdRun -
Version: 21.10.6 build 5660
Created: 21-12-2021 16:55 UTC (17:55 CEST)
System: Linux 3.10.0-1160.el7.x86_64
Runtime: Groovy 3.0.9 on OpenJDK 64-Bit Server VM 1.8.0_262-b10
Encoding: UTF-8 (UTF-8)
Process: 64296@node148 [192.168.1.148]
CPUs: 1 - Mem: 503.6 GB (99.1 GB) - Swap: 0 (0)
Jul-19 23:26:14.518 [main] DEBUG nextflow.Session - Work-dir: /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/work [gpfs]
Jul-19 23:26:14.581 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Jul-19 23:26:14.629 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Jul-19 23:26:14.898 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 2; maxThreads: 1000
Jul-19 23:26:15.182 [main] DEBUG nextflow.Session - Session start invoked
Jul-19 23:26:15.305 [main] WARN nextflow.NextflowMeta$Preview - DSL 2 PREVIEW MODE IS DEPRECATED - USE THE STABLE VERSION INSTEAD -- Read more at https://www.nextflow.io/docs/latest/dsl2.html#dsl2-migration-notes
Jul-19 23:26:21.714 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Jul-19 23:26:21.779 [main] INFO nextflow.Nextflow - Motif-Informed Network Inference from gene EXpression
=====================================================
Running TF motif enrichment filtering on TF-F_motifs
Running single-cell cluster enrichment using the top 700 upregulated genes per cluster
Filtering out regulons of single-cell clusters where the TF is expressed in less than 10 % of the cells
Plotting expression specificity and DE calls for the top 150 regulons

Jul-19 23:26:22.192 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 2; maxThreads: 1000
Jul-19 23:26:22.610 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:run_grnboost matches process run_grnboost
Jul-19 23:26:22.650 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-19 23:26:22.650 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-19 23:26:22.690 [main] DEBUG nextflow.executor.Executor - [warm up] executor > slurm
Jul-19 23:26:22.714 [main] DEBUG n.processor.TaskPollingMonitor - Creating task monitor for executor 'slurm' > capacity: 100; pollInterval: 5s; dumpInterval: 5m
Jul-19 23:26:22.746 [main] DEBUG n.executor.AbstractGridExecutor - Creating executor 'slurm' > queue-stat-interval: 1m
Jul-19 23:26:23.042 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:unzip_motifMappings matches process unzip_motifMappings
Jul-19 23:26:23.044 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-19 23:26:23.044 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-19 23:26:23.068 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:run_enricher_motifs matches process run_enricher_motifs
Jul-19 23:26:23.077 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-19 23:26:23.078 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-19 23:26:23.120 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:filter_motifs matches process filter_motifs
Jul-19 23:26:23.129 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-19 23:26:23.129 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-19 23:26:23.177 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:get_topDEGs matches process get_topDEGs
Jul-19 23:26:23.184 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-19 23:26:23.184 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-19 23:26:23.235 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:run_enricher_cluster matches process run_enricher_cluster
Jul-19 23:26:23.238 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-19 23:26:23.238 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-19 23:26:23.287 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:filter_expression matches process filter_expression
Jul-19 23:26:23.306 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-19 23:26:23.306 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-19 23:26:23.355 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:make_info_file matches process make_info_file
Jul-19 23:26:23.370 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-19 23:26:23.377 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-19 23:26:23.421 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:clustermap_regs matches process clustermap_regs
Jul-19 23:26:23.426 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-19 23:26:23.427 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-19 23:26:23.816 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:network_centrality matches process network_centrality
Jul-19 23:26:23.821 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-19 23:26:23.821 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-19 23:26:23.830 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:getFiles_enrichment matches process getFiles_enrichment
Jul-19 23:26:23.832 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-19 23:26:23.832 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-19 23:26:23.847 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:GO_enricher matches process GO_enricher
Jul-19 23:26:23.857 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-19 23:26:23.857 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-19 23:26:23.894 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:ranking_df_std matches process ranking_df_std
Jul-19 23:26:23.895 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-19 23:26:23.895 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-19 23:26:23.953 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:makeBorda matches process makeBorda
Jul-19 23:26:23.956 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-19 23:26:23.956 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-19 23:26:23.980 [main] DEBUG nextflow.script.ProcessConfig - Config settings withName:heatmap_tops matches process heatmap_tops
Jul-19 23:26:23.983 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: slurm
Jul-19 23:26:23.983 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'slurm'
Jul-19 23:26:23.989 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: clustermap_regs, filter_expression, ranking_df_ref, makeBorda, make_info_file, get_topDEGs, run_enricher_motifs, network_centrality, getFiles_enrichment, unzip_motifMappings, run_grnboost, run_enricher_cluster, GO_enricher, filter_motifs, heatmap_tops, ranking_df_std, check_reference
Jul-19 23:26:24.008 [main] DEBUG nextflow.Session - Ignite dataflow network (21)
Jul-19 23:26:24.050 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > run_grnboost
Jul-19 23:26:24.055 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > unzip_motifMappings
Jul-19 23:26:24.066 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > run_enricher_motifs
Jul-19 23:26:24.066 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > filter_motifs
Jul-19 23:26:24.083 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > get_topDEGs
Jul-19 23:26:24.083 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > run_enricher_cluster
Jul-19 23:26:24.092 [PathVisitor-2] DEBUG nextflow.file.PathVisitor - files for syntax: glob; folder: /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/tomato/INPUTS/; pattern: *_allMarkers.txt; options: [:]
Jul-19 23:26:24.101 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > filter_expression
Jul-19 23:26:24.102 [PathVisitor-3] DEBUG nextflow.file.PathVisitor - files for syntax: glob; folder: /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/tomato/INPUTS/; pattern: *_cells2clusters.txt; options: [:]
Jul-19 23:26:24.103 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > make_info_file
Jul-19 23:26:24.103 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > clustermap_regs
Jul-19 23:26:24.103 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > network_centrality
Jul-19 23:26:24.106 [PathVisitor-4] DEBUG nextflow.file.PathVisitor - files for syntax: glob; folder: /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/tomato/INPUTS/; pattern: *_identities.txt; options: [:]
Jul-19 23:26:24.113 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > getFiles_enrichment
Jul-19 23:26:24.114 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > GO_enricher
Jul-19 23:26:24.114 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > ranking_df_std
Jul-19 23:26:24.141 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > makeBorda
Jul-19 23:26:24.143 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > heatmap_tops
Jul-19 23:26:24.143 [main] DEBUG nextflow.script.ScriptRunner - > Await termination
Jul-19 23:26:24.143 [main] DEBUG nextflow.Session - Session await
Jul-19 23:26:24.145 [PathVisitor-1] DEBUG nextflow.file.PathVisitor - files for syntax: glob; folder: /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/tomato/INPUTS/; pattern: *_matrix.txt; options: [:]
Jul-19 23:26:24.473 [Actor Thread 14] DEBUG nextflow.container.SingularityCache - Singularity found local store for image=docker://vibpsb/mini-ex:latest; path=/work/hguojian/snRNA-seq/MINI-EX/MINI-EX/singularity_cache/vibpsb-mini-ex-latest.img
Jul-19 23:26:24.836 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - [SLURM] submitted process run_grnboost (1) > jobId: 49627772; workDir: /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/work/22/6a2a199cb46045f118423e0b434e03
Jul-19 23:26:24.847 [Task submitter] INFO nextflow.Session - [22/6a2a19] Submitted process > run_grnboost (1)
Jul-19 23:26:24.948 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - [SLURM] submitted process get_topDEGs (1) > jobId: 49627773; workDir: /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/work/1e/dad283c416641a9a84272e063a472e
Jul-19 23:26:24.948 [Task submitter] INFO nextflow.Session - [1e/dad283] Submitted process > get_topDEGs (1)
Jul-19 23:26:25.031 [Task submitter] DEBUG nextflow.executor.GridTaskHandler - [SLURM] submitted process unzip_motifMappings > jobId: 49627774; workDir: /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/work/47/8d44befea8da020bfae361d0d6f829
Jul-19 23:26:25.031 [Task submitter] INFO nextflow.Session - [47/8d44be] Submitted process > unzip_motifMappings
Jul-19 23:26:32.773 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 49627773; id: 3; name: get_topDEGs (1); status: COMPLETED; exit: 0; error: -; workDir: /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/work/1e/dad283c416641a9a84272e063a472e started: 1689801992757; exited: 2023-07-19T21:26:30.931118Z; ]
Jul-19 23:26:32.865 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 49627774; id: 1; name: unzip_motifMappings; status: COMPLETED; exit: 0; error: -; workDir: /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/work/47/8d44befea8da020bfae361d0d6f829 started: 1689801992846; exited: 2023-07-19T21:26:30.172087Z; ]
Jul-19 23:31:27.765 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 1 -- submitted tasks are shown below
~> TaskHandler[jobId: 49627772; id: 2; name: run_grnboost (1); status: RUNNING; exit: -; error: -; workDir: /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/work/22/6a2a199cb46045f118423e0b434e03 started: 1689801987762; exited: -; ]
Jul-19 23:36:27.777 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 1 -- submitted tasks are shown below
~> TaskHandler[jobId: 49627772; id: 2; name: run_grnboost (1); status: RUNNING; exit: -; error: -; workDir: /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/work/22/6a2a199cb46045f118423e0b434e03 started: 1689801987762; exited: -; ] this message will repeat again and again.

runing command:

#!/bin/bash

#SBATCH -J MINI-EX
#SBATCH -p workq
#SBATCH --nodes=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=60G
#SBATCH -o MINI-EX-%j.out
#SBATCH -e MINI-EX-%j.err

module purge
module load bioinfo/Nextflow-v21.10.6
module load system/singularity-3.7.3

nextflow -C miniex_run_slurm.config run miniex.nf

miniex_run_slurm.config:

executor {
name = 'slurm'
queueSize = 4
}

process.container = "vibpsb/mini-ex:latest"
singularity {
enabled = true
cacheDir = "singularity_cache"
autoMounts = true
}

params {
expressionMatrix = "$baseDir/tomato/INPUTS/_matrix.txt"
markersOut = "$baseDir/tomato/INPUTS/_allMarkers.txt"
cell2clusters = "$baseDir/tomato/INPUTS/_cells2clusters.txt"
cluster2ident = "$baseDir/tomato/INPUTS/_identities.txt"
TF_list = "$baseDir/tomato/INPUTS/TF_list.txt"
termsOfInterest = null
// termsOfInterest = "$baseDir/tomato/INPUTS/GOsIwant.txt"

    grnboostOut = null

// grnboostOut = "/$baseDir/tomato/OUTPUTS/GRNBoost2_output/*_grnboost2.txt"

    featureFile_motifs = "$baseDir/data_sly/sly_2023.1_motifMapping.out.gz"
    infoTF = "$baseDir/data_sly/sly_TF2fam2mot.txt"
    GOfile = "$baseDir/data_sly/sly_go_gene_plaza_all_filt1k_extended_bp_noBP.txt"
    alias = "$baseDir/data_sly/sly_gene_aliases.txt"

    script_enricher = "$baseDir/bin/enricherv2.4"
    script_grnboost = "$baseDir/bin/MINIEX_grnboostMultiprocess.py"
    script_motifs = "$baseDir/bin/MINIEX_filterForMotifs.py"
    script_topDEGs = "$baseDir/bin/MINIEX_selectTopDEGs.py"
    script_expTFs = "$baseDir/bin/MINIEX_filterForTFExp.py"
    script_info = "$baseDir/bin/MINIEX_makeInfoFile.py"
    script_clustermap = "$baseDir/bin/MINIEX_clustermap.py"
    script_networkCentrality = "$baseDir/bin/MINIEX_network_analysis.py"
    script_checkReference = "$baseDir/bin/MINIEX_checkRef.py"
    script_filesEnrichment = "$baseDir/bin/MINIEX_makeFilesEnrichment.py"
    script_makedfRef = "$baseDir/bin/MINIEX_makeRankingDf_ref.py"
    script_makedfStd = "$baseDir/bin/MINIEX_makeRankingDf_std.py"
    script_makeborda = "$baseDir/bin/MINIEX_makeBorda.py"
    script_heatmapTops = "$baseDir/bin/MINIEX_visual_heatmap_top150.py"

    tops = "700"
    expressionFilter = "10"
    motifFilter = "TF-F_motifs"
    topRegs = "150"

}

process {
withName: run_grnboost {
clusterOptions = '--mem=50G -c 4'
}
withName: unzip_motifMappings {
clusterOptions = '--mem=2G'
}
withName: run_enricher_motifs {
clusterOptions = '--mem=4G'
}
withName: filter_motifs {
clusterOptions = '--mem=20G'
}
withName: get_topDEGs {
clusterOptions = '--mem=10G'
}
withName: run_enricher_cluster {
clusterOptions = '--mem=4G'
}
withName: filter_expression {
clusterOptions = '--mem=20G'
}
withName: make_info_file {
clusterOptions = '--mem=10G'
}
withName: network_centrality {
clusterOptions = '--mem=20G'
}
withName: getFiles_enrichment {
clusterOptions = '--mem=10G'
}
withName: GO_enricher {
clusterOptions = '--mem=4G'
}
withName: check_reference {
clusterOptions = '--mem=10G'
}
withName: ranking_df_ref {
clusterOptions = '--mem=10G'
}
withName: ranking_df_std {
clusterOptions = '--mem=10G'
}
withName: makeBorda {
clusterOptions = '--mem=20G'
}
withName: heatmap_tops {
clusterOptions = '--mem=20G'
}
withName: clustermap_regs {
clusterOptions = '--mem=20G'
}
}

runing output

N E X T F L O W ~ version 21.10.6
Launching miniex.nf [loving_mandelbrot] - revision: 7bb6de7c5c
WARN: DSL 2 PREVIEW MODE IS DEPRECATED - USE THE STABLE VERSION INSTEAD -- Read more at https://www.nextflow.io/docs/latest/dsl2.html#dsl2-migration-notes
Motif-Informed Network Inference from gene EXpression
=====================================================
Running TF motif enrichment filtering on TF-F_motifs
Running single-cell cluster enrichment using the top 700 upregulated genes per cluster
Filtering out regulons of single-cell clusters where the TF is expressed in less than 10 % of the cells
Plotting expression specificity and DE calls for the top 150 regulons

[- ] process > run_grnboost -

[- ] process > run_grnboost -
[- ] process > unzip_motifMappings -
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[- ] process > get_topDEGs -
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -

[- ] process > run_grnboost -
[- ] process > unzip_motifMappings -
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[- ] process > get_topDEGs -
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -

[- ] process > run_grnboost -
[- ] process > unzip_motifMappings -
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[- ] process > get_topDEGs -
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > heatmap_tops -

[- ] process > run_grnboost -
[- ] process > unzip_motifMappings -
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[- ] process > get_topDEGs -
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > heatmap_tops -
[AllStages, /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/tomato/INPUTS/AllStages_matrix.txt]

[- ] process > run_grnboost [ 0%] 0 of 1
[- ] process > unzip_motifMappings [ 0%] 0 of 1
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[- ] process > get_topDEGs [ 0%] 0 of 1
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > heatmap_tops -
[AllStages, /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/tomato/INPUTS/AllStages_matrix.txt]

executor > slurm (3)
[22/6a2a19] process > run_grnboost (1) [ 0%] 0 of 1
[47/8d44be] process > unzip_motifMappings [ 0%] 0 of 1
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[1e/dad283] process > get_topDEGs (1) [ 0%] 0 of 1
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > heatmap_tops -
[AllStages, /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/tomato/INPUTS/AllStages_matrix.txt]

executor > slurm (3)
[22/6a2a19] process > run_grnboost (1) [ 0%] 0 of 1
[47/8d44be] process > unzip_motifMappings [ 0%] 0 of 1
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[1e/dad283] process > get_topDEGs (1) [ 0%] 0 of 1
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > heatmap_tops -
[AllStages, /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/tomato/INPUTS/AllStages_matrix.txt]

executor > slurm (3)
[22/6a2a19] process > run_grnboost (1) [ 0%] 0 of 1
[47/8d44be] process > unzip_motifMappings [100%] 1 of 1 ✔
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[1e/dad283] process > get_topDEGs (1) [100%] 1 of 1 ✔
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > heatmap_tops -
[AllStages, /work/hguojian/snRNA-seq/MINI-EX/MINI-EX/tomato/INPUTS/AllStages_matrix.txt]

Hi.I'm trying to use MINI-EX，but It report an error

N E X T F L O W  ~  version 22.04.0
Launching `miniex.nf` [amazing_varahamihira] DSL2 - revision: 7bd79d770f
      Motif-Informed Network Inference from gene EXpression
      =====================================================
Running TF motif enrichment filtering on  TF-F_motifs
      Running single-cell cluster enrichment using the top 700 upregulated genes per cluster
      Filtering out regulons of single-cell clusters where the TF is expressed in less than 10 % of the cells
Plotting expression specificity and DE calls for the top 150 regulons

executor >  local (10)
[2f/f17293] process > unzip_motifMappings      [100%] 1 of 1 ✔
[c4/9e17ed] process > run_enricher_motifs (1)  [100%] 1 of 1 ✔
executor >  local (10)
[2f/f17293] process > unzip_motifMappings      [100%] 1 of 1 ✔
[c4/9e17ed] process > run_enricher_motifs (1)  [100%] 1 of 1 ✔
[9c/92e903] process > filter_motifs (1)        [100%] 1 of 1 ✔
[ef/327f3c] process > get_topDEGs (1)          [100%] 1 of 1 ✔
[ea/da6a47] process > run_enricher_cluster (1) [100%] 1 of 1 ✔
[08/06ac5a] process > filter_expression (1)    [100%] 1 of 1 ✔
[62/071a24] process > make_info_file (1)       [100%] 1 of 1, failed: 1 ✘
[-        ] process > clustermap_regs (1)      -
[-        ] process > network_centrality (1)   -
[-        ] process > getFiles_enrichment (1)  -
[-        ] process > GO_enricher              -
[-        ] process > ranking_df_std           -
[-        ] process > makeBorda                -
[-        ] process > heatmap_tops             -
Error executing process > 'make_info_file (1)'

Caused by:
  Process `make_info_file (1)` terminated with an error exit status (127)
Command executed:

  python3 MINIEX_makeInfoFile.py "sample_matrix.txt" "sample_grnboost2.txt" "sample_enrichedRegulons.txt" "sample_regulons.txt" zma_TF2fam2mot.txt sample_cells2clusters.txt sample_identities.txt "sample_TF_info_file.txt"

Command exit status:
  127

Command output:
  (empty)

Command wrapper:
  .command.run: line 18: module: command not found

I don't know how to fix it.

Dear Developers

I noticed that when I mistakenly supply either (1) a file with the wrong number of columns, or (2) a gzipped file that was supposed to be unzipped (such as the miniexExample_allMarkers.tsv file), the "check_user_input" stage does not flag this. Instead, an error gets thrown when these files are parsed later in the script.

Would it be possible to add checks for these two cases to the user input verification stage?

Thank you!

Hi.I'm trying to use MINI-EX，but It report an error
N E X T F L O W ~ version 22.10.0 Launching miniex.nf` [intergalactic_perlman] DSL2 - revision: 83499324ee
Motif-Informed Network Inference from gene EXpression
=====================================================
Running TF motif enrichment filtering on TF-F_motifs
Running single-cell cluster enrichment using the top 700 upregulated genes per cluster
Filtering out regulons of single-cell clusters where the TF is expressed in less than 10 % of the cells
Plotting expression specificity and DE calls for the top 150 regulons

[- ] process > run_grnboost -
[- ] process > unzip_motifMappings -
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[- ] process > get_topDEGs -
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > check_reference -
[- ] process > ranking_df_ref -
[- ] process > makeBorda -
[- ] process > heatmap_tops -
WARN: The echo directive has been deprecated - use to debug instead

[- ] process > run_grnboost [ 0%] 0 of 1
[- ] process > unzip_motifMappings [ 0%] 0 of 1
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[- ] process > get_topDEGs [ 0%] 0 of 1
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > check_reference -
[- ] process > ranking_df_ref -
[- ] process > makeBorda -
[- ] process > heatmap_tops -
[miniexExample, /public1/home/scb0721/sc52142/hq/MINI-EX-main/example/INPUTS/miniexExample_matrix.txt]
WARN: The echo directive has been deprecated - use to debug instead

[- ] process > run_grnboost [ 0%] 0 of 1
[- ] process > unzip_motifMappings [ 0%] 0 of 1
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[- ] process > get_topDEGs [ 0%] 0 of 1
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > check_reference -
[- ] process > ranking_df_ref -
[- ] process > makeBorda -
[- ] process > heatmap_tops -
[miniexExample, /public1/home/scb0721/sc52142/hq/MINI-EX-main/example/INPUTS/miniexExample_matrix.txt]
WARN: The echo directive has been deprecated - use to debug instead
Error executing process > 'run_grnboost (1)'

Caused by:
Failed to submit process to grid scheduler for execution

Command executed:

qsub -terse .command.run

Command exit status:
2

Command output:
Unknown option: terse
Usage:
qsub [-a start_time] [-A account] [-b y|n] [-e err_path] [-I] [-l
resource_list] [-m mail_options] [-M user_list] [-N job_name] [-o
out_path] [-p priority] [-pe shm task_cnt] [-P wckey] [-q destination]
[-r y|n] [-v variable_list] [-V] [-wd workdir] [-W
additional_attributes] [-h] [script]

Work dir:
/public1/home/scb0721/sc52142/hq/MINI-EX-main/work/a3/dcbbbccbc942696df1e150727f2153

Tip: when you have fixed the problem you can continue the execution adding the option -resume to the run command line

Oops .. something went wrong

[a3/dcbbbc] process > run_grnboost (1) [100%] 1 of 1, failed: 1
[- ] process > unzip_motifMappings [ 0%] 0 of 1
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[- ] process > get_topDEGs [ 0%] 0 of 1
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > check_reference -
[- ] process > ranking_df_ref -
[- ] process > makeBorda -
[- ] process > heatmap_tops -
[miniexExample, /public1/home/scb0721/sc52142/hq/MINI-EX-main/example/INPUTS/miniexExample_matrix.txt]
WARN: The echo directive has been deprecated - use to debug instead
Error executing process > 'run_grnboost (1)'

Caused by:
Failed to submit process to grid scheduler for execution

Command executed:

qsub -terse .command.run

Command exit status:
2

Command output:
Unknown option: terse
Usage:
qsub [-a start_time] [-A account] [-b y|n] [-e err_path] [-I] [-l
resource_list] [-m mail_options] [-M user_list] [-N job_name] [-o
out_path] [-p priority] [-pe shm task_cnt] [-P wckey] [-q destination]
[-r y|n] [-v variable_list] [-V] [-wd workdir] [-W
additional_attributes] [-h] [script]

Work dir:
/public1/home/scb0721/sc52142/hq/MINI-EX-main/work/a3/dcbbbccbc942696df1e150727f2153

Tip: when you have fixed the problem you can continue the execution adding the option -resume to the run command line

`
How do I solve it?

Dear Mini_Ex Team,

First thanks for providing such tool. I am excited to look at the results.
I encounter an issue running your example sample at the run_enricher_motifs step. I am working on an HPC using slurm based jobs.
Here is the error:

executor >  local (1), slurm (5)
[1e/ef54ac] process > check_user_input (1)       [100%] 1 of 1 ✔
[ee/ac2bd0] process > run_grnboost (1)           [100%] 1 of 1 ✔
[7f/ca0f1a] process > get_expressed_genes (1)    [100%] 1 of 1 ✔
[67/69742a] process > unzip_motif_mappings       [100%] 1 of 1 ✔
[8b/6e96e2] process > run_enricher_motifs (1)    [  0%] 0 of 1
[-        ] process > filter_motifs              -
[54/8a9c6a] process > get_top_degs (1)           [100%] 1 of 1 ✔
[-        ] process > run_enricher_cluster       -
[-        ] process > filter_expression          -
[-        ] process > make_info_file             -
[-        ] process > make_regulon_clustermap    -
[-        ] process > get_network_centrality     -
[-        ] process > make_go_enrichment_files   -
[-        ] process > run_enricher_go            -
[-        ] process > check_reference            -
[-        ] process > make_ref_ranking_dataframe -
[-        ] process > make_borda                 -
[-        ] process > score_edges                -
[-        ] process > make_top_regulons_heatmaps -
[-        ] process > make_regmaps               -
[-        ] process > make_log_file              -
Error executing process > 'run_enricher_motifs (1)'

Caused by:
  Process `run_enricher_motifs (1)` terminated with an error exit status (126)

Command executed:

  cat "MotifMappings.csv" | grep -f "miniexExample_expressedGenes.txt" > featureFileMotifsfiltered.txt
  enricherv3.2.4 featureFileMotifsfiltered.txt "miniexExample_grnboost2.tsv" -f 0.001 -n $(cat "miniexExample_expressedGenes.txt" | wc -l) --min-hits 2 --print-hits -o "miniexExample_enricherRegulons.txt"
  
  # Throw an error if no enrichment is found
  if [ "$(head miniexExample_enricherRegulons.txt | grep -P '^[^#]' | wc -l)" -eq 0 ]; then 
      echo 'ERROR: Enricher output empty!'
      exit 1
  fi

Command exit status:
  126

Command output:
  (empty)

Command error:
  .command.sh: line 3: /MiniEx/bin/enricherv3.2.4: Permission denied

Work dir:
  /scratch/bg93/work/8b/6e96e2408d7fb39e1bb5c7072cafa7

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`


Oops .. something went wrong

Could you tell me what could be the issue ?
Thanks in advance for your help,

Bruno

I tried running MINI-EX on a mac with apple silicone (M1) and swiftly got an error that the docker image architecture (amd64) was not compatible with my arm64 architecture.

For others having this issue, I got it to run using a Linux VM with Rosetta (Apple's "translator" software).

Hello,

I ran the following script to analyze the MINI-EX job. The job completed without any issues, but certain processes were not executed. I would greatly appreciate it if you could help me identify how I can correct my script. Thank you.

my script:

module load bioinfo/Nextflow/23.04.3
module load containers/singularity/3.9.9
nextflow -C miniex_run_slurm.config run miniex.nf -resume 

or nextflow -C miniex_run_slurm.config run miniex.nf

miniex_run_slurm.config:

` executor {
name = 'slurm'
queueSize = 5
}

process.container = "vibpsb/mini-ex:latest"
singularity {
enabled = true
cacheDir = "singularity_cache"
autoMounts = true
}

params {
expressionMatrix = "$baseDir/tomato/INPUTS/_matrix.txt"
markersOut = "$baseDir/tomato/INPUTS/_allMarkers.txt"
cell2clusters = "$baseDir/tomato/INPUTS/_cells2clusters.txt"
cluster2ident = "$baseDir/tomato/INPUTS/_identities.txt"
TF_list = "$baseDir/tomato/INPUTS/TF_list.txt"
//termsOfInterest = "$baseDir/tomato/INPUTS/GOsIwant.txt"
termsOfInterest = null

//grnboostOut = "/$baseDir/tomato/OUTPUTS/GRNBoost2_output/*_grnboost2.txt"
grnboostOut = null

doMotifAnalysis = true // set to <false> if no motif mapping data is available [CAUTION: without motif data MINI-EX is less reliable]
featureFile_motifs = "$baseDir/data_sly/sly_2023.1_motifMapping.out.gz"
infoTF = "$baseDir/data_sly/sly_TF2fam2mot.txt"
GOfile = "$baseDir/data_sly/sly_go_gene_plaza_all_filt1k_extended_bp_noBP.txt" //set to <null> if GO data is not available (when doing so, termsOfInterest should also be set to <null>)
alias = "$baseDir/data_sly/sly_gene_aliases.txt"

script_enricher = "$baseDir/bin/enricherv2.4"
script_checkInput = "$baseDir/bin/MINIEX_checkInput.py"
script_grnboost = "$baseDir/bin/MINIEX_grnboostMultiprocess.py"
script_motifs = "$baseDir/bin/MINIEX_filterForMotifs.py"
script_topDEGs = "$baseDir/bin/MINIEX_selectTopDEGs.py"
script_expTFs = "$baseDir/bin/MINIEX_filterForTFExp.py"
script_info = "$baseDir/bin/MINIEX_makeInfoFile.py"
script_clustermap = "$baseDir/bin/MINIEX_clustermap.py"
script_networkCentrality = "$baseDir/bin/MINIEX_network_analysis.py"
script_checkReference = "$baseDir/bin/MINIEX_checkRef.py"
script_filesEnrichment = "$baseDir/bin/MINIEX_makeFilesEnrichment.py"
script_makedfRef = "$baseDir/bin/MINIEX_makeRankingDf_ref.py"
script_makedfStd = "$baseDir/bin/MINIEX_makeRankingDf_std.py"

script_makeborda = "$baseDir/bin/MINIEX_makeBorda.py"
script_scoreEdges = "$baseDir/bin/MINIEX_scoreEdges.py"
script_heatmapTops = "$baseDir/bin/MINIEX_visual_heatmap_top150.py"
script_regmaps = "$baseDir/bin/MINIEX_regmap.py"

tops = "700"
expressionFilter = "10"
motifFilter = "TF-F_motifs" // To use the motifs of the TF family, specify "TF-F_motifs". To only use the motifs known for a TF, specify "TF_motifs".
topRegs = "150"

}

process {

withName: check_input_files {
    clusterOptions = '--mem=10G'
}
withName: get_expressed_genes {
    clusterOptions = '--mem=2G'
}
withName: run_grnboost {
    clusterOptions = '--mem=60G -c 5'
}
withName: unzip_motifMappings {
    executor='local'
}
withName: run_enricher_motifs {
    clusterOptions = '--mem=4G'
}
withName: filter_motifs {
    clusterOptions = '--mem=20G'
}
withName: filter_motifs_dummy {
    executor='local'
}
withName: get_topDEGs {
    clusterOptions = '--mem=10G'
}
withName: run_enricher_cluster {
    clusterOptions = '--mem=4G'
}
withName: filter_expression {
    clusterOptions = '--mem=20G'
}
    withName: make_info_file {
    clusterOptions = '--mem=10G'
}
withName: network_centrality {
    clusterOptions = '--mem=20G'
}
withName: getFiles_enrichment {
    clusterOptions = '--mem=10G'
}
withName: GO_enricher {
    clusterOptions = '--mem=4G'
}
    withName: check_reference {
    clusterOptions = '--mem=10G'
}
withName: ranking_df_ref {
    clusterOptions = '--mem=10G'
}
withName: ranking_df_std {
    clusterOptions = '--mem=10G'
}
withName: makeBorda {
    clusterOptions = '--mem=10G'
}
withName: scoreEdges {
    clusterOptions = '--mem=15G'
}
withName: heatmap_tops {
    clusterOptions = '--mem=20G'
}
withName: clustermap_regs {
    clusterOptions = '--mem=40G'
}
withName: regmaps {
    clusterOptions = '--mem=80G'
}

}
`

nf report:

N E X T F L O W ~ version 23.04.3
Launching miniex.nf [pedantic_aryabhata] DSL2 - revision: 7d53c557b4
Motif-Informed Network Inference from gene EXpression v.2.0

Running TF motif enrichment filtering on TF-F_motifs
Running single-cell cluster enrichment using the top 700 upregulated genes per cluster
Filtering out regulons of single-cell clusters where the TF is expressed in less than 10 % of the cells
Plotting expression specificity and DE calls for the top 150 regulons

WARN: The echo directive has been deprecated - use to debug instead
[- ] process > check_input_files -
[- ] process > run_grnboost -
[- ] process > get_expressed_genes -
[- ] process > unzip_motifMappings -
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[- ] process > get_topDEGs -
[- ] process > run_enricher_cluster -

[- ] process > check_input_files -
[- ] process > run_grnboost -
[- ] process > get_expressed_genes -
[- ] process > unzip_motifMappings -
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[- ] process > get_topDEGs -
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > scoreEdges -
[- ] process > heatmap_tops -
[- ] process > regmaps -
[DPA3, /work/user/kwang/MINI-EX/tomato/INPUTS/DPA3_matrix.txt]

executor > slurm (3)
[- ] process > check_input_files -
[e8/2a9b3b] process > run_grnboost (1) [ 0%] 0 of 1
[78/75664b] process > get_expressed_genes (1) [ 0%] 0 of 1
[1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[54/b9d02c] process > get_topDEGs (1) [ 0%] 0 of 1
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > scoreEdges -
[- ] process > heatmap_tops -
[- ] process > regmaps -
[DPA3, /work/user/kwang/MINI-EX/tomato/INPUTS/DPA3_matrix.txt]

executor > slurm (3)
[- ] process > check_input_files -
[e8/2a9b3b] process > run_grnboost (1) [ 0%] 0 of 1
[78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔
[1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > scoreEdges -
[- ] process > heatmap_tops -
[- ] process > regmaps -
[DPA3, /work/user/kwang/MINI-EX/tomato/INPUTS/DPA3_matrix.txt]

executor > slurm (4)
[- ] process > check_input_files -
[e8/2a9b3b] process > run_grnboost (1) [100%] 1 of 1 ✔
[78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔
[1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔
[00/1b3c1b] process > run_enricher_motifs (1) [ 0%] 0 of 1
[- ] process > filter_motifs -
[54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > scoreEdges -
[- ] process > heatmap_tops -
[- ] process > regmaps -
[DPA3, /work/user/kwang/MINI-EX/work/e8/2a9b3bd797fdf76811b4ec6eb5721b/DPA3_grnboost2.txt]

executor > slurm (4)
[- ] process > check_input_files -
[e8/2a9b3b] process > run_grnboost (1) [100%] 1 of 1 ✔
[78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔
[1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔
[00/1b3c1b] process > run_enricher_motifs (1) [ 0%] 0 of 1
[- ] process > filter_motifs -
[54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > scoreEdges -
[- ] process > heatmap_tops -
[- ] process > regmaps -
[DPA3, /work/user/kwang/MINI-EX/work/e8/2a9b3bd797fdf76811b4ec6eb5721b/DPA3_grnboost2.txt]

executor > slurm (5)
[- ] process > check_input_files -
[e8/2a9b3b] process > run_grnboost (1) [100%] 1 of 1 ✔
[78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔
[1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔
[00/1b3c1b] process > run_enricher_motifs (1) [100%] 1 of 1 ✔
[c4/ceedbb] process > filter_motifs (1) [ 0%] 0 of 1
[54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > scoreEdges -
[- ] process > heatmap_tops -
[- ] process > regmaps -

executor > slurm (5)
[- ] process > check_input_files -
[e8/2a9b3b] process > run_grnboost (1) [100%] 1 of 1 ✔
[78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔
[1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔
[00/1b3c1b] process > run_enricher_motifs (1) [100%] 1 of 1 ✔
[c4/ceedbb] process > filter_motifs (1) [ 0%] 0 of 1
[54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > scoreEdges -
[- ] process > heatmap_tops -
[- ] process > regmaps -

executor > slurm (6)
[- ] process > check_input_files -
[e8/2a9b3b] process > run_grnboost (1) [100%] 1 of 1 ✔
[78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔
[1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔
[00/1b3c1b] process > run_enricher_motifs (1) [100%] 1 of 1 ✔
[c4/ceedbb] process > filter_motifs (1) [100%] 1 of 1 ✔
[54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔
[64/081566] process > run_enricher_cluster (1) [ 0%] 0 of 1
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > scoreEdges -
[- ] process > heatmap_tops -
[- ] process > regmaps -

executor > slurm (6)
[- ] process > check_input_files -
[e8/2a9b3b] process > run_grnboost (1) [100%] 1 of 1 ✔
[78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔
[1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔
[00/1b3c1b] process > run_enricher_motifs (1) [100%] 1 of 1 ✔
[c4/ceedbb] process > filter_motifs (1) [100%] 1 of 1 ✔
[54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔
[64/081566] process > run_enricher_cluster (1) [ 0%] 0 of 1
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > scoreEdges -
[- ] process > heatmap_tops -
[- ] process > regmaps -

executor > slurm (6)
[- ] process > check_input_files -
[e8/2a9b3b] process > run_grnboost (1) [100%] 1 of 1 ✔
[78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔
[1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔
[00/1b3c1b] process > run_enricher_motifs (1) [100%] 1 of 1 ✔
[c4/ceedbb] process > filter_motifs (1) [100%] 1 of 1 ✔
[54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔
[64/081566] process > run_enricher_cluster (1) [100%] 1 of 1 ✔
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > scoreEdges -
[- ] process > heatmap_tops -
[- ] process > regmaps -

executor > slurm (6)
[- ] process > check_input_files -
[e8/2a9b3b] process > run_grnboost (1) [100%] 1 of 1 ✔
[78/75664b] process > get_expressed_genes (1) [100%] 1 of 1 ✔
[1f/1fd247] process > unzip_motifMappings [100%] 1 of 1, cached: 1 ✔
[00/1b3c1b] process > run_enricher_motifs (1) [100%] 1 of 1 ✔
[c4/ceedbb] process > filter_motifs (1) [100%] 1 of 1 ✔
[54/b9d02c] process > get_topDEGs (1) [100%] 1 of 1 ✔
[64/081566] process > run_enricher_cluster (1) [100%] 1 of 1 ✔
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > ranking_df_std -
[- ] process > makeBorda -
[- ] process > scoreEdges -
[- ] process > heatmap_tops -
[- ] process > regmaps -
Done!
Completed at: 12-Dec-2023 22:07:57
Duration : 11h 9m 2s
CPU hours : 11.1 (0% cached)
Succeeded : 6
Cached : 1

Hi, I'm trying to use MINIEX for my analysis but I run into an error trying the example. Can you help me with it?
I downloaded the repository directly from GitHub to my cluster so not sure what could be missing
The error is:
Error executing process > 'unzip_motifMappings'

Caused by:
java.io.IOException: Cannot run program "qsub" (in directory "/blue/mresende/share/gmadrid/MINIEX/MINI-EX/work/bb/d950bab0b71245a580a521aa9041cb"): error=2, No such file or directory

Here's the full info:

N E X T F L O W ~ version 22.11.1-edge
Launching miniex.nf [sharp_roentgen] DSL2 - revision: f3e1492824
Motif-Informed Network Inference from gene EXpression
=====================================================
Running TF motif enrichment filtering on TF-F_motifs
Running single-cell cluster enrichment using the top 700 upregulated genes per cluster
Filtering out regulons of single-cell clusters where the TF is expressed in less than 10 % of the cells
Plotting expression specificity and DE calls for the top 150 regulons

[- ] process > run_grnboost [ 0%] 0 of 1
[- ] process > unzip_motifMappings [ 0%] 0 of 1
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[- ] process > run_grnboost [ 0%] 0 of 1
[bb/d950ba] process > unzip_motifMappings [100%] 1 of 1, failed: 1
[- ] process > run_enricher_motifs -
[- ] process > filter_motifs -
[- ] process > get_topDEGs [ 0%] 0 of 1
[- ] process > run_enricher_cluster -
[- ] process > filter_expression -
[- ] process > make_info_file -
[- ] process > clustermap_regs -
[- ] process > network_centrality -
[- ] process > getFiles_enrichment -
[- ] process > GO_enricher -
[- ] process > check_reference -
[- ] process > ranking_df_ref -
[- ] process > makeBorda -
[- ] process > heatmap_tops -
Error executing process > 'unzip_motifMappings'

Caused by:
java.io.IOException: Cannot run program "qsub" (in directory "/blue/mresende/share/gmadrid/MINIEX/MINI-EX/work/bb/d950bab0b71245a580a521aa9041cb"): error=2, No such file or directory

Command executed:

qsub -terse .command.run

Command exit status:

Command output:
(empty)

Dear MINI-EX developer,

Thank you for providing this useful tool for plant scRNA-seq analyis.

I have noticed that there are several folders, including data_ath, data_osa and data_zma in your github repository. Are these files also necessary inputs to perform GRN anlysis? how can I generate them if I'm working with a different plant species, e.g. tomato? I have searched the website and also your MP publication, but it's not very clear to me how these files (motifMapping.out, TF2fam2mo.txt, full_BP_expcur_ext_names.txt) being prepared from which commands, as threre are too many steps and different tools were described in the methods section.

Thanks in advance,
Best,

Hi,
I'm quite new to python (and especially setting up environments on Linux). I think you have created a very usefull module and would like to run it on my own data. Therefore I've tried to set it up, but run into some trouble.
I'm running Python 3.8.10 and i've installed the requirements.txt with pip. Then trying to run one of the processes separately, it seems that there is a requirement for a higher pandas version as version 0.25.3 doesn't have StringDtype.
What am I doing wrong?
Thanks in advance.

Here's my stacktrace:

(miniex-env) harald@vm-harald-linux:~/python/miniex$ python3 bin/MINIEX_grnboostMultiprocess.py example/INPUTS/TF_list.txt example/INPUTS/"miniexExample_matrix.txt" 5 example/OUTPUTS/"miniexExample_grnboost2.txt"
Traceback (most recent call last):
File "bin/MINIEX_grnboostMultiprocess.py", line 10, in
from arboreto.algo import grnboost2, _prepare_input
File "/home/harald/python/miniex-env/lib/python3.8/site-packages/arboreto/algo.py", line 7, in
from arboreto.core import create_graph, SGBM_KWARGS, RF_KWARGS, EARLY_STOP_WINDOW_LENGTH
File "/home/harald/python/miniex-env/lib/python3.8/site-packages/arboreto/core.py", line 11, in
from dask.dataframe import from_delayed
File "/home/harald/python/miniex-env/lib/python3.8/site-packages/dask/dataframe/init.py", line 4, in
from dask.dataframe import backends, dispatch, rolling
File "/home/harald/python/miniex-env/lib/python3.8/site-packages/dask/dataframe/backends.py", line 20, in
from dask.dataframe.core import DataFrame, Index, Scalar, Series, _Frame
File "/home/harald/python/miniex-env/lib/python3.8/site-packages/dask/dataframe/core.py", line 35, in
from dask.dataframe import methods
File "/home/harald/python/miniex-env/lib/python3.8/site-packages/dask/dataframe/methods.py", line 22, in
from dask.dataframe.utils import is_dataframe_like, is_index_like, is_series_like
File "/home/harald/python/miniex-env/lib/python3.8/site-packages/dask/dataframe/utils.py", line 18, in
from dask.dataframe import ( # noqa: F401 register pandas extension types
File "/home/harald/python/miniex-env/lib/python3.8/site-packages/dask/dataframe/_dtypes.py", line 16, in
@make_array_nonempty.register(pd.StringDtype)
File "/home/harald/python/miniex-env/lib/python3.8/site-packages/pandas/init.py", line 214, in getattr
raise AttributeError("module 'pandas' has no attribute '{}'".format(name))
AttributeError: module 'pandas' has no attribute 'StringDtype'

Hi,
I am trying to run MINI-EX. I get following error:

$ nextflow -C miniex.config run miniex.nf -resume
N E X T F L O W  ~  version 21.10.5
Launching `miniex.nf` [compassionate_wozniak] - revision: b74059cce4
WARN: DSL 2 PREVIEW MODE IS DEPRECATED - USE THE STABLE VERSION INSTEAD -- Read more at https://www.nextflow.io/docs/latest/dsl2.html#dsl2-migration-notes
      Motif-Informed Network Inference from gene EXpression
      =====================================================
Running TF motif enrichment filtering on  TF-F_motifs
      Running single-cell cluster enrichment using the top 700 upregulated genes per cluster
      Filtering out regulons of single-cell clusters where the TF is expressed in less than 10 % of the cells
Plotting expression specificity and DE calls for the top 150 regulons

[-        ] process > run_grnboost         -
[-        ] process > unzip_motifMappings  [  0%] 0 of 1
[-        ] process > run_enricher_motifs  -
[-        ] process > filter_motifs        -
[-        ] process > get_topDEGs          [  0%] 0 of 1
[-        ] process > run_enricher_cluster -
[-        ] process > run_grnboost         -
[-        ] process > unzip_motifMappings  [  0%] 0 of 1
[-        ] process > run_enricher_motifs  -
[-        ] process > filter_motifs        -
[81/dd6326] process > get_topDEGs (1)      [100%] 1 of 1, failed: 1
[-        ] process > run_enricher_cluster -
[-        ] process > filter_expression    -
[-        ] process > clustermap_regs      -
[-        ] process > network_centrality   -
[-        ] process > getFiles_enrichment  -
[-        ] process > GO_enricher          -
[-        ] process > ranking_df_std       -
[-        ] process > makeBorda            -
[-        ] process > heatmap_tops         -
Pulling Singularity image docker://vibpsb/mini-ex:latest [cache /home/pavel/opt/MINI-EX/singularity_cache/vibpsb-mini-ex-latest.img]
Error executing process > 'get_topDEGs (1)'

Caused by:
  Failed to submit process to grid scheduler for execution

Command executed:

  qsub -terse .command.run

Command exit status:
  2

Command output:
  Unknown option: terse
  Usage:
      qsub [-a start_time] [-A account] [-b y|n] [-e err_path] [-I] [-l
      resource_list] [-m mail_options] [-M user_list] [-N job_name] [-o
      out_path] [-p priority] [-pe shm task_cnt] [-P wckey] [-q destination]
      [-r y|n] [-v variable_list] [-V] [-wd workdir] [-W
      additional_attributes] [-h] [script]

I am not sure if this error is regarding Singularity, Nexflow or MINI-EX as I haven't used any of them so far.

Thank you for any help,
Pavel

Is it possible to support for maize Zm-B73-REFERENCE-NAM-5.0？or provide the method to produce annotation file in data_zma?

Thank you very much.

I got unwanted behavior when running with multiple datasets. If one of the datasets is smaller, and thus runs faster, it finishes the make_ref_ranking_dataframe process and make_log_file before another dataset is ready for these steps, and these steps get marked as done. These steps and their succeeding steps are then skipped for the other datasets. (see below for the intermediate output of the pipeline) Rerunning the pipeline (multiple times) with the -resume option does complete the outputs one by one, but sometimes does nothing.

Intermediate output MINI-EX:

N E X T F L O W  ~  version 23.10.0
Launching `[miniex.nf](http://miniex.nf/)` [goofy_hilbert] DSL2 - revision: 05a0ca7fa0
Motif-Informed Network Inference from gene EXpression v2.2
===========================================================
Running TF motif enrichment filtering on TF_motifs
Running single-cell cluster enrichment using the top 700 upregulated genes per cluster
Filtering out regulons of single-cell clusters where the TF is expressed in less than 10 % of the cells
Plotting expression specificity and DE calls for the top 150 regulons

executor >  local (60)
[00/7373df] process > check_user_input (1)           [100%] 1 of 1 ✔
[55/0f30a9] process > get_expressed_genes (3)        [100%] 5 of 5 ✔
[04/4ad1e7] process > unzip_motif_mappings           [100%] 1 of 1 ✔
[d4/744308] process > run_enricher_motifs (5)        [100%] 5 of 5 ✔
[a9/58089a] process > filter_motifs (5)              [ 80%] 4 of 5
[aa/41602b] process > get_top_degs (1)               [100%] 5 of 5 ✔
[5d/09060e] process > run_enricher_cluster (4)       [100%] 4 of 4
[92/507679] process > filter_expression (4)          [100%] 4 of 4
[a0/3758e5] process > make_info_file (4)             [100%] 4 of 4
[0d/e84a82] process > make_regulon_clustermap (4)    [100%] 4 of 4
[1e/352c80] process > get_network_centrality (4)     [100%] 4 of 4
[47/964b1f] process > make_go_enrichment_files (4)   [100%] 4 of 4
[bb/077588] process > run_enricher_go (4)            [100%] 4 of 4
[72/9bdb0a] process > check_reference (4)            [100%] 4 of 4
[4e/727600] process > make_ref_ranking_dataframe (1) [100%] 1 of 1 ✔
[5a/3b2984] process > make_borda (1)                 [100%] 1 of 1
[35/97ba6a] process > score_edges (1)                [100%] 1 of 1
[55/c9d70e] process > make_top_regulons_heatmaps (1) [100%] 1 of 1
[61/d3f1b3] process > make_regmaps (1)               [100%] 1 of 1
[ff/076eb1] process > make_log_file (1)              [100%] 1 of 1 ✔