Dear Ning
Because I am new to Null model , I tried every methods like tNST,pNST,iCAMP,qpen,etc. BUT I got completely different results.
I feed the data of two groups named T and W.
From tNST,i got

From pNST,i got

From iCAMP with the bmin as 72(determined by the maximum RAsig），i got


From qpen (by the way ,i was confused by the function 'qpen.cm' with the para 'metagroup' which i gave the 'treat' like the
other processes, but the qpen.cm was interrupted with an error. AND the reason i use 'qpen.cm' was that I can't get individual
results for each group from 'qpen', which i think i get from 'qpen.test' maybe ? Am i right?) , i got

Sincerely

bin.size.limit = 12
sig.index="Confidence" # see other options in help document of icamp.big.
icres=iCAMP::icamp.big(comm=comm, pd.desc = pd.big$pd.file, pd.spname=pd.big$tip.label,

•                    pd.wd = pd.big$pd.wd, rand = rand.time, tree=tree,
  

•                    prefix = prefix, ds = 0.2, pd.cut = NA, sp.check = TRUE,
  

•                    phylo.rand.scale = "within.bin", taxa.rand.scale = "across.all",
  

•                    phylo.metric = "bMPD", sig.index=sig.index, bin.size.limit = bin.size.limit, 
  

•                    nworker = nworker, memory.G = memory.G, rtree.save = FALSE, detail.save = TRUE, 
  

•                    qp.save = FALSE, detail.null = FALSE, ignore.zero = TRUE, output.wd = save.wd, 
  

•                    correct.special = TRUE, unit.sum = rowSums(comm), special.method = "depend",
  

•                    ses.cut = 1.96, rc.cut = 0.95, conf.cut=0.975, omit.option = "no",meta.ab = NULL)
  

The names are re-ranked.
Now calculating max phylogenetic distance.
----------Now binning-----------------Sun Apr 17 13:30:44 2022
Now computing path. begin at Sun Apr 17 13:30:45 2022. Please wait...
Now computing path. begin at Sun Apr 17 17:37:25 2022. Please wait...
Now computing dist to root. begin at Sun Apr 17 21:43:29 2022. Please wait...
----------Now binning com and sp without omitting small bins-----------------Sun Apr 17 21:45:58 2022
----bMPD bin i=1 in 471 ---- Sun Apr 17 21:45:59 2022
Now calculating observed betaMPD. Begin at Sun Apr 17 21:46:00 2022. Please wait...
Now randomizing by parallel computing. Begin at Sun Apr 17 21:46:03 2022. Please wait...
Now fixing special cases. Begin at Sun Apr 17 22:13:18 2022. Please wait...
----bMPD bin i=2 in 471 ---- Sun Apr 17 22:13:18 2022
Now calculating observed betaMPD. Begin at Sun Apr 17 22:13:19 2022. Please wait...
Now randomizing by parallel computing. Begin at Sun Apr 17 22:13:19 2022. Please wait...
Now fixing special cases. Begin at Sun Apr 17 22:14:25 2022. Please wait...
----bMPD bin i=3 in 471 ---- Sun Apr 17 22:14:26 2022
Now calculating observed betaMPD. Begin at Sun Apr 17 22:14:26 2022. Please wait...
Now randomizing by parallel computing. Begin at Sun Apr 17 22:14:27 2022. Please wait...
Now fixing special cases. Begin at Sun Apr 17 22:15:34 2022. Please wait...
----bMPD bin i=4 in 471 ---- Sun Apr 17 22:15:35 2022
Now calculating observed betaMPD. Begin at Sun Apr 17 22:15:35 2022. Please wait...
Now randomizing by parallel computing. Begin at Sun Apr 17 22:15:35 2022. Please wait...
Now fixing special cases. Begin at Sun Apr 17 22:16:40 2022. Please wait...
----bMPD bin i=5 in 471 ---- Sun Apr 17 22:16:41 2022
Now calculating observed betaMPD. Begin at Sun Apr 17 22:16:41 2022. Please wait...
Now randomizing by parallel computing. Begin at Sun Apr 17 22:16:42 2022. Please wait...
Now fixing special cases. Begin at Sun Apr 17 22:17:45 2022. Please wait...
All match very well.
Now calculating observed MPD. Begin at Sun Apr 17 22:17:46 2022. Please wait...
Now randomizing by parallel computing. Begin at Sun Apr 17 22:17:47 2022. Please wait...
Error in checkForRemoteErrors(val) :
2 nodes produced errors; first error: 'size'参数不对
Dear professor
how we solve this problem?

Hi,
I got the following error when doing phylogenetic binning.

The data run normally for the other three seasons, but this error occurred during the summer. How to resolve it?

Hi Daliang,

My community composition matrix includes 1,000 samples x 550,000 OTUs. I ran the iCAMP analyses on a HPC with 94 CPUs and 470 Gb memory. However, an error message of out-of-memory occurs during the phylogenetic signal test and iCAMP::icamp.big.

Do you have any tips on such a large dataset?

Best,

Chen

Hi Ning,

Thanks for the great package. I went through your articles & the example code, however, I couldn't find how one can find optimal values for ds and bin.size.limit. Appreciate any advice/thoughts

On a related note, can categorical variables be used for identifying niche preferences i.e. used in the environment.txt file? We sampled across depth and as such depth integrates multiple niche dimensions, many of which we did not measure explicitly

Cheers, Adi

Hello Daliang, l followed the script, and when l ran this part :
"binps=iCAMP::ps.bin(sp.bin = sp.bin,sp.ra = sp.ra,
spname.use = spname.use,
pd.desc = pd.big$pd.file, pd.spname = pd.big$tip.label, pd.wd = pd.big$pd.wd,
nd.list = niche.dif$nd,nd.spname = nd.spname,ndbig.wd = niche.dif$nd.wd,
cor.method = "kendall",r.cut = 0.1, p.cut = 0.05, min.spn = 10)

The error message is "Error in cor(as.vector(xdis), ydis, method = method, use = use) :
missing observations in cov/cor"

How could i fix this?

I get this error while I try to find optimal values for ds and bin.size.limit.

Error in dimnames(x) <- dn : length of 'dimnames' [2] not equal to array extent

traceback()
2: colnames<-(*tmp*, value = *vtmp*)
1: iCAMP::ps.bin(sp.bin = sp.bin, sp.ra = sp.ra, spname.use = spname.use,
pd.desc = pd.big$pd.file, pd.spname = pd.big$tip.label, pd.wd = pd.big$pd.wd,
nd.list = niche.dif$nd, nd.spname = niche.dif$names, ndbig.wd = niche.dif$nd.wd,
cor.method = "pearson", r.cut = 0.1, p.cut = 0.05, min.spn = 5)

Here is the code for this test.

ds = 0.2 # setting can be changed to explore the best choice
bin.size.limit = 5# setting can be changed to explore the best choice. # here set as 5 just for the small example dataset. For real data, usually try 12 to 48.
phylobin=taxa.binphy.big(tree = tree, pd.desc = pd.big$pd.file,pd.spname = pd.big$tip.label,
                         pd.wd = pd.big$pd.wd, ds = ds, bin.size.limit = bin.size.limit,
                         nworker = nworker)

# 8.2 # test within-bin phylogenetic signal.
sp.bin=phylobin$sp.bin[,3,drop=FALSE]
sp.ra=colMeans(comm/rowSums(comm))
abcut=3 # you may remove some species, if they are too rare to perform reliable correlation test.
commc=comm[,colSums(comm)>=abcut,drop=FALSE]
dim(commc)
spname.use=colnames(commc)
binps=iCAMP::ps.bin(sp.bin = sp.bin,sp.ra = sp.ra,spname.use = spname.use,
                    pd.desc = pd.big$pd.file, pd.spname = pd.big$tip.label, pd.wd = pd.big$pd.wd,
                    nd.list = niche.dif$nd,nd.spname = niche.dif$names,ndbig.wd = niche.dif$nd.wd,
                    cor.method = "pearson",r.cut = 0.1, p.cut = 0.05, min.spn = 5)

traceback()

But it ran smoothly if I set bin.size.limit more than 11. About this test, I have a phylogenetic tree with 2718 tips. However, I want to combine their tips with phylogenetic bins (deeper levels) in order to make a reduced phylogeny for analysis of ecological processes. Many thanks for your help in advance.
Test_size_11.PhyloSignalDetail.csv
Test_size_11.PhyloSignalSummary.csv

Hi Daliang,
While running iCAMP pipeline (icamp.test.r) on my own dataset it is failing to match OTU IDs (Step #5). However, I am not sure why this issue comes up since all tree tip labels match the OTU names in the count and taxonomy tables.

> summary(tree$tip.label %in% colnames(comm))
  Mode    TRUE 
logical   22003

> summary(tree$tip.label %in% (clas$SpeciesID))
   Mode    TRUE 
logical   22003 

Any suggestions on what to change?

I generated the nwk tree in QIIME2, imported it into R and done some filtering using phyloseq. I exported the final tree using the phy_tree() function from the phyloseq package and saved it as .nwk with write.tree() function from the ape package.

Good afternoon,
I installed iCAMP in R (v. 4.2.1 (2022-06-23) in LINUX) and tested the install with the example data. I am using Rstudio and connecting with a remote server. Everything works until step 9 (the iCAMP analysis) where I get the message

Error in checkForRemoteErrors(val) :
4 nodes produced errors; first error: there is no package called 'iCAMP'

The entire output to the console is:

All match very well.
----------Now binning-----------------Mon Jan 9 15:47:47 2023
Now computing path. begin at Mon Jan 9 15:47:48 2023. Please wait...
Now computing path. begin at Mon Jan 9 15:47:49 2023. Please wait...
Now computing dist to root. begin at Mon Jan 9 15:47:50 2023. Please wait...
----------Now binning com and sp without omitting small bins-----------------Mon Jan 9 15:47:50 2023
----bMPD bin i=1 in 3 ---- Mon Jan 9 15:47:51 2023
Set of permutations < 'minperm'. Generating entire set.
Now calculating observed betaMPD. Begin at Mon Jan 9 15:47:51 2023. Please wait...
Now randomizing by parallel computing. Begin at Mon Jan 9 15:47:51 2023. Please wait...
Error in checkForRemoteErrors(val) :
4 nodes produced errors; first error: there is no package called 'iCAMP'

I am using the example data and the example code as provided. Is there a known issue with the version of R that fails to find iCAMP? I have tried a reinstall (that gave the same error). I also tried using one core (still failed).

All help/insights gratefully received.

P

Dear developer, I am an administrator of an HPC cluster. One of our users is running iCAMP on a cluster system running SLURM and ubuntu 22.04, with an installation of R 4.3.0. We notice that whenever iCAMP is called, it spawns many threads, some of which show a status of "D" (uninterruptable sleep). This has caused the NFS filesystem to be non responsive which affects other users. Do iCAMP or any of its dependent packages place file locks? If so on which file(s) and what happens if we disable file locking on the NFS partition? Thank you!

Dear author, when I use the code of iCMAP ，the error occurred:
“ icamp.out=icamp.big(comm=comm,tree=tree,#pd.wd=pd.wd,

•                 rand=999, nworker=4,bin.size.limit=12,ds=0.2)
  

The names are re-ranked.
----------Now binning-----------------Sun Feb 12 16:12:40 2023
Now computing path. begin at Sun Feb 12 16:12:40 2023. Please wait...
Now computing path. begin at Sun Feb 12 16:12:44 2023. Please wait...
Now computing dist to root. begin at Sun Feb 12 16:12:45 2023. Please wait...
----------Now binning com and sp without omitting small bins-----------------Sun Feb 12 16:12:47 2023
----bMPD bin i=1 in 37 ---- Sun Feb 12 16:12:48 2023
Now calculating observed betaMPD. Begin at Sun Feb 12 16:12:48 2023. Please wait...
Now randomizing by parallel computing. Begin at Sun Feb 12 16:12:48 2023. Please wait...
Now fixing special cases. Begin at Sun Feb 12 16:13:07 2023. Please wait...
All match very well.
Now calculating observed MPD. Begin at Sun Feb 12 16:13:12 2023. Please wait...
Now randomizing by parallel computing. Begin at Sun Feb 12 16:13:12 2023. Please wait...
Error in checkForRemoteErrors(val) :
4 nodes produced errors; first error: 'size'参数不对
此外: There were 11 warnings (use warnings() to see them) ”

I don't know wha't wrong with the parameters, my data have same formation with example data, and I already adjusted the parameter but the issue not addressed.Do you have any suggestions?

Dear Ning,

It would be great to have a function which can correlate bin-level ecological processes (as you do in your paper) with environmental variables using Mantel tests/MRM

Cheers, Adi

Dear Dr. Ning
I would like to use the package iCAMP to analyze my own data. However, I don't know how to get the category information. Core, others, and rare OTU definitions were based on their relative abundance from the OTU table ? What is the threshold value for core, others, or rare otu?

Best wishes

Long

Thanks for the great package! I've run this on a dataset before without issue. However, I am having difficulty running the icamp.big() command with my current dataset. The output says:

----------Now binning-----------------Fri Feb 11 13:02:56 2022
Now computing path. begin at Fri Feb 11 13:03:04 2022. Please wait...
Now computing path. begin at Fri Feb 11 13:06:18 2022. Please wait...
Now computing dist to root. begin at Fri Feb 11 13:06:32 2022. Please wait...

----------Now binning com and sp without omitting small bins-----------------Fri Feb 11 13:08:22 2022
----bMPD bin i=1 in 13 ---- Fri Feb 11 13:08:23 2022
Now calculating observed betaMPD. Begin at Fri Feb 11 13:08:48 2022. Please wait...
Now randomizing by parallel computing. Begin at Fri Feb 11 13:11:32 2022. Please wait...
Error in serialize(data, node$con) : ignoring SIGPIPE signal
Calls: ... postNode -> sendData -> sendData.SOCKnode -> serialize
Execution halted

It appears the task has run out of memory. Does this seem to be the case? I am running this on 32 cores with 350 G of memory. The dataset is large, but not exceptionally so (87 samples and 39675 ASVs). Is it just a matter of giving the job more memory, or is there perhaps a different underlying issue?

A portion of my code is presented below. Thanks!

nworker = 32
memory.G = 350
rand.time = 1000
bin.size.limit = 24
sig.index="Confidence"

icres=iCAMP::icamp.big(comm=comm, pd.desc = pd.big$pd.file, pd.spname=pd.big$tip.label,
     pd.wd = pd.big$pd.wd, rand = rand.time, tree=tree,
     prefix = prefix, ds = 0.2, pd.cut = NA, sp.check = TRUE,
     phylo.rand.scale = "within.bin", taxa.rand.scale = "across.all",
     phylo.metric = "bMPD", sig.index=sig.index, bin.size.limit = bin.size.limit,
     nworker = nworker, memory.G = memory.G, rtree.save = FALSE, detail.save = TRUE,
     qp.save = FALSE, detail.null = FALSE, ignore.zero = TRUE, output.wd = save.wd,
     correct.special = TRUE, unit.sum = rowSums(comm), special.method = "depend",
     ses.cut = 1.96, rc.cut = 0.95, conf.cut=0.975, omit.option = "no",meta.ab = NULL)

Hi Ning,

It would be great if other normalisation measures (e.g. hellinger or log) can be made available across the iCAMP R functions which calculate beta diversity. I am looking to apply the hellinger transformations and as such down-weight highly abundant species in my samples

Regards, Adi

This package ran excellently on a previous version of R, but after updating to 4.03 and reinstalling packages, I am now getting the following error:

Error in makePSOCKcluster(names = spec, ...) :
Cluster setup failed. 4 of 4 workers failed to connect.

I tried saw on a different page this issue can be resolved by specifying clusters:
parallel::makeCluster(4, setup_strategy = "sequential")

But this did not seem to work for me.

Thanks!

----bMPD bin i=1 in 1 ---- Wed Nov 23 22:31:43 2022
Now calculating observed betaMPD. Begin at Wed Nov 23 22:31:43 2022. Please wait...
Now randomizing by parallel computing. Begin at Wed Nov 23 22:31:45 2022. Please wait...
Now fixing special cases. Begin at Wed Nov 23 22:31:53 2022. Please wait...
Error in h(simpleError(msg, call)) :
error in evaluating the argument 'x' in selecting a method for function 'as.matrix': subscript out of bounds

when using
icamp.out <- icamp.big(comm = comm, tree = tree,
pd.wd = getwd(),
ses.cut = 1.96,
rc.cut = 0.95,
bin.size.limit = 5,
rand = 99, nworker = 4)

Hi Ning,

How would you suggest using iCAMP to deal with a dataset comprising groups of samples from four different metacommunities? The group comparisons within each metacommunity is the same, just that I would like to treat the turnovers within each one separately and then compare them at the end. Thanks for your time

Regards, Adi

Dear professor Ning,

Hope everthing is going well with you.

when I run ，the error occurred:
"> icbin=iCAMP::icamp.bins(icamp.detail = icres$detail,treat = treat,

•              clas=clas,silent=FALSE, boot = TRUE,
  

•              rand.time = rand.time,between.group = TRUE)
  

Now summarizing method=CbMPDiCbraya i=1 j=1. Mon Feb 27 21:43:00 2023
bootstrapping rt=1. Mon Feb 27 21:43:00 2023
bootstrapping rt=201. Mon Feb 27 21:43:01 2023
bootstrapping rt=401. Mon Feb 27 21:43:02 2023
bootstrapping rt=601. Mon Feb 27 21:43:03 2023
bootstrapping rt=801. Mon Feb 27 21:43:04 2023
Now summarizing method=CbMPDiCbraya i=1 for between.group j=1 v=2. Mon Feb 27 21:43:05 2023
Error in oldname[j, 1:2] : subscript out of bounds"

I don't know wha't wrong with it. Do you have any suggestions? Thank you very much!

Best regards,

xiaojie

Hi Daliang,
When I used the iCAMP package to calculate betaMPD, some problems happened. I have set the thread numbers before I used the R code. Typed the code like this: export OPENBLAS_NUM_THREADS=10
However, the problems are still here. I have no idea how to deal with these problems.

the error information is following:
----bMPD bin i=14 in 200 ---- Sun Sep 26 01:49:55 2021
Now calculating observed betaMPD. Begin at Sun Sep 26 01:49:55 2021. Please wait...
Now randomizing by parallel computing. Begin at Sun Sep 26 01:49:58 2021. Please wait...
Now fixing special cases. Begin at Sun Sep 26 01:52:01 2021. Please wait...
All match very well.
Now calculating observed MPD. Begin at Sun Sep 26 01:52:01 2021. Please wait...
Now randomizing by parallel computing. Begin at Sun Sep 26 01:52:04 2021. Please wait...
OpenBLAS blas_thread_init: pthread_create failed for thread 76 of 80: 资源暂时不可用
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 6191435 max
OpenBLAS blas_thread_init: pthread_create failed for thread 77 of 80: 资源暂时不可用
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 6191435 max
OpenBLAS blas_thread_init: pthread_create failed for thread 78 of 80: 资源暂时不可用
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 6191435 max
OpenBLAS blas_thread_init: pthread_create failed for thread 79 of 80: 资源暂时不可用
OpenBLAS blas_thread_init: RLIMIT_NPROC 4096 current, 6191435 max
(END)

Dear Ning,

I tried reproducing results from your Nat. Comm. paper. I get 658 bins (ds = 0.2, bin.size.limit = 12) as you report in the paper. However, when I had a look at the maximum and mean phylogenetic distances, it looks like a good number of bins have a fair number of OTUs which lie outside the signal threshold as a result of merging bins. In this case, will bNRI still provide reliable results?

Regards, Adi

Dear Ning
I tried to use the following R code to quantify the assembly process of microbial communities: wd0 = getwd()
save.wd = paste0(tempdir(),"/pdbig")
nworker=6
rand.time = 1000
bin.size.limit = 24
otu <- read.table("otutab.txt", header = TRUE, row.names = 1, stringsAsFactors = FALSE, comment.char = "")
tree <- ape::read.tree("otus.nwk")
library(iCAMP)
icamp.out <- icamp.big(comm = otu,tree = tree,pd.wd = save.wd,rand = rand.time,nworker = nworker,bin.size.limit = bin.size.limit,ds = 0.2 ). But it failed, and the error message was as follows, "unexpected name: DMH40 DMH41 DMH42 DMH44
The names are re-ranked.
----------Now binning-----------------Sat Dec 9 12:54:23 2023
Error in UseMethod("is.rooted") :
'is.rooted' has no method available for 'NULL' target object
Calls: icamp.big -> -> is.rooted".
I tried to find the answer on various websites, but no one encountered a similar situation. If you have time, can you help me find this error information
Thank you very much,
yang

An error occurred before the updated version.
When I updated to the latest version, I still had problems

`otu <- otu_table(ps_rs_filtered) %>% data.frame(.) %>% t() #row is sampleID
tax <- tax_table(ps_rs_filtered) %>% data.frame(.)
rand.time=1000
nworker=30
memory.G=200
sampid.check=match.name(rn.list=list(otu=otu,treat=treat)) #Mismatch warning: treat.rowname has 24 mismatched names.
treat=sampid.check$treat
otu=sampid.check$otu
otu=otu[,colSums(otu)>0,drop=FALSE]
ncol(otu)# 32778
nrow(otu)# 72

spid.check=match.name(cn.list=list(otu=otu),rn.list=list(tax=tax),tree.list=list(tree=tree)) #Mismatch warning: tax.rowname has 45320 mismatched names.
#The names are re-ranked.
Warning message:
In ape::drop.tip(tr, rm.tip) : drop all tips of the tree: returning NULL

otu=spid.check$otu
tax=spid.check$tax
tree=spid.check$tree

setwd(save.wd)
set.seed(12345)
if(!file.exists("pd.desc"))
{
pd.big=iCAMP::pdist.big(tree = tree, wd=save.wd, nworker = nworker, memory.G = memory.G)
}else{
pd.big=list()
pd.big$tip.label=read.csv(paste0(save.wd,"/pd.taxon.name.csv"),row.names = 1,stringsAsFactors = FALSE)[,1]
pd.big$pd.wd=save.wd
pd.big$pd.file="pd.desc"
pd.big$pd.name.file="pd.taxon.name.csv"
}

save(pd.big,file = "pd.big.rda")
save.image(file = "icamp.Rdata")

#Now computing path. begin at Sun Jul 17 20:48:19 2022. Please wait...
Error in checkForRemoteErrors(val) :
2 nodes produced errors; first error: subscript out of bounds

`

Dear Ning

I am trying to use this package to estimate microbial community assembly, but when I run taxa.binphy.big, the value of bin.id.new for each OTU is "NA". I also tried icamp.big, but an error message "Error in com.bin[[i]]: attempt to select less than one element in integerOneIndex." occurred. Hope you could help me to fix the error. Thank you for your time!

Here is the code I used:
ds = 0.2
bin.size.limit = 12
sig.index="Confidence"
phylobin=taxa.binphy.big(tree = tree, pd.desc = pd.big$pd.file,pd.spname = pd.big$tip.label,
pd.wd = pd.big$pd.wd, ds = ds, bin.size.limit = bin.size.limit)
sp.bin=phylobin$sp.bin[,3,drop=FALSE]
sp.bin
icres=iCAMP::icamp.big(comm=comm, pd.desc = pd.big$pd.file, pd.spname=pd.big$tip.label,
pd.wd = pd.big$pd.wd, rand = 999, tree=tree,
ds = 0.2, pd.cut = NA, sp.check = TRUE,
phylo.rand.scale = "within.bin", taxa.rand.scale = "across.all",
phylo.metric = "bMPD", sig.index=sig.index, bin.size.limit = bin.size.limit,
rtree.save = FALSE, detail.save = TRUE,
qp.save = FALSE, detail.null = FALSE, ignore.zero = TRUE, output.wd = save.wd,
correct.special = TRUE, unit.sum = rowSums(comm), special.method = "depend",
ses.cut = 1.96, rc.cut = 0.95, conf.cut=0.975, omit.option = "no",meta.ab = NULL)

Besides, for QPEN calculation, there is no qpen.test() function for significane test.

Hi Daliang,
When I run

9.5 # input community matrix as relative abundances (values < 1) rather than counts

comra=comm/rowSums(comm)
prefixra=paste0(prefix,"RA")
bin.size.limit = 24 # For real data, usually use a proper number according to phylogenetic signal test or try some settings then choose the reasonable stochasticity level. our experience is 12, or 24, or 48. but for this example dataset which is too small, have to use 5.
icres6=iCAMP::icamp.big(comm=comra,tree=tree,pd.desc=pd.big$pd.file, pd.spname=pd.big$tip.label, pd.wd=pd.big$pd.wd,
rand=rand.time,prefix=prefixra,ds=0.2,pd.cut=NA,sp.check=TRUE,
phylo.rand.scale="within.bin",taxa.rand.scale="across.all",
phylo.metric="bMPD",sig.index="Confidence",
bin.size.limit=bin.size.limit,nworker=nworker,memory.G=memory.G,
rtree.save=FALSE,detail.save=TRUE,qp.save=FALSE,detail.null=FALSE,
ignore.zero=TRUE,output.wd=save.wd,correct.special=TRUE,unit.sum=rowSums(comra),
special.method="depend",ses.cut = 1.96,rc.cut = 0.95,conf.cut=0.975,
omit.option="no",meta.ab=NULL, taxo.metric="bray", transform.method=NULL,
logbase=2, dirichlet=TRUE)

Error shows" 4 nodes produced errors; first error: missing values are not allowed with argument 'na.rm = false' "
How can I solve it? Thank you very much!

Hi Daliang,

I try to quantify the community assembly processes by using the following code:

setwd(save.wd) if(!file.exists("pd.desc")) { pd.big=iCAMP::pdist.big(tree = tree, wd=save.wd, nworker = nworker, memory.G = memory.G) }else{ pd.big=list() pd.big$tip.label=read.csv(paste0(save.wd,"/pd.taxon.name.csv"),row.names = 1,stringsAsFactors = FALSE)[,1] pd.big$pd.wd=save.wd pd.big$pd.file="pd.desc" pd.big$pd.name.file="pd.taxon.name.csv" }

iCAMP::qpen.cm(comm=comm,pd=pd.big$pd.file,pd.big.wd=pd.big$pd.wd, pd.big.spname=pd.big$tip.label,ab.weight=TRUE, rand.time=rand.time, nworker=nworker,project=prefix, wd=save.wd, save.bNTIRC=TRUE, meta.group = metadata[,"habitat2",drop=F])

However, I encountered the following error:

Setting parallel cluster for path computing cost 1.400004 secs. Tue Dec 19 13:42:17 2023
Parallel for 600 tips cost 1.988125 mins. Tue Dec 19 13:44:16 2023
Path computing by parallel may take 2.19445240951909 hours. Tue Dec 19 13:44:18 2023
Now computing path for the rest 39666 tips. begin at Tue Dec 19 13:44:19 2023. Please wait...
Computing path for the rest 39666 tips actually took 2.771099 mins. Tue Dec 19 13:47:04 2023
Now setting big matrix file on the disk. Tue Dec 19 13:47:05 2023
Setting parallel cluster for pdist computing cost 1.330648 secs. Tue Dec 19 13:47:20 2023
Parallel computing Pdist for the first 600 runs cost 2.446895 mins. Tue Dec 19 13:49:47 2023
The rest Pdist computing by parallel may take 1.33220872485118 hours. Tue Dec 19 13:49:47 2023
Computing pdist for the rest 39666 tips actually took 16.60026 mins. Tue Dec 19 14:06:23 2023
All match very well.
The names are re-ranked.
All match very well.
All match very well.
Now calculating observed betaMNTD. Begin at Tue Dec 19 14:06:33 2023. Please wait...
Now randomizing by parallel computing. Begin at Tue Dec 19 14:18:16 2023. Please wait...
Error in checkForRemoteErrors(val) :
120 nodes produced errors; first error: incorrect number of subscripts on matrix
Calls: ... clusterApply -> staticClusterApply -> checkForRemoteErrors
In addition: Warning message:
'memory.size()' is Windows-specific
Execution halted

Dear Dr. Ning,

I am curious as to why I am getting different observed Bray-Curtis dissimilarity values when using vegdist (method = "bray) and RC.pc(taxo.metric = "bray").

If you know what might be going on, I would appreciate hearing your thoughts.

Thank you,
Magda

Code using the example data from the iCAMP package:

data("example.data")
comm=example.data$comm
rand.time=20
nworker=2

RC=RC.pc(comm=comm, rand = rand.time,
nworker = nworker, weighted = TRUE,
output.bray = TRUE,
taxo.metric = "bray",
sig.index="RC")

RC$BC.obs

vegdist(comm, method = "bray")

Hi,
I am working through the SimpleOTU example and I cannot find the category.txt file. Is there any problem with it? How do you define the three categories? I searched in the icamp.test file and did not find anything. I also looked for it in the published article and its R scripts but I cannot find anything.

Thank you very much for your help and this nice library

Manuel

Hi,

Really sorry as I'm sure I am missing something very obvious - but I can't find the help documents. For example, in 8.2 it says '# see help document of the function "ps.bin" for the meaning of output' and in step 9 it says 'ig.index="Confidence" # see other options in help document of icamp.big'.

Would you be able to point me in the direction of the help documents.

Thanks,
Josh

Hi @DaliangNing ,

I'm using the RC.pc function witch works well on my raw data. But when i rarefy and normalized my data with log and hellinger, I have the error:

Error in checkForRemoteErrors(val) :
4 nodes produced errors; first error: invalid 'size' argument

I've tried with the phyloseq data set Globalpatternsand i'm facing the same issue.
Do you know how to solve this problem?

library(vegan)
library(phyloseq)
library(iCAMP)
data(GlobalPatterns)
sort(sample_sums(GlobalPatterns))
GlobalPatterns <- rarefy_even_depth(GlobalPatterns,sample.size = 50000)
count <- t(as(otu_table(GlobalPatterns), "matrix"))
count <- log1p(count)
count <- as.data.frame(decostand (count, 'hellinger'))
RCbray <- RC.pc(count, rand = 1000, weighted = TRUE)

Dear Ning:
In my analysis,my treat file has juet one OTU in each group.When I use icamp package to calculate icamp.big,R will feedback error:

Error in [.data.frame(weight, (sig.phy > sig.phy.cut | sig.phy2 > sig.phy2.cut)) :
dims [product 2]与对象长度[11]不匹配

But when I add OTU to each group and make sure the number of OTUs in each gruop greater or eaqul to 2,the program can calculate smoothly.

Why is this happening?

Hi,

Can I use this tool to analyze ITS data? I know that building phylogenetic tree using ITS data is problematic because the length of ITS region is highly variable.

Dear Ning,
I get this error while I try to ran the example code.

the error:

Error in CreateFileBackedBigMatrix(as.character(backingfile), as.character(backingpath), :
Problem creating filebacked matrix.

the example code:

data("example.data")
comm=example.data$comm
tree=example.data$tree

save.wd=tempdir()
pd.wd=paste0(save.wd,"/pdbig")
nworker=4
rand.time=20

bin.size.limit=5

setwd(save.wd)
icamp.out=icamp.big(comm=comm,tree=tree,pd.wd=pd.wd,
rand=rand.time, nworker=nworker,
bin.size.limit=bin.size.limit)

Dear Ning,

The icamp.cate() function uses Wtuvk$name1 and Wtuvk$name2 to refer to the sample IDs. However the results from the corresponding icamp.bins() seems to name those two columns as Wtuvk$samp1 and Wtuvk$samp2 instead. The same issue also occurred in the local variable Ptuvct. I'm not 100% sure if I understand the code correctly but I've tried replacing all the $name* stuff with samp* in the code by fix() and that worked.

Hi Professor Ning,
When I run "# 6 # calculate pairwise phylogenetic distance matrix.":

since microbial community data usually has a large number of species (OTUs or ASVs), we use "big.matrix" in R package "bigmemory" to handle the large phylogenetic distance matrix.

setwd(save.wd)
if(!file.exists("pd.desc"))

• {
• pd.big=iCAMP::pdist.big(tree = tree, wd=save.wd, nworker = nworker, memory.G = memory.G)

• output files:

• path.rda: a R object to list all the nodes and edge lengthes from root to every tip. saved in R data format. an intermediate output when claculating phylogenetic distance matrix.

• pd.bin: BIN file (backingfile) generated by function big.matrix in R package bigmemory. This is the big matrix storing pairwise phylogenetic distance values. By using this bigmemory format file, we will not need memory but hard disk when calling big matrix for calculation.

• pd.desc: the DESC file (descriptorfile) to hold the backingfile (pd.bin) description.

• pd.taxon.name.csv: comma delimited csv file storing the IDs of tree tips (OTUs), serving as the row/column names of the big phylogenetic distance matrix.

• }else{

• if you already calculated the phylogenetic distance matrix in a previous run

• pd.big=list()
• pd.big$tip.label=read.csv(paste0(save.wd,"/pd.taxon.name.csv"),row.names = 1,stringsAsFactors = FALSE)[,1]
• pd.big$pd.wd=save.wd
• pd.big$pd.file="pd.desc"
• pd.big$pd.name.file="pd.taxon.name.csv"
• }
  Now computing path. begin at Wed Mar 8 13:12:29 2023. Please wait...
  Error in checkForRemoteErrors(val) :
  2 nodes produced errors; first error: subscript out of bounds
  In addition: Warning message:
  'memory.limit()' is no longer supported

Error shows" Error in checkForRemoteErrors(val) : 2 nodes produced errors; first error: subscript out of bounds"

I don't know what's wrong with the parameters. How can I solve it? Thank you very much!

Best regards,

Xiaojie

category.txt
classification.txt
otus.txt
treat2col.txt

The function icamp.big fails when restricting it to only strict bins with the error Invalid 'type' (list) of argument in R
This seems to because of how the community table is subsetted based on the strict bin IDs.

bin.lev = levels(as.factor(sp.bin[, 1]))
bin.num = length(bin.lev)
com.bin = lapply(1:bin.num, function(i) {
comm[, match(rownames(sp.bin)[which(sp.bin == i)], colnames(comm))]
  })

should instead be (if I am not mistaken)

bin.lev = levels(as.factor(sp.bin[, 1]))
bin.num = length(bin.lev)
com.bin = lapply(1:bin.num, function(i) {
comm[, match(rownames(sp.bin)[which(sp.bin == bin.lev[i])], colnames(comm))]
  })

Hi,
Metagenomic data or other OTU/ASV beyond 16S could be used for iCAMP? Thank you!

hello, I get the same error. I check the node number in the tree, the number isn't abnormally low compared to the tip number.

the code like this

----bMPD bin i=15 in 435 ---- Tue Aug 1 21:42:18 2023
Now calculating observed betaMPD. Begin at Tue Aug 1 21:42:18 2023. Please wait...
Now randomizing by parallel computing. Begin at Tue Aug 1 21:43:31 2023. Please wait...
Now fixing special cases. Begin at Tue Aug 1 21:44:54 2023. Please wait...
All match very well.
Now calculating observed MPD. Begin at Tue Aug 1 21:44:55 2023. Please wait...
Error in makePSOCKcluster(names = spec, ...) :
Cluster setup failed. 15 of 15 workers failed to connect.
Error in makePSOCKcluster(names = spec, ...) :
Cluster setup failed. 14 of 15 workers failed to connect.
Now randomizing by parallel computing. Begin at Tue Aug 1 21:47:54 2023. Please wait...
Error in serialize(data, node$con) : ignoring SIGPIPE signal

Can you give me some advices?

Hi there,
Is it necessary to rarify the count data based on sample containing lowest number of sequences before starting iCAMP analysis ?

Regards,
Dinesh

Hi,
When I set the bin.size.limit parameters to12 and 72, their results were the same and very different from pNST. But the pNST result is more consistent with the inferences of my other results.

Therefore, I would like to know if the result can be closer to the results of pNST by changing the parameters of bin.size.limit to 36 and 48.

In addition, if I want to use the results of pNST, how can I get the contributions of various processes of rare or Abundant species like ICAMP pipeline step 15 : summarize core, rare, and other taxa.

It seems like something wrong happened in ''icres=iCAMP::icamp.big''.

icres=iCAMP::icamp.big(comm=comm, pd.desc = pd.big$pd.file, pd.spname=pd.big$tip.label,
pd.wd = pd.big$pd.wd, rand = rand.time, tree=tree,
prefix = prefix, ds = 0.2, pd.cut = NA, sp.check = TRUE,
phylo.rand.scale = "within.bin", taxa.rand.scale = "across.all",
phylo.metric = "bMPD", sig.index=sig.index, bin.size.limit = bin.size.limit,
nworker = nworker, rtree.save = FALSE, detail.save = TRUE,
qp.save = FALSE, detail.null = FALSE, ignore.zero = TRUE, output.wd = save.wd,
correct.special = TRUE, unit.sum = rowSums(comm), special.method = "depend",
ses.cut = 1.96, rc.cut = 0.95, conf.cut=0.975, omit.option = "no",meta.ab = NULL)

----bMPD bin i=27 in 67 ---- Wed Nov 1 18:25:33 2023
Now calculating observed betaMPD. Begin at Wed Nov 1 18:25:33 2023. Please wait...
Now randomizing by parallel computing. Begin at Wed Nov 1 18:25:35 2023. Please wait...
Now fixing special cases. Begin at Wed Nov 1 18:27:02 2023. Please wait...
All match very well.
Now calculating observed MPD. Begin at Wed Nov 1 18:27:02 2023. Please wait...
Now randomizing by parallel computing. Begin at Wed Nov 1 18:27:04 2023. Please wait...
Error in checkForRemoteErrors(val) :
4 nodes produced errors; first error: invalid 'size' argument

I have modified the format of ASV (absolute abundance, integers) table. However, the same error is still present.
I would greatly appreciate it if you could help me solve the error. Thanks for your time.

When I run the line of code: ieggr::save.file(t(comm),filename = "16s-otu.resamp")
there is an error: Error in library(ieggr) : 不存在叫‘ieggr’这个名字的程辑包

I have tried changed the version of R and installed the package in a variety of ways, but all falied. It seems there is no file of the package ieggr in github or other website. I would very appreciate it if you could provide me with the local files of package "ieggr" .

Hi Ning,

When comm is the raw counts matrix (i.e. no rarefaction or total sum scaling), how is abundance weighting done when bMNTD & bNTI are calculated? From the bMNTD function, I see that when abundance.weighted = TRUE, the counts are transformed into relative proportions before being multiplied with MNTD. Is my reading of how the function works correct?

Regards, Adi

Hi
Great package!
I found that there may be a unnecessary variable "c1" in Line 95 lapply function in bNTI.big.r file. This may lead to an error when nworker=1.

Chi

Hi,
i got some confusions about my results "ALL.QPEN.Bootstrapping.Summary.csv"

in this figure, the management group "S", which relative importance of assembly process is zero except HoS, but it seems irrational?
if it is needed, i can upload extra materials for checking

Best regards,

hello, when i run icamp.bins (# 10 # iCAMP bin level statistics), i met the following error: Error in oldname[j, 1:2] : subscript out of bounds
Calls: icamp.bins -> -> t -> sapply -> lapply -> FUN
Execution halted

The nohup.out:
nohup: ignoring input
Now summarizing method=CbMPDiCbraya i=1 j=1. Sun Feb 20 21:25:44 2022
bootstrapping rt=1. Sun Feb 20 21:25:45 2022
bootstrapping rt=201. Sun Feb 20 21:26:04 2022
bootstrapping rt=401. Sun Feb 20 21:26:22 2022
bootstrapping rt=601. Sun Feb 20 21:26:42 2022
bootstrapping rt=801. Sun Feb 20 21:27:00 2022
Now summarizing method=CbMPDiCbraya i=1 j=2. Sun Feb 20 21:27:19 2022
bootstrapping rt=1. Sun Feb 20 21:27:20 2022
bootstrapping rt=201. Sun Feb 20 21:27:38 2022
bootstrapping rt=401. Sun Feb 20 21:27:57 2022
bootstrapping rt=601. Sun Feb 20 21:28:16 2022
bootstrapping rt=801. Sun Feb 20 21:28:35 2022
Now summarizing method=CbMPDiCbraya i=1 j=3. Sun Feb 20 21:28:54 2022
bootstrapping rt=1. Sun Feb 20 21:28:54 2022
bootstrapping rt=201. Sun Feb 20 21:29:13 2022
bootstrapping rt=401. Sun Feb 20 21:29:32 2022
bootstrapping rt=601. Sun Feb 20 21:29:51 2022
bootstrapping rt=801. Sun Feb 20 21:30:10 2022
Now summarizing method=CbMPDiCbraya i=1 j=4. Sun Feb 20 21:30:29 2022
bootstrapping rt=1. Sun Feb 20 21:30:29 2022
bootstrapping rt=201. Sun Feb 20 21:30:48 2022
bootstrapping rt=401. Sun Feb 20 21:31:07 2022
bootstrapping rt=601. Sun Feb 20 21:31:25 2022
bootstrapping rt=801. Sun Feb 20 21:31:44 2022
Now summarizing method=CbMPDiCbraya i=1 j=5. Sun Feb 20 21:32:03 2022
bootstrapping rt=1. Sun Feb 20 21:32:04 2022
bootstrapping rt=201. Sun Feb 20 21:32:22 2022
bootstrapping rt=401. Sun Feb 20 21:32:42 2022
bootstrapping rt=601. Sun Feb 20 21:33:01 2022
bootstrapping rt=801. Sun Feb 20 21:33:19 2022
Now summarizing method=CbMPDiCbraya i=1 j=6. Sun Feb 20 21:33:38 2022
bootstrapping rt=1. Sun Feb 20 21:33:39 2022
bootstrapping rt=201. Sun Feb 20 21:33:58 2022
bootstrapping rt=401. Sun Feb 20 21:34:16 2022
bootstrapping rt=601. Sun Feb 20 21:34:35 2022
bootstrapping rt=801. Sun Feb 20 21:34:54 2022
Now summarizing method=CbMPDiCbraya i=1 j=7. Sun Feb 20 21:35:12 2022
bootstrapping rt=1. Sun Feb 20 21:35:13 2022
bootstrapping rt=201. Sun Feb 20 21:35:32 2022
bootstrapping rt=401. Sun Feb 20 21:35:51 2022
bootstrapping rt=601. Sun Feb 20 21:36:10 2022
bootstrapping rt=801. Sun Feb 20 21:36:29 2022
Now summarizing method=CbMPDiCbraya i=1 j=8. Sun Feb 20 21:36:48 2022
Error in oldname[j, 1:2] : subscript out of bounds
Calls: icamp.bins -> -> t -> sapply -> lapply -> FUN
Execution halted

I am wondering where the problem lays. Looking forward for your reply. Big thanks!

Hi,

I tried to find an output about the each ecological process of each bin between each two samples but failed. I found the results of the file named icbin$Wtuvk turned out to be pretty much what I was looking for, but they only showed the names of dominant process, not the detailed values. Does the output exist to get what I want? If not, how do I calculate to get them.

Sincerely

Hi Daliang,

I'm sorry to bother you, but I would like to ask you about two questions that have been troubling me for a long time

The first question is that can I perform separate iCAMP analysis for each group's data by setting the 'treat' parameter in 'iCAMP:bins' to NULL? Also, I want to ask if it's valid to describe the results as follows: for example, if I perform iCAMP analysis separately for Region A and Region B and find that diffusion limitation accounts for 60% in Region A and 50% in Region B, can I conclude that diffusion limitation plays a greater role in the construction of the community in Region A compared to Region B?

The reason I ask this question is because I currently have microbial data from multiple regions with control and experimental groups. I am considering directly performing iCAMP analysis on all the data, with the first column of the treatment file representing different regions and the second column representing different treatments. However, this calculation may not be appropriate. For example, when we divide the data by region, we cannot consider the impact of different treatments on microbial community construction. Similarly, dividing the data by treatment may overlook the influence of different regions. Of course, I believe I can solve this problem by pairwise comparison, but this may result in a cumbersome presentation of results. Therefore, it would be very convenient to perform iCAMP analysis separately for each group and then compare the results if feasible.

Maybe what I'm most concerned about is whether the results of iCAMP analysis on a microbiome will differ significantly depending on the presence or absence of a treatment file or different comparison objects (for example, comparing with sample A under condition A for the first time and with sample B under condition B for the second time). Because I think the assembly of microbiome in a certain group will not be changed by how they compare with other group. And do you have any better suggestions for this issue?

Another question I have is that after I conducted iCAMP analysis, my treatment file was divided into two groups: RS and RP. In the file named "ProcessImportance_EachGroup," besides the relative proportion of each ecological process for RS and RP, there is also a row named "RS_vs_RP." Could you please explain what this row represents?

And by the way could you give me some general rule for choosing the most appropriate bin.size.limit?

Best regards,

When I set detail.null= TRUE using bNTIn.p. I encountered this error: Error in 1:ncol(bMNTD.randm) : argument of length 0. But if I set detail.null= FALSE (default) using bNTIn.p, It's working smothly. And I can get the beltaNTI value. I am not sure whether I may use this result. Qpen also have same error. Here is the dataset I used.
comm_and_phydist.zip

bNTI_placement.location=bNTIn.p(comm=OTU.location, dis=phydist, nworker = 4, weighted = TRUE, rand = 1000,output.bMNTD=TRUE,detail.null=TRUE)
Now calculating observed betaMNTD. Begin at Thu Mar 25 13:01:14 2021. Please wait...
Now randomizing by parallel computing. Begin at Thu Mar 25 13:01:16 2021. Please wait...
Error in 1:ncol(bMNTD.randm) : argument of length 0
bNTI_placement.location=bNTIn.p(comm=OTU.location, dis=phydist, nworker = 4, weighted = TRUE, rand = 1000,output.bMNTD=TRUE)
Now calculating observed betaMNTD. Begin at Thu Mar 25 13:02:41 2021. Please wait...
Now randomizing by parallel computing. Begin at Thu Mar 25 13:02:43 2021. Please wait...
qpen(comm=OTU.location, pd=phydist,rand.time = 1000,ab.weight = TRUE)
The names are re-ranked.
Now calculating observed betaMNTD. Begin at Thu Mar 25 13:10:56 2021. Please wait...
Now randomizing by parallel computing. Begin at Thu Mar 25 13:10:57 2021. Please wait...
Error in 1:ncol(bMNTD.randm) : argument of length 0

.