Process in "D" state on NFS filesystem in a HPC environment about icamp1 HOT 2 CLOSED

iCAMP uses some functions (e.g., big.matrix) from the R package bigmemory and parallel computing function (e.g., parLapply) from package parallel. I do not know which will cause file locking. Two possible reasons: (1) Many functions in iCAMP can set parallel computing thread number by the parameter 'nworker', e.g., in icamp.big. If the user sets nworker as a very large number (over CPU cores), it will occupy too many threads and make the computer not responsive at all. (2) The big.matrix function used in some iCAMP function (e.g., pdist.big) saves a relatively big file with suffix '.bin' somewhere in the harddisk, and the physical address will be saved in a '.desc' file; then, the data in '.bin' file will be used as if it is saved in memory, thus the calculation can go beyond the limit of memory. The '.bin' file occupies a certain place which will not work if physically changed (copy/paste) to another place. But this is unlikely to make computer not responsive.

I used iCAMP a lot on our Linux and Windows servers without problems as long as I limit the nworker (thread number) lower than the CPU core number. But I have not deployed it in HPC yet.

from icamp1.

I hope the problem has been solved. If no more questions, I will close this issue.

from icamp1.