Error in serialize(data, node$con, xdr = FALSE) : ignoring SIGPIPE signal about biocparallel HOT 5 CLOSED

Can you provide a more completely reproducible example?

from biocparallel.

I'm working on finding an example with data I can share. Sorry for the vagueness.

from biocparallel.

Sorry for the delay, this is a more reproducible set of instructions. On a linux server

in the conda environment generated by the following

mamba create -n outrider -c bioconda bioconductor-outrider

in R 4.2.2

library(OUTRIDER)
ods <- readRDS("test_ods.RDS")
register(MulticoreParam(20))
OUTRIDER(ods)

gives the following output

> OUTRIDER(ods)
Fri Dec 16 19:41:10 2022: SizeFactor estimation ...
Fri Dec 16 19:41:11 2022: Controlling for confounders ...
Using provided q with: 44
Fri Dec 16 19:41:11 2022: Using the autoencoder implementation for controlling.
[1] "Fri Dec 16 19:41:17 2022: Initial PCA loss: 6.7195743956208"

	Error in serialize(data, node$con, xdr = FALSE) : ignoring SIGPIPE signal
Error in serialize(data, node$con, xdr = FALSE) : 
  error writing to connection

bpparam() gives the following

class: MulticoreParam
bpisup: FALSE; bpnworkers: 20; bptasks: 0; bpjobname: BPJOB
bplog: FALSE; bpthreshold: INFO; bpstopOnError: TRUE
bpRNGseed: ; bptimeout: NA; bpprogressbar: FALSE
bpexportglobals: TRUE; bpexportvariables: FALSE; bpforceGC: TRUE; bpfallback: TRUE
bplogdir: NA
bpresultdir: NA
cluster type: FORK

It fails if I register anything more than a single worker. However if I downgrade the biocparallel version it does work with a MulticoreParam(20)

Attached is a tar file containing the RDS object (containing publically available data) and the yaml description
github_upload.tar.gz

from biocparallel.

Sorry for the delay. I think the basic problem is that the workers drop their connection if they've been idle for more than options("timeout") seconds, which is by default 60 seconds. This can be emulated with

library(BiocParallel)
options(timeout = 10)
register(MulticoreParam(2))
bpstart()
Sys.sleep(11)  # oh oh, workers have closed their connections...
res <- bplapply(1:10, identity)

resulting in

Error in serialize(data, node$con, xdr = FALSE) : ignoring SIGPIPE signal
Error in serialize(data, node$con, xdr = FALSE) : 
  error writing to connection

A workaround is to set options(timeout = 60 * 60) for instance. I thought that MulticoreParam(timeout = 60*60) would help, but that sets the maximum computation time. I believe the Bioconductor build systems process the OUTRIDER vignette (which has a realistic dataset) because it sets the environment variable R_DEFAULT_INTERNET_TIMEOUT=600

We'll try to arrive at a more permanent solution or at least a more informative error message @Jiefei-Wang .

from biocparallel.

This should be fixed in BiocParallel 1.33.8 (devel) or 1.32.5 (release), which should propagate in the next 48 hours.

from biocparallel.