Error:
Performing 2-fold outer CV, using 1 core
Error in if (any(penalty.factor == Inf)) { :
missing value where TRUE/FALSE needed

I identified the bug and resolved it by replacing the else statement in nest_filter_balance.R, line 30 with:

else {
    args <- list(y = ytrain, x = xtrain)
    args <- append(args, filter_options)
    fset <- do.call(filterFUN, args)
    filt_xtrain <- xtrain[, fset, drop = FALSE]
    filt_xtest <- xtest[, fset, drop = FALSE]
    
    # Match indices of filtered set with original features
    fset_indices <- match(fset, colnames(x))
    filt_pen.factor <- penalty.factor[fset_indices]

Pretty sure the issue was that you were filtering on the gene names outputted by the custom filter, rather than their indexes.

Cheers,

Parallelisation of the outer loops in nestcv.train() by setting cv.cores = 2 or more fails with method = "xgbTree" or method = "xgLinear" on linux/windows. This appears to be due to automatic multithreading invoked by xgboost using OpenMP, which is not available on Mac OS. Calling nestcv.train with cv.cores = 1 works fine, but hangs indefinitely if cv.cores = 2 or more.

Work around is to disable OpenMP multithreading using the following command before calling nestcv.train.

RhpcBLASctl::omp_set_num_threads(1L)

Aim is to incorporate this into nestcv.train in a future release.

Related to this issue:
topepo/caret#1106

Fitting a caret model with nestcv.train() and model = "xgbTree" causes a crash in caret::train() when fitting the model if parallelisation with mclapply() is used and tuneLength is 5 or more. Unknown cause. Can be fixed by switching to parLapply() by setting multicore_fork = FALSE.

Error message:

Error in `[.data.frame`(data, , "pred") : undefined columns selected

Reprex below:

## xgbTree tuneLength=5 mclapply bug reprex

library(nestedcv)
library(caret)
library(parallel)
library(mlbench)  # Boston housing dataset

data(BostonHousing2)
dat <- BostonHousing2
y <- dat$cmedv
x <- subset(dat, select = -c(cmedv, medv, town, chas))

# no error with tuneLength = 3 or 4
fit <- nestcv.train(y, x, method = "xgbTree",
                    tuneLength = 3, verbose = TRUE,
                    cv.cores = 8)

# produces an error in caret::train()
fit <- nestcv.train(y, x, method = "xgbTree",
                    tuneLength = 5, verbose = TRUE,
                    cv.cores = 8)

# runs fine
fit <- nestcv.train(y, x, method = "xgbTree",
                    tuneLength = 5, verbose = TRUE,
                    multicore_fork = FALSE,
                    cv.cores = 8)

The problem occurs in caret::train(..., method = "xgbTree") when wrapped in mclapply()

# single caret fit - ok
fit1 <- caret::train(x, y, method = "xgbTree", tuneLength = 5)

# source of problem is caret::train(method="xgbTree") when wrapped in mclapply()
# no error if tuneLength = 3
out <- mclapply(1:2, function(i) {
  fit1 <- caret::train(x, y, method = "xgbTree", tuneLength = 3)
  fit1
}, mc.cores = 2)

# multicore - causes an error if tuneLength = 5
out <- mclapply(1:2, function(i) {
  fit1 <- caret::train(x, y, method = "xgbTree", tuneLength = 5)
  fit1
}, mc.cores = 2)

I think this might relate to this issue in caret and multithreading:
topepo/caret#1106

For the moment the simplest workaround is to disable multicore forking by setting the multicore_fork argument to FALSE when using nestcv.train(). This switches to using parLapply() which works fine.

Hi @myles-lewis,

I am preparing a new release of fastshap for CRAN, and it seems like it will break some functionality in your package. You can see a list of changes here. I suspect the biggest changes affecting your package iare that 1) fastshap >= 0.1.0 will no longer return a tibble, but rather a matrix, and 2) the autoplot() function have been removed in favor of using the much better shapviz package.

Just wanted to give you a heads up before I plan to submit in two weeks.

nestedcv

Newly broken

• checking examples ... ERROR

  Running examples in ‘nestedcv-Ex.R’ failed
  The error most likely occurred in:
  
  > ### Name: pred_nestcv_glmnet
  > ### Title: Prediction wrappers to use fastshap with nestedcv
  > ### Aliases: pred_nestcv_glmnet pred_nestcv_glmnet_class1
  > ###   pred_nestcv_glmnet_class2 pred_nestcv_glmnet_class3 pred_train
  > ###   pred_train_class1 pred_train_class2 pred_train_class3
  > 
  > ### ** Examples
  ...
  Error in `autoplot()`:
  ! Objects of class <explain> are not supported by autoplot.
  ℹ have you loaded the required package?
  Backtrace:
      ▆
   1. ├─ggplot2::autoplot(sh)
   2. └─ggplot2:::autoplot.default(sh)
   3.   └─cli::cli_abort(...)
   4.     └─rlang::abort(...)
  Execution halted
  

In nest_filt_bal, some subsetting occurs, but the subsetting operation for xtrain on line 15 appears to be missing a drop = FALSE as seen 2 lines later for xtest.