Sometimes we only know the search_space (param_set of the OptimInstance) but not the domain (search space after trafo) of the objective. We should not always be obliged to define a domain because it is not used anyways.

Therefore it should be optional to define the domain.

At some point we have to carefully review the complete documentation.

Archive:
x_rep_1 x_rep_2 V1
1: 0.00145 0.004792 0.00002507
2: -0.90146 -0.875556 1.57923700

Hi,

I stumbled on this by chance, not sure if it can be classified as bug but I thought you should know about it.

When combined with a learner that fails occasionally stagnation terminator returns the error:

Error in if (self$terminator$is_terminated(self)) { : 
  missing value where TRUE/FALSE needed

Example:

lrn_rpart <- lrn("classif.rpart")
ig <- po("filter", flt("information_gain"))

ps <- ParamSet$new(list(
  ParamDbl$new("classif.rpart.cp", lower = 0, upper = 0.05),
  ParamInt$new("information_gain.filter.nfeat", lower = 20L, upper = 60L),
  ParamFct$new("information_gain.type", levels = c("infogain",
                                                   "gainratio")) # I know gainratio does not work well with Sonar
))

glrn <- ig %>>%
  lrn_rpart

glrn <- GraphLearner$new(glrn) 

glrn$encapsulate <-  c(train = "evaluate", predict = "evaluate")

cv5 <- rsmp("cv", folds = 5)

tsk <- mlr_tasks$get("sonar")

instance <- TuningInstance$new(
  task = tsk,
  learner = glrn,
  resampling = cv5,
  measures = msr("classif.ce"),
  param_set = ps,
  terminator =  term("stagnation", iters = 5, threshold = 0)
)

tuner <- TunerRandomSearch$new()
set.seed(123)
tuner$tune(instance)

After 6 configurations evaluated the error occurs - I trust it is due to the NaN in the performance measure

instance$archive()
   nr batch_nr  resample_result task_id                     learner_id resampling_id iters params tune_x warnings errors classif.ce
1:  1        1 <ResampleResult>   sonar information_gain.classif.rpart            cv     5 <list> <list>        0      0  0.2648084
2:  2        2 <ResampleResult>   sonar information_gain.classif.rpart            cv     5 <list> <list>        0      0  0.2596980
3:  3        3 <ResampleResult>   sonar information_gain.classif.rpart            cv     5 <list> <list>        0      5        NaN
4:  4        4 <ResampleResult>   sonar information_gain.classif.rpart            cv     5 <list> <list>        0      0  0.2454123
5:  5        5 <ResampleResult>   sonar information_gain.classif.rpart            cv     5 <list> <list>        0      0  0.2501742
6:  6        6 <ResampleResult>   sonar information_gain.classif.rpart            cv     5 <list> <list>        0      0  0.2737515

All the best,

Milan

Probably as a Method of OptimInstanceMulticrit or Archive.

related: mlr-org/mlr3hyperband#45

If we call OptimInstance$eval_batch(xdt) from inside an optimizer we might have more information than just the x values (e.g. AcqFunction value that lead to this x value in MBO).

Either we allow more information in xdt or we allow adding more info afterwards (which would be a bit cumbersome),

probably we simply pass a 3rd paramset in addition to domain and codomain that lists all of them?

in evaluator? also make sure they dont happen twice if we connect this to mlr3tuning!

mlr-org/mlr3misc#42

For example TerminatorStagnation would probably not be directly applicable for multi-objective?

It would make sense to have this as a reference optimizer implementation in bbotk

There should be a function that can be passed into an arbitrary tuner e.g. optimize(obj$tuner_objective).

xss_trafoed should be added as a list of lists to the Archive in add_evals even when just one parameter is optimized.

In the description under technical details

-inlcude basic example
-inlcude MOO
-inlcude Parallelization
-include how to handle y + "extra" returns from the objective

I implemented two Optimizers for threshold tuning: OptimizerNloptr and OptimizerGenSA.
You can find them here.

In general, I would need to factor out the threshold tuning logic and the optimization logic, but this should be trivial.

It would be cool if they could be added to bbotk or mlr3tuning.

See also: mlr-org/mlr3tuning#231

• param_set is used in many objects to control how this object behaves
• here the param_set is actually the search space
• Therefore: Name it search_space ?

other suggestions welcome

Should be defined properly. Also the signature of assign_result:

Suggestion:

list(
  xdt, # data.table with one or multiple rows, 
       # subset of the archive, meaning xdt is subset of search_space
  y, # numerical vector for single-crit, data.table for multi-crit
  x_opt # list (of lists) with one or mulitple elements. 
        # transformed x values, subset of domain of objective
)

Alternative: the result is just a data.table in the exact same way as in archive$data. Basically a subset of arhcive$data and in cases the optimizer returns a result that was not previously evaluated such a data.table has to be constructed.

Actually I would prefer the alternative suggestion because it feels simpler and more coherent.

in the archive? the objective? do they simply rerurn it?

Currently removed because it lived in the encapsulation.

nds_selection.R returns the best subset of points by non-dominated sorting with hypervolume contribution for tie-breaking.

Also needed for multiobjective black-box optimization. Should go into the Archive class here?

I opened an issue in mlr3tuning as well; be aware of duplication.

That would be the right place where you can put a ParamSet specific sampler that could be used for Hyperband or RandomSearch etc.

Alternatively we can have a Subclass but I don't see a big need for that here.

Prolem: The user does not directly see if the sampler actually works. i.e. some tuners like the GridSearch would just use the ParamSet directly and ignore the Sampler.

thats a datatable of multiple points. isnt this useful in quite a few places?
also has an assert?

It would be nice to have a quick plot function that shows the tuning curve and for 1 and 2d problems some response surface with a simple interpolation.

Would that be part of mlr3viz?

For sequential feature selection a terminator that terminates after the performance does not improve more than threshold over the last batch would be useful.

how do we define the interface for this? tags?

implemented in main dir under random_search.R
feels ok

FIXME: we could add some basic, simple optimizers from R here. connecting them here would enable them for many tasks in optimization, not only mlr3tuning. think then how mlr3mbo extends this system then / regiusters itself

I think this goes beyont bbotk?

i now wrote this due to the dt interface....

fn = function(dt) {
  y = map_dbl(seq_row(dt), function(i) {
    x = dt[i,]
    sum(x^2)
  })
  data.table(y = y)
}

There are a few interesting approaches of optimizers that change their behavior depending on the progress of the optimization. We could easily support those by having all terminators returning values between 0 and 1, whereas values >= 1 mean that the optimization should terminate.

not sure how to handle this:

the package should allow for the following:
if multiple points are evaluated, this should be parallelized (by future) and encapsulated (by callr).
now it seems reasonable to copy over / do something similar as in mlr3.
NB: I have no problem with copy-pasting that code, thats not the issue here!

If I do that, and bbotk is used in mlr3tuning, we now have these features twice. That seems confusing to the user?
Example: I could now switch on the parallel-option for bbotk, but I could also switch it on for mlr3.
The same for encapsulation.

What would be the best way out here? @mllg

this looks a bit weird now, but maybe ok. the reason was i guess because mlr3 put them in a dict
i have opted not to change this for now

At the moment we just actively support extras going into the instance and coming out of the objective to be stored in the archive if they are always the same (same names). What happens if some extras are just added for some evaluations?