Cross validate on multiple splits (kfold/time series split)? about carefree-learn HOT 5 CLOSED

1. For kfold, yes! With the help of carefree-data, carefree-learn is able to perform KRandom and KFold easily. In fact, I've provided some example codes (./tests/benchmark/test_openml.py) to illustrate how to utilize these kinds of splits.

   For instance, you can perform kfold with:

import cflearn
from cfdata.tabular import *

model = "fcnn"
config = {}
x, y = TabularDataset.iris().xy

benchmark = cflearn.Benchmark(
    "iris",
    TaskTypes.CLASSIFICATION,
    models=model,
)
results = benchmark.k_fold(
    10,  # num repeat
    x, y,
    run_tasks=True,
    benchmarks={model: {"custom": config}},
)
results.comparer.log_statistics()

Which will produce:

================================================================================================================================
|        metrics         |                       acc                        |                       auc                        |
--------------------------------------------------------------------------------------------------------------------------------
|                        |      mean      |      std       |     score      |      mean      |      std       |     score      |
--------------------------------------------------------------------------------------------------------------------------------
|    fcnn_iris_custom    |    0.873333    |    0.105198    |    0.768135    |    0.977333    |    0.024074    |    0.953259    |
================================================================================================================================

BTW The above codes will create a folder called __iris__ in your cwd, and you can walk around the files to see what's going on under the hood 😄

2. For time series, currently it is not supported, but will be supported in the near future in carefree-data 😉

   Once carefree-data supports time series, carefree-learn will follow up with some time series models ASAP 😆

from carefree-learn.

Hi @carefree0910 , Thank you for all the work and looking forward to testing the updates. I will be looking at them soon. Thanks again!

from carefree-learn.

That is fantastic news! I am looking forward to this update :)

from carefree-learn.

Hi @jmrichardson ! Since time series datasets have lots of formats, so I'm just wondering whether it is convinient for you to provide me a toy example of your dataset (e.g. 100 samples from the original dataset which are already desensitized), so I can design the APIs better, thanks in advance! 😆

from carefree-learn.

Hi @jmrichardson ! I've updated some codes and now time series datasets are supported! Hope you enjoy these examples, and if they don't meet your demand, feel free to re-open this issue at any time! 😆

from carefree-learn.