santiviquez / ageml Goto Github PK

For docstrings, we could use the numpy style guide. Or any other in case someone has strong opinions about it.

Motivation: describe the problem to be solved
Right now, in each simulation run in the TemporalDegradation test, the model is fitted using the default parameters. It would be nice if each simulation were fitted on some optimal parameters that are defined automatically inside the flow. In this case, the aging experiment would resemble a real-life scenario where only optimal models are analyzed, and we get to study how this hyperparameter becomes invalid at some point. Hence, we see some performance changes.

Describe the solution you'd like
Use something like optuna to automate the hyperparameter search, possibly optuna.integration.OptunaSearchCV.

The user can provide the search space in the TemporalDegradation.run() method, together with how many tries they want to perform.

example:

experiment = TemporalDegradation(
    timestamp_column_name='inference_time',
    target_column_name='demand',
    n_train_samples=52,
    n_test_samples=12,
    n_prod_samples=24,
    n_simulations=10)

random_forest_params = {
    'n_estimators': optuna.distributions.IntDistribution(100, 400, 1),
    'max_depth': optuna.distributions.IntDistribution(1, 13),
    'min_samples_split': optuna.distributions.IntDistribution(2, 10)}

experiment.run(
    data, 
    model=RandomForestRegressor(),
    hyperparameter_distributions=random_forest_params,
    n_hyperparameter_tunning_trials=10)

Additional context
I tried something like this in an old commit; it might work as inspiration, but the code there is very hard code, so take it with a grain of salt.