Limit max SP for benchmarking to increase speed of training

PyCaret Benchmarks

The purpose of this repository is to assist with the benchmarking of the pycaret module. Currently, the benchmarks are only for time series forecasting and feature extraction. The benchmarking is done using the M3 and M4 competition dataset, but can be extended to other datasets as well.

Result Summary

M3 Benchmark Summary

Monthly
- SMAPE: Bar Chart | Metric vs. Time
- MAPE: Bar Chart | Metric vs. Time
- Primary Model % Coverage: Bar Chart | Metric vs. Time
Quarterly
- SMAPE: Bar Chart | Metric vs. Time
- MAPE: Bar Chart | Metric vs. Time
- Primary Model % Coverage: Bar Chart | Metric vs. Time
Yearly
- SMAPE: Bar Chart | Metric vs. Time
- MAPE: Bar Chart | Metric vs. Time
- Primary Model % Coverage: Bar Chart | Metric vs. Time
Other
- SMAPE: Bar Chart | Metric vs. Time
- MAPE: Bar Chart | Metric vs. Time
- Primary Model % Coverage: Bar Chart | Metric vs. Time

M4 Benchmark Summary

Weekly
- SMAPE: Bar Chart | Metric vs. Time
- MASE: Bar Chart | Metric vs. Time
- OWA: Bar Chart | Metric vs. Time
- MAPE: Bar Chart | Metric vs. Time
- Primary Model % Coverage: Bar Chart | Metric vs. Time
Hourly
- SMAPE: Bar Chart | Metric vs. Time
- MASE: Bar Chart | Metric vs. Time
- OWA: Bar Chart | Metric vs. Time
- MAPE: Bar Chart | Metric vs. Time
- Primary Model % Coverage: Bar Chart | Metric vs. Time

Time Series Benchmarking (Windows)

The benchmarking is done in 4 steps as described below.

Step 1: Create forecasts

You can edit the batch file include all the models and categories you want to benchmark. Then run using the following command. This will execute experiment.py in a loop for all combinations.

scripts\mseries\run_experiments.bat

This will create 2 files per model-category combination - one with the predictions and the other with the run statistics.

Steps 2, 3, & 4

Steps 2, 3, 4 have also been complied into a single script which can be run as follows (you can edit this to select only a subset of datasets or plots).

scripts\mseries\run_postprocessing.bat

Or if you want, you can run them individually as follows.

Step 2: Evaluate results (metrics, time, etc.)

Once you have run Step 1 for all the combinations of interest, you can run the evaluation script to compile the benchmark results (metrics, time, etc.)

python scripts/mseries/evaluation.py --dataset=M3

This will produce a file called data\m3\current_evaluation_full.csv and data\m3\current_evaluation_full.csv with the summary of the benchmark.

Step 3: Update running metrics (i.e. combine previous results with current results)

Next, you can combine these results with the already run benchmarks in the past.

python scripts/mseries/update.py --dataset=M3

Step 4: Plot results

Finally, you can plot the results using plotly.

python scripts/mseries/plot.py --dataset=M3 --group=Monthly

Time Series Feature Extraction (Windows)

This library can also be used to extract features from time series data. The following command will extract features from the M3 dataset and save them to a csv file. This can be useful to evaluate the characteristics of the data before modeling and deciding on the appropriate settings to use for modeling (especially at scale).

scripts\mseries\run_extract_properties.bat

This will save the results into a folder as such data\m4\results\properties
The captured properties include
- Total length of series (train + test)
- Total length of training data
- Total length of test data
- Whether the data is strictly positive or not
- Whether the data is white noise or not
- The recommended value of 'd' and 'D' to use for models like ARIMA
- Whether the data has seasonality or not
- The candidate seasonal periods to be tested
- The significant seasonal periods present
- All the seasonal periods to use for modeling (when models accept multiple seasonal periods)
- The primary seasonal period to use for modeling (when models only accept a single seasonality)
- The significant seasonal periods present when harmonics are removed.
- All the seasonal periods to use for modeling with harmonics removed (when models accept multiple seasonal periods)
- The primary seasonal period to use for modeling when harmonics are removed (when models only accept a single seasonality)

	elif execution_engine == "ray":
	all_results = []
	function_remote = ray.remote(function_single_group)
	for single_group in grouped_data.groups.keys():
	result_single_group = function_remote.remote(
	data=grouped_data.get_group(single_group), **function_kwargs
	)
	all_results.append(result_single_group)
	all_results = ray.get(all_results)
	# Combine all results into 1 dataframe
	all_results = pd.concat(all_results)

	# Uncomment after this is implementedhttps://github.com/pycaret/pycaret/issues/3202
	# We only need the y_test time points. y_test values should be unknown to
	# avoid any chance of leakage
	# test["y"] = np.nan

	test_results.to_csv(f"data/forecasts-{prefix}.csv", index=False)
	time_df = pd.DataFrame(
	{
	"time": [time_taken],
	"dataset": [dataset],
	"group": [ts_category],
	"model": [model],
	"engine": [engine],
	"execution_mode": [execution_mode],
	"run_date": [RUN_DATE],
	"pycaret_version": [PYCARET_VERSION],
	}
	)
	time_df.to_csv(f"data/time-{prefix}.csv", index=False)

pycaret / benchmarks Goto Github PK

benchmarks's Introduction

PyCaret Benchmarks

Result Summary

M3 Benchmark Summary

M4 Benchmark Summary

Time Series Benchmarking (Windows)

Step 1: Create forecasts

Steps 2, 3, & 4

Step 2: Evaluate results (metrics, time, etc.)

Step 3: Update running metrics (i.e. combine previous results with current results)

Step 4: Plot results

Time Series Feature Extraction (Windows)

benchmarks's People

Contributors

Watchers

benchmarks's Issues

Use Cases

Code to update

Move to location

Current

Future

Recommend Projects

Recommend Topics

Recommend Org