I installed the calibration framework from scratch as described and was using an conda environment, but when I try to run the classification examples I got an error. It looks like that it is related to the ENIR.
Get path of all Near Isotonic Regression models with mPAVA ...
Traceback (most recent call last):
File "/home/labor/calibration-framework/examples/classification/CIFAR.py", line 169, in
cross_validation(model, use_cuda=use_cuda, domain=domain)
File "/home/labor/calibration-framework/examples/classification/CIFAR.py", line 135, in cross_validation
success = cross_validation_5_2(models=models, datafile=datafile, bins=bins, save_models=save_models, domain=domain)
File "/home/labor/miniconda3/envs/cal/lib/python3.10/site-packages/netcal/Decorator.py", line 90, in new_f
return f(*args, **kwds)
File "/home/labor/miniconda3/envs/cal/lib/python3.10/site-packages/examples/classification/utils.py", line 236, in cross_validation_5_2
instance.fit(build_set_sm, build_set_gt)
File "/home/labor/miniconda3/envs/cal/lib/python3.10/site-packages/netcal/Decorator.py", line 62, in new_f
return f(*args, **kwds)
File "/home/labor/miniconda3/envs/cal/lib/python3.10/site-packages/netcal/binning/ENIR.py", line 239, in fit
self._multiclass_instances = self._create_one_vs_all_models(X, y, ENIR, self.score_function,
File "/home/labor/miniconda3/envs/cal/lib/python3.10/site-packages/netcal/AbstractCalibration.py", line 568, in _create_one_vs_all_models
model.fit(onevsall_confidence, onevsall_ground_truth)
File "/home/labor/miniconda3/envs/cal/lib/python3.10/site-packages/netcal/Decorator.py", line 62, in new_f
return f(*args, **kwds)
File "/home/labor/miniconda3/envs/cal/lib/python3.10/site-packages/netcal/binning/ENIR.py", line 270, in fit
self._model_scores, self._binning_models = self._elbow(X, y, model_list, self.score_function, alpha=0.001)
File "/home/labor/miniconda3/envs/cal/lib/python3.10/site-packages/netcal/Decorator.py", line 35, in new_f
return f(*args, **kwds)
File "/home/labor/miniconda3/envs/cal/lib/python3.10/site-packages/netcal/AbstractCalibration.py", line 498, in _elbow
model_scores = self._calc_model_scores(confidences, ground_truth, model_list, score_function)
File "/home/labor/miniconda3/envs/cal/lib/python3.10/site-packages/netcal/Decorator.py", line 35, in new_f
return f(*args, **kwds)
File "/home/labor/miniconda3/envs/cal/lib/python3.10/site-packages/netcal/AbstractCalibration.py", line 468, in _calc_model_scores
model_scores = np.exp((np.min(score) - score) / 2.)
File "<array_function internals>", line 200, in amin
File "/home/labor/miniconda3/envs/cal/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 2946, in amin
return _wrapreduction(a, np.minimum, 'min', axis, None, out,
File "/home/labor/miniconda3/envs/cal/lib/python3.10/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation minimum which has no identity
Used setup:
-
conda environment
conda create --name cal python=3.10
conda activate cal
-
Repo clone and install
git clone https://github.com/EFS-OpenSource/calibration-framework
cd calibration-framework/
python3 -m pip install .
-
Execute examples
cd examples/classification/
python CIFAR.py