henrikbostrom / crepes Goto Github PK

I have some relatively beginer questions after trying the intro code:

• The wrapper do not modify the predict_proba, is that right ?
• When compared with calibrating a rf on the complete training dataset X_train (same seed), the conformal approach doesn't always improve the metric (70% of the time it improve the Brier score), is there any reason that it doesn't work all the time ?
• Small comment but we would need:

from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split

for the code to work out of the box.

Hi,

Would be very usefull to be able to predict prediction interval for the predict_proba scores returned by a binary classifier. Is it on the road map?

Thanks!

Hi,

In order to ensure repdoducability of experiments, it would be useful to be able to set a random seed for the computation of smoothed p-values. In particular when working with small datasets. One solution might be to pass the seed as an optional argument to WrapClassifyer/WrapRegressor.

A few minimal examples should be provided in the README-file.

Hello, I am trying to use the crepes library at scale in a production environment (appx 100 million rows and 75 columns) but I am struggling with the amount of time required to fit the DifficultyEstimator and calibrating the CP wrapper. See below for the python code.

`
from crepes import WrapRegressor
from crepes.extras import DifficultyEstimator, binning
import pandas as pd

def calibrate_cp(
rf: WrapRegressor, x_tr: pd.DataFrame, y_tr: pd.DataFrame, x_ca: pd.DataFrame, y_ca: pd.DataFrame
) -> Union[WrapRegressor, DifficultyEstimator, list]:
"""Calibrates the conformal predictive system wrapper.

Args:
    rf (WrapRegressor): A wrapped regression model
    x_tr_n (pd.DataFrame): X train
    y_tr_n (pd.DataFrame): Y train
    x_ca_n (pd.DataFrame): X calibration
    y_ca_n (pd.DataFrame): Y calibration

Returns:
    Union[WrapRegressor, DifficultyEstimator, list]: Returns the calibrated wrapper, DifficultyEstimator (NearestNeighbors), and thersholds of the bins.
"""
de_kkn = DifficultyEstimator()
logging.info("Fitting the DifficultyEstimator")
de_kkn.fit(x_tr.to_numpy(), y=y_tr.to_numpy(), scaler=True)
logging.info("Calculate sigmas")
sigmas_cal = de_kkn.apply(x_ca.to_numpy())
logging.info("Calculate bins")
bins_cal, bin_thresholds = binning(rf.predict(x_ca), bins=5)
logging.info("Calibrate CP wrapper")
rf.calibrate(x_ca, y_ca.to_numpy().reshape((len(y_ca.to_numpy()),)), sigmas=sigmas_cal, bins=bins_cal, cps=True)
return rf, de_kkn, bin_thresholds

`

I therefore want to improve the scalability of crepes by adding GPU support (Any other suggestions for performance enhancements are also welcome).
This could for example be achieved by allowing for alternative implementations of kkn such as faiss:
https://towardsdatascience.com/make-knn-300-times-faster-than-scikit-learns-in-20-lines-5e29d74e76bb
Where the user would then set an optional parameter when initializing the class or a global config variable deciding which backend to use.

I thought I'd see if there is any interest in having this in the repository. If not I'll just make it in my own repo.

Hinge loss does not seem to work unless y_cal is re-indexed from zero.

alphas_cal = hinge(model.predict_proba(X_cal), model.classes_, y_cal)

KeyError: 0

The above exception was the direct cause of the following exception:

KeyError Traceback (most recent call last)
in <cell line: 1>()
----> 1 alphas_cal = hinge(model.predict_proba(X_cal), model.classes_, y_cal)

/usr/local/lib/python3.10/dist-packages/crepes/extras.py in hinge(X_prob, classes, y)
69 if y is not None:
70 class_indexes = np.array(
---> 71 [np.argwhere(classes == y[i])[0][0] for i in range(len(y))])
72 result = 1-X_prob[np.arange(len(y)),class_indexes]
73 else:

/usr/local/lib/python3.10/dist-packages/crepes/extras.py in (.0)
69 if y is not None:
70 class_indexes = np.array(
---> 71 [np.argwhere(classes == y[i])[0][0] for i in range(len(y))])
72 result = 1-X_prob[np.arange(len(y)),class_indexes]
73 else:

/usr/local/lib/python3.10/dist-packages/pandas/core/series.py in getitem(self, key)
1005
1006 elif key_is_scalar:
-> 1007 return self._get_value(key)
1008
1009 if is_hashable(key):

/usr/local/lib/python3.10/dist-packages/pandas/core/series.py in _get_value(self, label, takeable)
1114
1115 # Similar to Index.get_value, but we do not fall back to positional
-> 1116 loc = self.index.get_loc(label)
1117
1118 if is_integer(loc):

/usr/local/lib/python3.10/dist-packages/pandas/core/indexes/base.py in get_loc(self, key)
3653 return self._engine.get_loc(casted_key)
3654 except KeyError as err:
-> 3655 raise KeyError(key) from err
3656 except TypeError:
3657 # If we have a listlike key, _check_indexing_error will raise

KeyError: 0

Will this paper be implemented in this package?

https://link.springer.com/content/pdf/10.1007/s10472-023-09847-0.pdf?pdf=button

Hello, has anyone run into an issue with using the predict method with CPS? I am just using this on a simple linear regression toy model that has about 480,000 observations, since this will need to be scaled to a way larger dataset.

I fit the normalized Conformal Predictive System following the same steps as the notebook, but when I go to run the predict method I keep getting this error:

MemoryError: Unable to allocate 939. KiB for an array with shape (120249,) and data type float64

The error was traced back to this portion of the base.py code:
--> 344 cpds = np.array([y_hat[i]+sigmas[i]*self.alphas 345 for i in range(len(y_hat))])

These were fit with the residuals for the calibration set so I am wondering if this is the issue? I have plenty of memory available to run something like this locally... like gigs worth, and the inability to allocate such an insignificant amount of memory is strange to me.

Hi,

in a Kaggle competition I try to epxlore the Crepes library. In the ReadMe a DifficultyEstimator is used, which calls KNN under the hood.

As I used LGBM as the main regressor, I did not scale the data. However I wonder if this is required once I make use of the DifficultyEstimator. Or is this done by Crepes already?

The question came up, because this notebook performs much worse than another one where I only used the WrapRegressor. So I assume I have some issue in the code, may it be some missing preprocessing or something else.

For reference the notebooks:

• WrapRegressor only
• Mondrian with DifficultyEstimator.

When using python 3.11, I got an error on row 1023 (and row 1049) in base.py when making the cps.predict call below:

calY_pred = model.predict(calX)
testY_pred = model.predict(testX)

cps = ConformalPredictiveSystem()
cps.fit(residuals=calY-calY_pred)

interval = cps.predict(y_hat=testY_pred, lower_percentiles=[5,50], higher_percentiles=[95,50])

I do not get the error when calling any of the calls below:

interval = cps.predict(y_hat=testY_pred, lower_percentiles=[5,50])
interval = cps.predict(y_hat=testY_pred, higher_percentiles=[95,50])

The error can be solved by changing the if-clause(s) to if len(percentile_indexes) == 0.

Functions outside classes should also be documented.

A memory issue (#9) was fixed in version 0.5.0; instead of always generating the cpds array within the predict method of ConformalPredictiveSystem, this is now done only when requested (by return_cpds=True) or when including "CRPS" among the metrics to evaluate (which currently is the default). The latter should preferably be calculated without generating the full cpds array (which may consume a lot of memory for very large calibration and test sets).

If I submit values for higher_percentile or lower_percentile from a numpy array, these values are not seen as float (as they are indeed numpy.float64) -- hence they are never transformed into the 1-valued list (see line 361 in base.py).

A working solution could be to replace each line of the kind

if type(lower_percentiles) in [int, float]:

with:

if isinstance(lower_percentiles, (int, float)):

which would return True for both float and numpy.float64 types.

Running the following:
de = DifficultyEstimator()
display(de)
results in an error due to that the fittedattribute is missing.
--> 144 if self.fitted and self.estimator_type == "knn":
145 return (f"DifficultyEstimator(fitted={self.fitted}, "
146 f"type={self.estimator_type}, "

AttributeError: 'DifficultyEstimator' object has no attribute 'fitted'.

The fitted attribute (or an is_fitted function) would be nice to have even for not yet fitted estimators.

Currently calling the evaluate() function of the ConformalPredictiveSystem class with argument metrics=["CRPS"] fails with the following error:

UnboundLocalError: local variable 'lower_percentile' referenced before assignment

Example call:

 score = model.evaluate(
                y_hat=t[predict_col].values,
                y=t[target_col].values,
                bins=bins_t,
                sigmas=sigma_t,
                y_min=0,
                metrics={"CRPS"})

Investigating the code shows there might be an issue in the implementation of evaluate() function.

tic = time.time()
if metrics is None:
    metrics = ["error","eff_mean","eff_med","CRPS","time_fit",
                "time_evaluate"]
    lower_percentile = (1-confidence)/2*100
    higher_percentile = (confidence+(1-confidence)/2)*100
    test_results = {}
if "CRPS" in metrics:
    results, cpds = self.predict(y_hat, sigmas=sigmas, bins=bins, y=y,
                                  lower_percentiles=lower_percentile,
                                  higher_percentiles=higher_percentile,
                                  y_min=y_min, y_max=y_max,
                                  return_cpds=True, cpds_by_bins=True)

In this case when a call is made with metrics={"CRPS"}, lower_percentile is never initiated so it fails.

Concerning the wrappedclassifier with mondrian conformal prediction would it be safe to use the p-value of the predicted class as a calibrated probability or a good estimate of uncertainty. If not, any advice on could work as an estimate of uncertainty seeing that the predict_proba outputs uncalibrated probabilities based on the wrapped classifier. Note, I am working on a multiclass problem so Venn-ABERS does not seem to be applicable. I have already considered set size per instance but I was hoping for something more informative. Thanks in advance for any response.

ConformalPredictiveSystem.evaluate may return the result of a test of uniformity of p-values

I have been running into an error when using the conformal predictive system. When I try to obtain the prediction intervals using an higher percentile >90, I get an index error (refer to traceback below). Since the intervals compute with a maximum value of 90, could the error be an issue caused by a small dataset size of 144 train, 48 cal and 49 test samples (I have tried this with a larger dataset and the intervals compute with an higher_percentile>90 as expected)? Assuming dataset size is the issue, is there a way to make this work for small datasets?

File C:\WINDOWS\System32\erty\lib\site-packages\crepes\base.py:457, in ConformalPredictiveSystem.predict(self, y_hat, sigmas, bins, y, lower_percentiles, higher_percentiles, y_min, y_max, return_cpds, cpds_by_bins)
450 for b in range(len(bin_values)):
451 higher_indexes = [
452 int(np.ceil(higher_percentile/100
453 * (len(bin_alphas[b])+1)))-1
454 for higher_percentile in higher_percentiles]
455 result[bin_indexes[b],
456 no_prec_result_cols:no_prec_result_cols
--> 457 + len(higher_indexes)] = cpds[b]
458 [:,higher_indexes]
459 if y_max < np.inf:
460 result[:,no_prec_result_cols:no_prec_result_cols
461 + len(higher_percentiles)]
462 [result[:,no_prec_result_cols:no_prec_result_cols
463 + len(higher_percentiles)]>y_max] = y_max

IndexError: index 10 is out of bounds for axis 1 with size 10

The format of the optional output cpds of the method ConformalPredictiveSystem.predict need to be described.

The methods and functions do not currently provide informative error messages when given incorrect input.

Hello, thank you very much for your great package. I have a quick question, where does the equation for calculate_p come from? It seems to be roughly just comparing the input predicted probabilities (i.e., softmax values) to the rank-ordering of the calibration non-conformity scores. This kind of makes sense, but does this really correspond to a p-value necessarily? Is there a reference for this?