Hi there, I'm Mateusz Buda ๐
๐ผย Machine Learning Research Engineer
๐ย PhD candidate at Warsaw University of Technology
๐คย Narf AI
๐ย Scraping Fish
Statistical functions based on bootstrapping for computing confidence intervals and p-values comparing machine learning models and human readers
Home Page: https://mateuszbuda.github.io/2019/04/30/stat.html
License: MIT License
๐ผย Machine Learning Research Engineer
๐ย PhD candidate at Warsaw University of Technology
๐คย Narf AI
๐ย Scraping Fish
Dear author,
Thank you for sharing your code with us! May I ask is there a pip install method? I haven't used docker before.
Looking forward to your reply. Thx.
Thank you so much for your util.
If I want to analysis precision-recall or f1, how can I use your util?
Thanks alot @mateuszbuda for the tool. I am trying to compare the two models (for example: random forest, xgboost) using p value.
I have two predicted results for both trained models on test set:
y_pred1 = model1.predict(x_test)
y_pred2 = model2.predict(x_test)
y_true = y_test
from sklearn.metrics import r2_score
import stat_util
y_true = y_test
p, z = stat_util.pvalue(y_true, y_pred1, y_pred2, score_fun=r2_score)
output: 0.0
I am always getting p = 0.0. Would you please correct me if I have not done it correctly. Thank you.
Hi Mateusz. Thanks for the super helpful repo! But if I'm not mistaken, I think there's a theoretical error in the way you're calculating your p-value.
Right now, you're computing the p-value for a difference in the performance of two models, measured with AUROC, using the following code:
from sklearn.metrics import roc_auc_score
import matplotlib.pyplot as plt
import stat_util
p, z = stat_util.pvalue(y_true, y_pred1, y_pred2, score_fun=roc_auc_score)
bins = plt.hist(z)
plt.plot([0, 0], [0, np.max(bins[0])], color="black")
print('p =', p)
plt.xlabel('Difference in AUROC')
plt.ylabel('Frequency')
plt.show()
Here, your null hypothesis is that the two models perform equally well, i.e. the difference between their AUROC is equal to 0. And your alternative hypothesis is that model 1 performs better than model 2 (based on the internal code for the function). z
contains a list of differences in AUROC for randomly sampled subsets of predictions from both models. p
is the probability of the difference in AUROC being 0. In other words, your p-value is the probability of the null hypothesis (difference in AUROC = 0) being true. However, theoretically, the p-value is actually the probability of an observed sample statistic, given that the null hypothesis is true. These two statements aren't the same.
Hence, the revision I propose using your same function, is this:
from sklearn.metrics import roc_auc_score
import matplotlib.pyplot as plt
import stat_util
_, z = stat_util.pvalue(y_true, y_pred1, y_pred2, score_fun=roc_auc_score)
# Observed Sample Statistic (Difference in AUROC of Model 1 and 2)
sample_diff = roc_auc_score(y_true,y_pred1) - roc_auc_score(y_true, y_pred2)
# Simulate Distribution of Null Hypothesis using Statistics from Bootstrapping
null_vals = np.random.normal(loc = 0.00, scale = np.std(np.abs(z)), size=2000)
null_dist = plt.hist(null_vals, color='red', alpha=1.0)
# Display Observed Sample Statistic in Same Plot
plt.axvline(sample_diff, color="black")
# Compute p-Value
print('p =',1-percentileofscore(null_vals, sample_diff, kind="weak") / 100.0)
plt.xlabel('Difference in AUROC')
plt.ylabel('Frequency')
plt.show()
Here, the p-value is the probability of observing a difference in AUROC of the two models equal to the observed sample statistic [roc_auc_score(y_true,y_pred1) - roc_auc_score(y_true, y_pred2)] or greater, given that the null hypothesis is true.
If we only compare whether the p-value is below a certain significance level (eg. 0.05) or not, then your method and mine agree for all 10 cases of different model predictions, that I've tried them on. However, the actual p-value itself is always different.
What do you think?
Reference:
[1] https://www.khanacademy.org/math/ap-statistics/tests-significance-ap/idea-significance-tests/v/p-values-and-significance-tests
[2] https://towardsdatascience.com/bootstrapping-for-inferential-statistics-9b613a7653b2
Two tailed p-values from pvalue are often larger than 1. This is easily tested by using the function with y_pred1=y_pred2, or by setting the scale of y_pred1 to a larger value than that of y_pred2 in the example notebook.
The issue arises here:
p = percentileofscore(z, 0.0, kind="weak") / 100.0
if two_tailed:
p *= 2.0
I think the fix is in the replacement:
p = percentileofscore(z, 0.0, kind="weak") / 100.0
if two_tailed:
p = 2*min(p, 1-p)
This will provide also restore that p-values for two-tailed tests are the same regardless of the order of y_pred1 and y_pred2.
However, this will still give undesired p values of 0.0 if y_pred1=y_pred2, because the percentileofscore argument kind = "weak". To remedy this, one could do
p = percentileofscore(z, 0.0, kind="mean") / 100.0
if two_tailed:
p = 2*min(p, 1-p)
Which will result in p values of 1.0 again, but slightly changes the p values compared to the old calculation, even when p<1-p.
Thank you so much for your ml-stat-util.
I want to know the method you compute the p-value. Is it the same as the DeLong test?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.