Hypothesis testing interpretation? about dabest-python HOT 6 CLOSED

PS: this question and proposition fits into my understanding of the philosophy behind this project as trying to draw the data and test distribution for what they are so that the reader can draw his/her own interpretation instead of the author enforcing an interpretation onto the reader.

from dabest-python.

Strictly speaking, the "Control 1 null distribution" is a auto-comparison. So the bootstrap distribution of Control 1 - Control 1. It's not actually the null distribution for any other comparison. In fact, each comparison has its own null distribution, strictly speaking.

You could show P(data|h0) for every comparison here: resample with replacement from Control 1 and Test XX at the same time, compute the mean difference, and highlight the portion of the distribution that exceeds the mean of Test XX. So in fact, you would get a third axes, all of which show the null distributions centered at 0, but with different shapes and different endzones highlighted.

I hope that was clear....?

from dabest-python.

Dear @josesho , thank you very much for your detailed reply.

About the bootstrap distribution of Control 1 - Control 1, yes that's what I meant (I've read since then to better understand the stats going on, sorry for the confusion :-) ). My idea would be to allow for a comparison of intervals, with not only the point estimate (bootstrap mean) of Control 1, but a distribution estimate of Control 1, with Control 1 - Control 1 serving as a "normalized" distribution/interval for other comparisons, as to allow interval inference as described in:

Dienes, Z. (2014). Using Bayes to get the most out of non-significant results. Frontiers in psychology, 5, 781.

Notably this figure is quite what I had in mind.

Do you think such comparisons of the confidence intervals between eg, Test 1 - Control 1 vs Control 1 - Control 1 would be sensible and an adequate confidence interval comparison as summarized in this paper? If not, why?

About your second paragraph, I thought that p(data|h0) was what we computed here, such as Test 1 - Control 1, by computing a bootstrap mean and standard deviation of both and then do a difference of Test 1 - Control 1 means + scaling of stdev? Are you suggesting that doing this calculation inside each bootstraps (instead of with the final bootstrap estimate from all bootstraps) would provide a more accurate p(data|h0)?

from dabest-python.

Ah I think I understand what you mean in the second paragraph, but did you intend to write p(h0|data) perhaps? In which case, I see that we could compute the probability of h0 given data by using the data as a reference and plotting a third axe (or simply using the data as the control group?), but then it would be quite meaningless as we would amalgamate our hypothesis h1 with the data, assuming it's the population instead of a sample, and thus having no meaningful use for inference.

So I think it's safe to drop that idea :-) However, I'm still interested in plotting a distribution for the control condition bootstrap if that makes sense, as to allow intervals comparison and inference, and making it easier to either be more conservative in inference (by requiring that Test 1 - Control 1 confidence interval is outside of Control 1 - Control 1 CI for example, instead of just the point estimate at 0) or to make it possible to accept h0 (when Test 1 - Control 1 CI is contained inside Control 1 - Control 1 CI). Does that make sense? If so, I can work on it :-)

from dabest-python.

Please forget what I wrote above about scaling with standard deviation, this is disadvised by several sources, so actually the simple difference is better in the general case :-)

• U is for Unease: Reasons for Mistrusting Overlap Measures for Reporting Clinical Trials, 2010
• https://garstats.wordpress.com/2016/05/02/robust-effect-sizes-for-2-independent-groups/

from dabest-python.

Closing this, will reopen more specific PRs

from dabest-python.