pairing mean_diff with Mann-Whitney test about dabest-python HOT 5 CLOSED

We are now using permutation tests as the default statistical test in place of Mann-Whitney. See PR #96; feel free to upgrade to v0.3.0

from dabest-python.

Actually now that I think about it, even median_diff doesn't necessarily match the Mann-Whitney conclusion. Median of differences would match MW (as does Cliff's delta), but median_diff is calculating difference of medians (not sure what corresponding test could be used there).
I guess this sort of decision is always complex and should be left to the user, but the tutorial example should definitely be adjusted at least, in my opinion!

from dabest-python.

Hi @tmchartrand ,

You are correct in pointing out the incongruency. The overarching intention of estimation plots is to de-emphasise the dichotomous all-or-nothing nature of hypothesis tests, which current usage of P values exacerbates.

In the webapp at estimationstats.com, we do state below the results that

the P value(s) reported are the likelihood(s) of observing the effect size(s), if the null hypothesis of zero difference is true; they are included here to satisfy a common requirement of scientific journals.

which hopefully serves to inform the reader on what a P value really is...

We will update the tutorial to bring the intent and thinking in line with an estimation framework that de-emphasises P values; thanks for pointing this out!

from dabest-python.

Thanks for the reply!
I'm not sure if my meaning got across fully though. It's not so much the tension between p-values and estimation plots in general that I was trying to bring up. As I see it, they can provide complementary views of the same question, provided the effect size measures the same properties of the data as the test statistic does - this is the case when a mean difference effect size plot is paired with a t-test, but not when it is paired with a MW test.

from dabest-python.

from dabest-python.