lukasvermeer / ab-stats Goto Github PK

Don Green writes:

Slide 11: The third point is really more expansive than "measurement error" -- it includes all manner of excludability violations that would break the symmetry between treatment and control.

Lukas Vermeer responds:

Hm yes you are correct. I was thinking only of ill defined metrics, but randomisation failure would also be a possible explanation.

I need a good, layman's term to describe this category of potential explanations. "Violations of assumptions" seems too vague.

This presentation was forked by Alex Deng. He made some changes to the wording and added a few new topics. I should check if there is anything I want to merge back into this presentation.

Don Green writes:

Slide 4: the third point is sort of orthogonal to the fundamental problem of causal inference. I would drop it and simply work with the usual potential outcomes (fixed values rather than probabilistic draws) because it confuses the issue of causal inference with the issue of how to define a potential outcome.

Lukas responds:

Hm fair point. I do think it is an important point to discuss in the context of online experiments for e-commerce, since almost everybody in this space is running experiments to increase conversion (e.g. rate at which people purchase a product).

Don Green also writes:

Slide 8: it would be better if the example used the schedule of potential outcomes you provide in Slide 3 (e.g., table 2.1 in my book) because the audience is not expecting an example out of the blue. The simulated example is fine but should come afterwards.

Lukas responds:

Yes I've noticed a few times the audience gets confused at this point, so it's good you point it out.

I struggled with this, because eventually I want to use examples that use binary outcomes (like conversion), because that is what most people in the audience will be used to. However, I couldn't use a binary example for the preceding slides, because it leads to pretty nonsensical results with only six units.

I think I should add a new section about defining potential outcomes between slide 7 and 8, and update the preceding slides accordingly.

People are often surprised to see p-values are distributed uniformly under the null. I think that would be nice to show.

Perhaps the umbrella example could be used to intro causal diagrams as well?

The presentation explains what a p-value is, but not how to derive it. I think randomisation inference would be a good way to help people understand what p-values are as well as explain how to compute them