5. Permutation Tests

Simulating scenarios, simulating datasets, simulating random variables.

Author

Johanna Hardin

Published

September 30, 2024

Artwork by @allison_horst.

Agenda

September 30, 2024

  1. Review: logic of hypothesis testing
  2. Logic of permutation tests
  3. Examples - 2 samples and beyond

September 30, 2024

  1. Conditions, exchangeability, random structure
  2. Different statistics within the permutation test

Readings

Reflection questions

  • What is a test statistic?

  • What is a p-value?

  • Why for a two sample comparison (treatment A vs treatment B) is it okay to use \(\overline{X}_A - \overline{X}_B\) for a test statistic in a permutation test, but for a t-test the test statistic is necessarily \(t^* = \frac{\overline{X}_A - \overline{X}_B}{\sqrt{s^2_A/n_A + s^2_B/n_B}}\) (that is, divided by a measure of variability)?

  • How do you know what to permute in order to create a null sampling distribution?

  • What does “exchangeability” mean (as a technical condition) when discussing permutation tests?

  • What is the difference between a permutation test and a randomization test? Are there times when doing a randomization test is possible?

  • What is power? What are type I and type II errors?

Ethics considerations

  • In a permutation test, sometimes there are many test statistics to choose from (which address the same hypotheses). Why wouldn’t you want to try them all and choose the one that gives you the highest level of significance?

  • When is it acceptable to claim that the resulting “significant” outcome is actually a causal relationship (and not just an association)?

Slides

Additional Resources

:::