p-values

Summary

The p-value is the probability, computed under the assumption that the null hypothesis is true, of obtaining a test statistic at least as extreme as the one observed. It measures the compatibility between the data and the null hypothesis, not the probability that the null is true.^[1]

Prerequisites

• Hypothesis Testing
• Point Estimation
• Normal Distribution

Definition / Notation

For an observed test statistic $t_{\text{obs}}$ :

for an upper-tailed test, with the inequality reversed for a lower-tailed test and a two-tailed version for $H_a:\theta \ne {\theta }_0$ .

Parameters / Assumptions

• The null hypothesis is fully specified, or nuisance parameters are handled.
• The sampling distribution of the test statistic under $H_0$ is known.
• The test and the form of the alternative are selected before seeing the data.

Essential Result

Decision rule:

• If p-value $<\alpha$ , reject $H_0$ at level $\alpha$ .
• If p-value $\ge \alpha$ , do not reject $H_0$ at level $\alpha$ .

A small p-value indicates that the observed result is unusual under $H_0$ ; it does not measure the size of an effect or the posterior probability of $H_0$ .^[2]

Worked Example

Continuing the normal example with $z=2$ for a two-tailed test:

Since $0.0456<0.05$ , we would reject $H_0$ at the 5% significance level.

Common Mistakes

• Saying " $H_0$ is true with probability p".
• Treating p-value as an effect-size measure.
• Comparing p-values across different studies or sample sizes directly.
• Treating 0.05 as an absolute boundary between truth and falsehood.

Connections

• Hypothesis Testing
• Type I and Type II Errors
• Confidence Intervals

References