Explain SMBC — the wiki for Saturday Morning Breakfast Cereal

The Joke

A scientist explains p-values to a non-scientist. The explanation starts simply but quickly reveals that p-values are widely misunderstood even by scientists, are not the same as the probability that the hypothesis is true, and are the basis for an enormous amount of published research that may be wrong.

The non-scientist asks what they should believe instead, and the scientist admits they're not sure.

The Humor

The comic exposes one of the genuine crises in modern science: p-value misinterpretation. A p-value of 0.05 means "if the null hypothesis is true, there's a 5% chance of seeing results this extreme." It does NOT mean "there's a 95% chance the hypothesis is correct." Yet this misinterpretation is rampant, even among published researchers.

The humor is in the dawning horror — the more accurately the scientist explains the situation, the worse it sounds. The foundation of empirical science turns out to be shakier than anyone would like to admit.

Technical Context

The "replication crisis" — the finding that many published scientific results fail to replicate — is partly driven by misuse of p-values, p-hacking (running analyses until you get p < 0.05), and publication bias (journals only publishing positive results). The American Statistical Association issued an unprecedented statement in 2016 clarifying what p-values do and don't mean.