A Deceptively Simple-Sounding Claim
A news headline or marketing claim that "a new study shows treatment X works" sounds straightforward, but the underlying statistical finding it is usually based on is considerably more nuanced — and frequently misunderstood, even by well-meaning clinicians.
What "Statistically Significant" Actually Means
Most clinical trials report whether a result is "statistically significant," which technically means the observed difference between groups was unlikely to have occurred purely by chance, given the study's sample size and variability — not that the difference is necessarily large, clinically meaningful, or relevant to any individual patient (Sullivan & Feinn, 2012).
Why Effect Size Is the More Important Number
A very large study can find a statistically significant result for an effect so small it has no real-world relevance to a patient's life, while a smaller study with a genuinely large, meaningful effect might fail to reach statistical significance at all. The effect size — how large and clinically meaningful the actual difference is — is generally the more important number for a clinician to look at, and it is frequently glossed over entirely in simplified summaries of research findings.
References & Further Reading
- Sullivan GM, Feinn R. Using Effect Size — or Why the P Value Is Not Enough. J Grad Med Educ. 2012;4(3):279-282.