Part 8 — every earlier lesson leaned on p-values and confidence intervals. Time to open that black box properly.
You've been using p < 0.05 and confidence intervals since Lesson 3 as a kind of pass/fail signal. That's enough to get through most appraisals, but it also invites a specific kind of overconfidence — treating "statistically significant" as a stamp of truth rather than what it actually is: a statement about how surprising the data would be under a specific, narrow assumption. This lesson takes the black box apart.
Here is the precise definition, worth sitting with because almost every common misreading comes from skipping a clause in it:
Notice what it does not say. It is not the probability that the null hypothesis is true. It is not the probability that your finding is a fluke. It says nothing at all about the size or clinical importance of an effect — a trivial, meaningless difference can produce a tiny p-value if the sample is large enough, and a large, important-looking difference can produce a big p-value if the sample is small.
| Common misreading | What a p-value of 0.03 actually means |
|---|---|
| "There's a 97% chance the treatment works" | If the treatment had truly no effect, you'd see a result this extreme (or more extreme) 3% of the time by chance alone |
| "There's a 3% chance this is a false positive" | Says nothing about that probability directly — that depends on the prior probability the effect is real, which the p-value doesn't know about (this is exactly the PPV-vs-prevalence trap from Lesson 5, transposed onto statistics) |
| "The effect is real and important" | Says nothing about effect size — only about how compatible the data are with "no effect" |
That middle row deserves a callback: it's structurally the same mistake as confusing sensitivity with PPV in Lesson 5. A p-value is like sensitivity — a property of the test procedure under an assumption — while "is this effect actually real" is like PPV — it depends on how plausible the effect was to begin with, information the p-value alone can't supply.
A confidence interval is built from the same statistical logic as a p-value, but it reports something more clinically useful: a range of effect sizes compatible with the data, not just a single accept/reject verdict.
Study 1's interval is narrow and sits entirely away from "no effect" — a precise, statistically solid finding. Study 2's interval is wide and straddles "no effect" — this is very likely underpowered, and "not significant" here means "we couldn't tell," not "there's no effect." Study 3's interval is narrow and centered right on "no effect" — this is a study with enough precision to say, with some confidence, that any true effect is probably small. Width matters as much as position: "not significant" and "no effect" are not the same claim, and only a narrow interval straddling the line lets you say the second one with any confidence.
This distinction has come up implicitly since Lesson 3's NNT discussion, and it's worth making explicit: a result can be statistically significant and clinically trivial, or clinically important and statistically non-significant, and these are genuinely different failure modes.
A blood-pressure drug trial with 50,000 participants finds a mean systolic pressure reduction of 1.2 mmHg, p < 0.001. The huge sample makes even a clinically negligible difference statistically detectable. Compare that to a trial of 40 participants finding a 15 mmHg reduction with a p-value of 0.08 — a much larger, potentially meaningful effect that the small sample simply wasn't powered to confirm with confidence.
Why this matters: "statistically significant" answers "is this probably not zero?" It says nothing about "is this big enough to matter to a patient?" That second question needs the effect size itself — ARR, NNT, mean difference — interpreted against what actually changes a patient's health or experience, exactly the kind of number Lesson 3 taught you to demand.
The threshold of p < 0.05 traces back to a somewhat arbitrary suggestion by the statistician Ronald Fisher in the 1920s, and it has since calcified into something treated as a bright line between "real" and "not real." Two consequences follow directly from taking that convention too literally:
That second consequence is exploitable, deliberately or not, and it has a name: p-hacking — analyzing data in enough different ways, or testing enough different subgroups and outcomes, until something clears the significance threshold, then reporting only that result as if it were the pre-specified question.
Common forms worth watching for when appraising a paper:
The single best defense as a reader is checking whether the trial was pre-registered (on a registry like ClinicalTrials.gov) with a stated primary outcome and analysis plan, and then comparing that registration against what actually got published. A mismatch between the two is one of the clearest tells in the whole appraisal toolkit.
Separate from p-value games, there's a design-level issue worth flagging on its own: many trials measure a surrogate endpoint — a stand-in outcome that's faster or cheaper to measure than the outcome patients actually care about — instead of a patient-important outcome.
| Surrogate endpoint | Patient-important outcome it's standing in for |
|---|---|
| LDL cholesterol level | Heart attack, stroke, death |
| Tumor shrinkage on imaging | Survival, quality of life |
| Blood glucose (HbA1c) | Diabetic complications, death |
| Bone mineral density | Fracture |
Surrogates are attractive because trials using them can be smaller and shorter — you don't have to wait years to count strokes if you can measure cholesterol in weeks. The trap is that a drug can reliably move the surrogate without moving the outcome that matters, or can even move the surrogate in the "right" direction while causing harm overall. A surrogate is only trustworthy as a stand-in when it's been well validated as closely tracking the real outcome across many prior trials — and even then, treat a trial that reports only a surrogate as an incomplete answer to a therapy question, not a substitute for one.
Most papers include a limitations paragraph, and most readers skim past it. It's often the most honest part of the paper — look for these specifically, and be equally suspicious of a limitations section that's suspiciously thin: