A shortlist of the moments across nine lessons where the numbers do something a little counterintuitive — the kind of thing worth remembering even if you forget everything else.
Every lesson in this course has one or two ideas that tend to genuinely surprise people the first time they see them worked through with real numbers — not because the math is hard, but because intuition runs the other way. This page collects them in one place. Each card links back to the lesson where it's explained in full.
Sensitivity and specificity describe the test. They don't change with who you point the test at. But predictive value — the thing a patient actually wants to know, "given I tested positive, do I actually have this?" — depends enormously on how common the disease is in the population being tested.
Same test. 90% sensitivity, 90% specificity, unchanged in both scenarios. The only thing that moved was how common the disease was in the room. This is the single most common source of confusion in diagnostic testing, and it's why validating a test in a specialist clinic and then deploying it as a mass screening tool is a genuinely different act, statistically, than it looks.
→ See the full worked example in Lesson 5Relative risk reduction is the number that ends up in a headline or a drug ad, because it's always the bigger-looking number. Absolute risk reduction and Number Needed to Treat are the numbers that tell you what actually happens to a room full of real patients — including the 24 out of 25 who take on the drug's side effects for no personal benefit at all. Neither number is wrong. Only one of them is honest about scale.
→ See the ARR/RRR/NNT framework in Lesson 3In a forest plot, it's common to see every single trial's confidence interval touch or cross the line of "no effect" — none of them, alone, could claim statistical significance. And yet the pooled diamond at the bottom, combining all of their data, can sit clearly away from that line. This isn't a trick; it's exactly what averaging is supposed to do. Five noisy, individually inconclusive sensor readings pointing the same direction are real signal, even when none of them alone clears the bar.
→ See the annotated forest plot in Lesson 4This is the cleanest illustration of confounding in the whole course precisely because nobody is tempted to believe the causal story. Smoking drives both lighter-carrying and cancer risk, creating a spurious statistical link between two things that have no direct connection at all. The uncomfortable part is that most real confounders in observational medical research are nowhere near this obvious — which is exactly why residual confounding, from a variable nobody thought to measure, can never be fully ruled out in an observational study, no matter how careful the adjustment.
→ See the confounding triangle in Lesson 6GRADE deliberately keeps these separate. "Don't smoke during pregnancy" is a strong recommendation built on evidence that's technically low-certainty — you can't ethically randomize the exposure, so the evidence base is observational. It's strong anyway, because the plausible harm is severe and the cost of following the advice is essentially nothing. Meanwhile, a well-proven drug with high-certainty evidence behind it can still carry only a conditional recommendation, if the benefit is modest enough that reasonable patients would disagree about whether it's worth the trade-off. A guideline that collapses these two dials into one confident-sounding sentence is hiding information you need.
→ See the GRADE framework in Lesson 7Three studies can each report a "non-significant" result and mean three completely different things: one might be genuinely null with a narrow, precise interval sitting right on top of "no effect." Another might be badly underpowered, with a wide interval that happens to straddle the line but is entirely compatible with a large real effect. Only the width of the interval tells you which is which — the p-value alone collapses that distinction into a single pass/fail bit and throws away the part that actually mattered.
→ See the three-study CI comparison in Lesson 8Cholesterol, tumor size on a scan, blood glucose, bone density — these are fast, cheap, convenient things to measure, which is exactly why so many trials measure them instead of the outcome patients actually care about. A drug can reliably improve the surrogate and simply fail to move survival, quality of life, or fracture risk at all — or, in rarer and more troubling cases, improve the surrogate while causing net harm. The convenience of the surrogate is precisely what makes it worth being suspicious of.
→ See the surrogate endpoint table in Lesson 8Test 20 unrelated subgroups — men, women, over-65s, diabetics, and so on — against a treatment with truly zero effect, and the math says you should expect roughly one of them to cross p < 0.05 just from noise. That's not a hypothetical edge case; it's the arithmetic of the threshold itself. It's the exact same failure mode as tuning a model against its own test set until a number looks good — and the fix is the same in both fields: decide the primary comparison before you see the data, and hold it fixed.
Every one of these surprises comes from the same root cause: a single summary number, taken at face value, hides a second variable that changes everything — prevalence, absolute scale, sample size, an unmeasured confounder, evidence certainty, interval width, the outcome being measured, or the number of chances taken to find it. The whole discipline of critical appraisal, from Lesson 1 onward, is really just the habit of asking "what's the second variable here?" before accepting the first number you're handed.