Lesson 4: Systematic Reviews & Meta-Analyses

In Lesson 3 you appraised a single RCT. But no single trial is ever the final word — it has its own sample, its own setting, its own quirks. A systematic review tries to find every relevant trial on a question, appraise them with the same rigor you learned in Lesson 3, and combine them into one answer. When that combination is done statistically, it's called a meta-analysis.

Systematic Review vs. Ordinary Literature Review

Not everything called a "review" is a systematic review. The difference is method, not just topic:

A systematic review doesn't have to include a meta-analysis — sometimes the included studies are too different to combine numerically, and the review just summarizes them narratively (this is sometimes done, and is still valuable, but weaker than a pooled estimate).

Reading a Forest Plot

A forest plot is the standard way to visualize a meta-analysis: each row is one trial, and the diamond at the bottom is the pooled result across all of them.

How to read it:

• The vertical center line represents "no difference" (relative risk = 1.0). If a trial's confidence interval crosses this line, that trial alone didn't reach statistical significance.
• Each dot is a trial's point estimate; the line through it is its 95% confidence interval. A wider line means a smaller, less precise trial.
• Box/dot size roughly reflects weight — bigger trials with more events usually get more influence over the pooled result.
• The diamond at the bottom is the pooled estimate; its width is the pooled confidence interval. Because it combines all the data, it's almost always narrower (more precise) than any single trial.

Heterogeneity: When Is It Fair to Pool?

Averaging only makes sense if the trials are answering roughly the same question. Heterogeneity is the term for how much trials disagree beyond what you'd expect from chance alone.

• Clinical heterogeneity — different populations, doses, follow-up durations, or outcome definitions across trials.
• Statistical heterogeneity — measured by the I² statistic, which estimates the percentage of variation across trials due to real differences rather than random chance.

High heterogeneity doesn't automatically mean "don't pool" — but it does mean you should ask why trials disagree (different doses? different patient ages?) before trusting the single pooled number, and a random-effects model (which assumes true effects vary across settings) is usually more honest than a fixed-effect model (which assumes one true effect everywhere) when I² is high.

Funnel Plots and Publication Bias

Studies with exciting, positive results are more likely to get published than "boring" null results — which can bias a meta-analysis toward overstating an effect. A funnel plot checks for this: plot each trial's effect size against its precision (or sample size). With no publication bias, trials should scatter symmetrically around the pooled estimate, forming a rough funnel/triangle shape — small trials scattered widely at the bottom, large trials tightly clustered near the top.

If small studies are missing from one side of the funnel — typically the side showing a null or negative result — that gap is a red flag for publication bias: small "boring" trials may have been left unpublished, skewing the pooled result.

Appraising a Systematic Review Itself

The three-question framework from Lesson 3 still applies — just aimed at the review, not a single trial:

1. Are the results valid? Was the search comprehensive (multiple databases, no language restriction)? Was study selection and quality appraisal done by at least two independent reviewers? Were the included studies actually similar enough to combine?
2. What are the results? What's the pooled effect size and confidence interval? How much heterogeneity was there, and was it explored (subgroup or sensitivity analysis)?
3. Will this help my patient? Do the included trials' populations resemble your patient? Were all patient-important outcomes covered, not just the ones easiest to pool?

AMSTAR-2 and PRISMA are the two most widely used formal checklists here: AMSTAR-2 rates the methodological quality of a review, while PRISMA is a reporting checklist authors are expected to follow when writing one up. Reviewers use AMSTAR-2; authors (and readers checking completeness) use PRISMA.