Evidence-Based Medicine · Course for Techies

LLM-Assisted EBM: A Prompt Library

Using a language model like a clinician's research assistant — fast at drafting, useless as a source of truth unless you check its work.

Every lesson in this course — from Lesson 2's PICO questions through Lesson 9's full case — involves work an LLM can genuinely speed up: rewriting a vague worry into PICO form, building a search string, skimming an abstract, pulling numbers out of a methods section, doing the ARR/RRR/NNT arithmetic, or role-playing a skeptical reviewer of your own appraisal. None of that work requires the model to be right on the first try. It requires you to be the one who catches it when it's wrong.

This isn't a special LLM failure mode — it's the same discipline Lesson 8 asked of you toward a published paper's own abstract: don't trust the headline number until you've checked it against what's actually reported. An LLM output just needs that same skepticism applied one step earlier, before the paper is even in hand.

Jump to a Stage

1. PICO Formulation 2. Search Strategy 3. Screening Abstracts 4. Extracting Methodology 5. ARR / RRR / NNT 6. Stress-Testing Your Appraisal

1. Formulating a PICO Question

Builds on Lesson 2

Turn a vague clinical worry into a structured question

Use this right after you've written down the messy version of the question — the LLM's job is to propose candidate P/I/C/O splits, not to decide which one is right.

I have a vague clinical question I want to turn into a well-formed PICO question. Here is the situation: [describe the patient/problem in plain language]. Please: 1. Propose 2-3 candidate PICO breakdowns (Population, Intervention, Comparison, Outcome), since my question may be interpretable more than one way. 2. For each, note whether it's a therapy, diagnosis, prognosis, harm, or cost-effectiveness question (per the EBM question-type framework), since that determines what study design I should search for. 3. Flag anything ambiguous in my original description that changes which PICO breakdown is correct, so I can clarify it myself rather than have you guess. Do not recommend a treatment or diagnostic approach at this stage -- I only want help structuring the question.

Verify: confirm the question-type classification yourself against Lesson 2's table before trusting which study design it points you toward.

2. Building a Search Strategy

Builds on Lesson 2

Turn your PICO into a real PubMed search string

Good for the mechanical, SQL-query-like part of search-building: synonyms, MeSH candidates, Boolean structure.

Here is my PICO question: P: [population] I: [intervention] C: [comparison] O: [outcome] Please draft a PubMed search string that: 1. Uses Title/Abstract fields for the key P, I, and O concepts. 2. Suggests likely MeSH terms for each concept (label them clearly as suggestions -- I will verify each one exists in the current MeSH database before using it). 3. Combines concepts with AND, and synonyms within a concept with OR. 4. Suggests relevant publication-type filters (e.g. "Randomized Controlled Trial", "Systematic Review") based on my question type. Also tell me: if this search returns thousands of results, which term should I tighten first? If it returns zero, which term should I loosen first?

Verify: MeSH terms drift and get retired; confirm every suggested MeSH term actually exists in PubMed's current MeSH database before running the search, and run the search yourself rather than trusting a model's claimed result count.

3. Screening Abstracts

Builds on Lessons 2 & 4

Triage a pile of search results against your PICO

Useful when a search returns dozens of abstracts and you need a first pass at which ones are even plausibly relevant — not a substitute for reading the ones it keeps.

I'm screening search results against this PICO question: P: [population] I: [intervention] C: [comparison] O: [outcome] Below is a list of paper titles and abstracts [paste them, or paste one at a time for a large batch]. For each one: 1. Classify as "likely relevant", "likely irrelevant", or "unclear -- needs full-text check", based only on whether the population, intervention, and outcome plausibly match my PICO. 2. Give a one-sentence reason for each classification, quoting no more than a short phrase from the abstract itself. 3. Do not summarize findings or draw conclusions about what the evidence shows overall -- I only want the relevance triage at this stage. Flag any abstract that looks like a systematic review or meta-analysis separately, since I'll want to check those first.

Verify: treat "likely irrelevant" as provisional, not final -- spot-check a sample of the discarded abstracts yourself, since an LLM can misjudge borderline population or outcome matches.

4. Extracting Trial Methodology

Builds on Lesson 3

Pull the Lesson 3 validity checklist out of a methods section

Feed it the paper's methods section (or the full text, if you have access) and let it locate the specific sentences that answer each validity question — then go read those sentences yourself.

Here is the methods section of a trial I'm appraising: [paste text]. For each of the following, tell me what the paper states, and quote the specific sentence or phrase it comes from so I can locate it myself. If the paper doesn't address a point, say "not stated" rather than guessing: 1. How was randomization performed (method of sequence generation)? 2. Was allocation concealed, and how? 3. What level of blinding was used, and of whom? 4. Was there a baseline comparability table, and were groups similar? 5. What was the dropout/loss-to-follow-up rate in each arm, and was it explained? 6. Was the primary analysis intention-to-treat or per-protocol? 7. What was the pre-specified primary outcome, and does it match what's reported as primary in the results? Do not editorialize about whether the trial is "good" or "bad" -- I only want the extracted facts with their locations, so I can judge validity myself using the Lesson 3 framework.

Verify: this is the highest-stakes extraction task in the list -- a wrong answer here corrupts every downstream judgment. Open the paper and confirm every quoted sentence actually says what the model claims it says.

5. Calculating ARR, RRR, and NNT

Builds on Lesson 3

Do the arithmetic, show the work

Arithmetic is exactly the kind of task where an LLM's answer is easy to independently check -- so always ask for the raw numbers and the formula, not just the final figure.

From this trial, here are the raw numbers: - Control group: [n] patients, [x] had the event - Treatment group: [n] patients, [y] had the event Please calculate, showing each step of arithmetic explicitly: 1. Risk in control group and risk in treatment group (as percentages) 2. Absolute Risk Reduction (ARR) 3. Relative Risk Reduction (RRR) 4. Number Needed to Treat (NNT), rounded up to the next whole number 5. If the outcome is a harm rather than a benefit, also calculate the Number Needed to Harm (NNH) using the same method State the formula you're using for each calculation before applying it, so I can check your arithmetic against my own by hand.

Verify: recompute at least the ARR and NNT by hand yourself. This is simple enough arithmetic that there's no excuse not to, and it's the fastest way to catch a misread number from the source paper.

6. Stress-Testing Your Own Appraisal

Builds on Lessons 3, 7 & 8

Have the model argue against your conclusion

The most useful role an LLM can play once you've formed a view: an adversarial reviewer, not a second opinion that just agrees with you.

I've appraised the following trial/review/guideline and reached this conclusion: [state your conclusion and the key evidence you're basing it on]. Please act as a skeptical peer reviewer, not a supportive colleague. Specifically: 1. What is the strongest argument that my conclusion is wrong, or overstated, given the evidence I've described? 2. What source of bias, confounding, or statistical misreading (per the frameworks in Lessons 3, 6, and 8) am I most likely to have missed or underweighted? 3. If this evidence is being weighed against a clinical practice guideline recommendation, is my conclusion more or less confident than the guideline's own stated strength of recommendation -- and if there's a mismatch, what would explain it? 4. What single additional piece of evidence, if I could get it, would most change your confidence in my conclusion? Push back genuinely -- don't soften criticism just because I've already committed to a conclusion.

Verify: a stress-test is a source of questions to go check, not an answer in itself -- if it raises a concern about bias or a missing consideration, resolve it against the primary literature, not by asking the model to reassure you.

General Habits Worth Keeping

Paste the source text in, don't rely on the model's memory. An LLM asked to recall a specific trial's numbers from training data is far more likely to fabricate or misremember than one working from text you've given it directly.
Ask for quotes and locations, not just conclusions. A claim you can trace back to a specific sentence is a claim you can check in seconds; an unsourced summary sentence isn't.
Treat a citation the model produces unprompted as unverified until you've found it yourself. This applies even to citations that look exactly like real journal, year, and DOI formatting — fluent formatting is not evidence of existence.
Use it to explain, not to decide. "Explain what I² means" is a safe use. "Should I recommend anticoagulation for this patient" is not — that's the judgment Lesson 9 walked through as belonging to you and the patient together.