Fact Graphs, Pathways & Appraised Evidence

0. Recap: Two Cognitive Modes, One Shared Risk

The base framework runs two modes over case narratives — Promption, which scaffolds messy narrative data into structured illness scripts, and Provocation, which stress-tests the resulting hypothesis for anchoring and premature closure. Both modes are fluent by design. Neither, on its own, is anchored to anything outside the model's own internal coherence.

That shared blind spot is the subject of this piece: two structured ground-truth layers — a knowledge graph and an institutional pathway — examined against real published examples, plus a third layer, appraised evidence, that answers a question neither of the first two can.

1. Layer One — The Knowledge Graph

Worked example: Chronic Kidney Disease

Applied to CKD, the graph does something a free-text answer can't: it turns a hedge into a checkable criterion. Instead of the model asserting "this looks like chronic kidney disease" as an internally coherent but unverified judgment, the graph holds the actual diagnostic threshold — eGFR below 60 mL/min/1.73m² persisting for three months or more — as a structured, queryable entry, alongside its linked findings, complications, and differential set.

What the graph fixes — and what it doesn't

• Converts tacit illness-scripts into explicit, checkable structure, so modules can validate against verified criteria rather than internal coherence alone.
• Enables a real calibration ledger: confidence tracked against documented criteria across many cases, not just one.
• Turns the adversarial shadow module into a systematic differential engine, drawing on the graph's own differential-diagnosis field rather than free-associating alternatives.
• Enables rule-based, auditable red-flag and escalation logic instead of prose-shaped urgency.

2. Layer Two — The Institutional Pathway

Worked example: Acute Pancreatitis

Where the CKD graph gives modules a fact to check, the pancreatitis pathway gives them something the fact graph structurally cannot: a sequence. Diagnosis requires 2 of 3 criteria; fluid resuscitation is titrated hour-by-hour against urine output; the well-known 48-hour organ-failure threshold defines severe disease under the Revised Atlanta Classification and is exactly the kind of concrete pause point the framework's Diagnostic Time-Out module needs, instead of a generic "pause and reconsider" prompt.

A new risk: borrowed authority

The net effect: a mature system needs to separately audit diagnostic correctness, sequencing correctness, and contextual/population correctness (special populations, contraindication branching) — rather than collapsing all three into a single aggregate "reasoning quality" score.

3. Layer Three — Appraised Evidence

Facts answer what. Pathways answer when. Appraised evidence — systematic reviews, meta-analyses, GRADE-rated guidelines — answers a third question neither layer touches: how strong is the justification, and under what conditions does it hold?

• Un-flattens uniform declarative confidence into GRADE-style certainty tiers, instead of one flat register of assertion.
• Surfaces genuine guideline disagreement — ACG vs. IAP/APA vs. NICE on pancreatitis management — as a teaching moment rather than silently picking one synthesis to present as consensus.
• Converts evidence-based-medicine modules from generic EBM-sounding prose into citation-anchored critique of a specific study's actual limitations.
• Restores legitimate use of precise, sourced numbers — effect sizes, confidence intervals — in place of unfalsifiable hedge words. The earlier problem was ungrounded precision, not precision itself.

4. The Complexity/Bias Tradeoff

Stacking all three layers isn't free. For common, densely-documented diseases, the base model is often already close to ground truth — adding a fact graph, a pathway, and an evidence layer adds seams where independently-sourced components can quietly contradict each other, for a comparatively small accuracy gain.

For rare or atypical presentations, the picture is sharper: unstructured reasoning over-fits the case to the nearest common script, while a structure-first approach can force a sparse or absent entry into the nearest available match — both produce confident wrongness, just via different mechanisms, and the structured failure is harder to detect because it wears the appearance of verified rigor.

A well-designed system can fail loudly — an explicit "no strong match" signal — rather than silently inheriting the base model's overconfidence. Disagreement across layers (fact graph vs. pathway vs. evidence) is itself a usable audit signal, provided the seams are surfaced rather than papered over. That argues for differential deployment: heavy structure is low-value where the base model is already reliable, and highest-value exactly where unstructured reasoning fails worst — conditional on the structure honestly reporting its own coverage density.

5. Proposed Next Step

An empirical study: run matched common and rare cases both with and without the ground-truth layers (fact graph, pathway, evidence), and measure three outcomes across both conditions —