StudyDiff

Understand why two scientific studies reach different conclusions —
with every claim verified against the source.

Live demo · Quick start · How it works · Examples · MCP server · Roadmap · Contributing

Two well-run papers often reach opposite conclusions. Usually the reason isn't that one is wrong — it's a methodological difference (a cell type, a dose, a follow-up window, an analysis choice) that a reader has to dig out of the methods sections by hand. StudyDiff does that digging. Give it two studies and it extracts each one's design, surfaces the differences that could explain the disagreement, and — critically — grounds every statement in the source text, so it never invents a finding.

It is built for a bench scientist deciding which of two conflicting papers to trust before planning an experiment.

Why it's different

Most "AI literature" tools generate a fluent answer and ask you to trust it. StudyDiff inverts that:

• It shows the evidence, then gets out of the way. The design differences up top; every value with the verbatim sentence that supports it underneath.
• It refuses to guess. Any field the source doesn't state is shown as not reported, never inferred.
• It verifies itself. A deterministic grounding check (no second LLM acting as judge) confirms every extracted value and every explanation is backed by a verbatim quote and traceable numbers. Anything that fails is downgraded before it can be used as a reason.
• It knows what it can't do, because it was measured — see below.

Does it work? A measured answer

StudyDiff used to rank the divergent design dimensions and present the top one as the likely driver of a disagreement. We built a benchmark to test that, and it doesn't work.

15 documented contradictions where the literature has since established why the papers disagreed — each label carrying its own citation, the set built blind before any accuracy number existed. Scored against StudyDiff's top-ranked driver:

Top-1 accuracy (strict)      13.3%  (95% CI 3.7-37.9%)   [2/15]
Baseline "always say assay"  13.3%  (95% CI 3.7-37.9%)   [2/15]
                             → discordant on 0 of 15 cases
Oracle ceiling (reachable)   66.7%  (95% CI 41.7-84.8%)  [10/15]
Non-assay-labelled cases      0.0%  (95% CI 0.0-22.8%)   [0/13]

The ranking was a fixed prior (DRIVER_RANK in src/compare.mjs) in which assay outranks everything. Two papers almost always use somewhat different methods, so assay almost always diverges, so it was picked 13 times out of 15 — and the two hits are exactly the two assay-labelled cases. It is not merely as good as guessing a constant; it is behaviourally identical to it on every case in the set.

Fixing grounding first (Phase 2) removed that excuse. Recovering 20 of 26 false-positive rejections doubled the oracle ceiling from 33% to 67% — the established cause is now an available candidate in 10 of 15 cases instead of 5 — and top-1 accuracy did not move at all. The ranker was handed the right answer five more times and took none of them.

What changed as a result: the app no longer nominates a primary driver. It presents the divergent dimensions as an unranked list, because that list is informative (it contains the established cause 10 times in 15) while the ordering is not. Choosing among them needs domain knowledge the tool doesn't have.

What still holds: which dimensions differ, which are identical (ruled out), and the verbatim sentence behind every value. None of that depends on the ranking.

Confirmed blind, on a second set

Those 15 cases had by then been read across two phases — failures analysed, spans re-audited — so every post-fix figure from them is development-set accuracy, not a blind measurement. So we built a second set and measured it once.

eval/cases-heldout.json is 15 further documented contradictions, curated to a protocol written and committed before any case was selected (eval/HELDOUT-PROTOCOL.md), by a curator kept blind to the dev set's per-case failures, in deliberately different fields: microbiome, marine ecology, toxicology, psychology, critical care, oncology, infectious disease. No paper and no contradiction is shared with the dev set — selftest enforces that mechanically.

Top-1 accuracy (strict)      13.3%  (95% CI 3.7-37.9%)   [2/15]
Baseline "always say assay"  20.0%  (95% CI 7.0-45.2%)   [3/15]
                             → discordant on 1 of 15 cases
Oracle ceiling (reachable)   73.3%  (95% CI 48.0-89.1%)  [11/15]
Non-assay-labelled cases      0.0%  (95% CI 0.0-24.3%)   [0/12]

On unseen data the prior scores below the constant guess — by exactly one case. The intervals overlap almost entirely and the two strategies disagree on 1 of 15, so the honest statement is that it remains indistinguishable from guessing assay every time, not that it is worse.

The line that does not move is the last one. Across both sets, 25 cases where the established cause was something other than assay, the prior identified none of them. And the ceiling here is higher than on the dev set — 73.3% against 66.7% — so extraction put the right answer in front of the ranker more often, and it was taken no more often. That is the Phase 1–2 conclusion reproduced on data the development loop never saw, which is the only way it could have been strengthened.

The number is reported as-is, and is never pooled with the dev-set number: summing them into an "n=30" figure would relaunder read data as blind data. Nothing in src/ was changed on the basis of it. The set was fetched once on a single pre-registered arm, but the scoring was not one clean pass — the first run reported n=14 after one paper failed to fetch, and the full set was scored after retrieving it. Both figures, the sixteen defects an adversarial verification pass found and corrected before any scoring, and the reasoning behind each label are recorded in the file's own provenance block rather than summarised away.

npm run eval:heldout            # the blind number, offline and free
npm run eval:selftest:heldout   # set integrity, incl. zero overlap with the dev set

Full method, the pre-registered decisions, and every prediction that turned out wrong: eval/README.md and eval/PHASE2.md. The benchmark sets are eval/cases.json (development) and eval/cases-heldout.json (held-out).

npm run eval            # offline, free, no API key — regenerates the numbers above
npm run eval:selftest   # validates the harness maths and set integrity

eval/cache/ and eval/cache-heldout/ are committed on purpose. They aren't build output, they're evidence: the published numbers are reproducible from artefacts in the repo rather than taken on faith.

Quick start

Under 60 seconds, no API key, no network:

git clone https://github.com/nickjlamb/studydiff && cd studydiff
npm install
npm run demo                        # explains a real, famous contradiction
npm run demo -- resveratrol-sirt1   # a second worked example
npm run demo -- treg-stability      # a third: Treg lineage stability

Run the web app:

cp .env.example .env    # add ANTHROPIC_API_KEY for live comparisons
npm run serve           # http://localhost:4173

The built-in examples run on cached data and need no key. To compare live, add your key and use the PMID / DOI, Upload PDF, or Paste inputs.

How it works

flowchart LR
  IN["Two papers<br/>PMID · DOI · PDF · text"] --> R["Retrieve<br/>PubMed / PMC / PDF"]
  R --> E["Extract<br/>Claude → structured study cards"]
  E --> V["Verify<br/>deterministic grounding"]
  V -- "ungrounded → not reported" --> E
  V --> C["Compare<br/>divergent vs. shared design"]
  C --> X["Present<br/>divergent dimensions + ruled out"]