Studydiff README
The full upstream README, mirrored here for reference. Install config, tool schemas, adoption signals, and an original overview live on the Studydiff listing page.
StudyDiff
Understand why two scientific studies reach different conclusions —
with every claim verified against the source.
Live demo · Quick start · How it works · Examples · MCP server · Roadmap · Contributing
Two well-run papers often reach opposite conclusions. Usually the reason isn't that one is wrong — it's a methodological difference (a cell type, a dose, a follow-up window, an analysis choice) that a reader has to dig out of the methods sections by hand. StudyDiff does that digging. Give it two studies and it extracts each one's design, surfaces the differences that could explain the disagreement, and — critically — grounds every statement in the source text, so it never invents a finding.
It is built for a bench scientist deciding which of two conflicting papers to trust before planning an experiment.
Why it's different
Most "AI literature" tools generate a fluent answer and ask you to trust it. StudyDiff inverts that:
- It shows the evidence, then gets out of the way. The design differences up top; every value with the verbatim sentence that supports it underneath.
- It refuses to guess. Any field the source doesn't state is shown as not reported, never inferred.
- It verifies itself. A deterministic grounding check (no second LLM acting as judge) confirms every extracted value and every explanation is backed by a verbatim quote and traceable numbers. Anything that fails is downgraded before it can be used as a reason.
- It knows what it can't do, because it was measured — see below.
Does it work? A measured answer
StudyDiff used to rank the divergent design dimensions and present the top one as the likely driver of a disagreement. We built a benchmark to test that, and it doesn't work.
15 documented contradictions where the literature has since established why the papers disagreed — each label carrying its own citation, the set built blind before any accuracy number existed. Scored against StudyDiff's top-ranked driver:
The ranking was a fixed prior (DRIVER_RANK in src/compare.mjs) in which assay
outranks everything. Two papers almost always use somewhat different methods, so assay
almost always diverges, so it was picked 13 times out of 15 — and the two hits are exactly
the two assay-labelled cases. It is not merely as good as guessing a constant; it is
behaviourally identical to it on every case in the set.
Fixing grounding first (Phase 2) removed that excuse. Recovering 20 of 26 false-positive rejections doubled the oracle ceiling from 33% to 67% — the established cause is now an available candidate in 10 of 15 cases instead of 5 — and top-1 accuracy did not move at all. The ranker was handed the right answer five more times and took none of them.
What changed as a result: the app no longer nominates a primary driver. It presents the divergent dimensions as an unranked list, because that list is informative (it contains the established cause 10 times in 15) while the ordering is not. Choosing among them needs domain knowledge the tool doesn't have.
What still holds: which dimensions differ, which are identical (ruled out), and the verbatim sentence behind every value. None of that depends on the ranking.
Confirmed blind, on a second set
Those 15 cases had by then been read across two phases — failures analysed, spans re-audited — so every post-fix figure from them is development-set accuracy, not a blind measurement. So we built a second set and measured it once.
eval/cases-heldout.json is 15 further documented contradictions, curated to a protocol written
and committed before any case was selected (eval/HELDOUT-PROTOCOL.md),
by a curator kept blind to the dev set's per-case failures, in deliberately different fields:
microbiome, marine ecology, toxicology, psychology, critical care, oncology, infectious disease.
No paper and no contradiction is shared with the dev set — selftest enforces that mechanically.
On unseen data the prior scores below the constant guess — by exactly one case. The intervals
overlap almost entirely and the two strategies disagree on 1 of 15, so the honest statement is
that it remains indistinguishable from guessing assay every time, not that it is worse.
The line that does not move is the last one. Across both sets, 25 cases where the established
cause was something other than assay, the prior identified none of them. And the ceiling here
is higher than on the dev set — 73.3% against 66.7% — so extraction put the right answer in
front of the ranker more often, and it was taken no more often. That is the Phase 1–2 conclusion
reproduced on data the development loop never saw, which is the only way it could have been
strengthened.
The number is reported as-is, and is never pooled with the dev-set number: summing them into an
"n=30" figure would relaunder read data as blind data. Nothing in src/ was changed on the basis
of it. The set was fetched once on a single pre-registered arm, but the scoring was not one clean
pass — the first run reported n=14 after one paper failed to fetch, and the full set was scored
after retrieving it. Both figures, the sixteen defects an adversarial verification pass found and
corrected before any scoring, and the reasoning behind each label are recorded in the file's own
provenance block rather than summarised away.
Full method, the pre-registered decisions, and every prediction that turned out wrong:
eval/README.md and eval/PHASE2.md. The benchmark
sets are eval/cases.json (development) and
eval/cases-heldout.json (held-out).
eval/cache/ and eval/cache-heldout/ are committed on purpose. They aren't build output,
they're evidence: the published numbers are reproducible from artefacts in the repo rather than
taken on faith.
Quick start
Under 60 seconds, no API key, no network:
Run the web app:
The built-in examples run on cached data and need no key. To compare live, add your key and use the PMID / DOI, Upload PDF, or Paste inputs.
How it works
- Retrieve — a PubMed/PMC client with a full-text-to-abstract fallback that tags how deep it read (
fulltext/abstract/pasted); uploaded PDFs are text-extracted server-side. - Extract — Claude turns each paper into a fixed study card (species, model, assay, dose, timing, endpoint, sample size, statistic, finding, limitations). Every field carries a verbatim supporting quote; absent fields default to not reported.
- Verify — grounding runs first: any value whose quote isn't in the source, or whose numbers don't trace, is downgraded to not reported. StudyDiff can't cite a fact it hasn't verified.
- Compare — deterministic: which design dimensions are reported by both papers, which of those diverge, and which are identical. Divergence is a token-overlap test on design dimensions and strict inequality on the conclusion itself, because a conclusion's polarity matters where a method list's content does.
- Present — the divergent dimensions unranked, each with both papers' values and the verbatim sentence behind them, and the identical ones explicitly ruled out. StudyDiff does not nominate one as the cause: it was measured and it doesn't work.
Two things hold the guarantee up: the API key never leaves the server, and verification is deterministic — grounding and comparison involve no model judgment at all, so the same verified evidence always yields the same divergent dimensions.
Why Claude?
StudyDiff's reliability comes from how it uses Claude, not just that it does:
- Structured extraction via tool-use. Each paper becomes a study card through a forced Claude (Sonnet) tool schema, so every field returns validated and carries a verbatim supporting quote — no free-text parsing, no "mostly-JSON" failures.
- Reasoning that maps claims to evidence. Claude reads the methods/abstract prose and identifies both the design value and the exact sentence that supports it — the hard part of turning unstructured papers into comparable, auditable cards.
- Claude proposes, grounding disposes. StudyDiff never uses an LLM as the judge. A deterministic check verifies Claude's output against the source and downgrades anything unsupported before it's shown. Pairing Claude's tool-use extraction with non-LLM verification is what lets the tool trust its own output and sidesteps the usual LLM-as-evaluator pitfalls.
- Built with Claude Code. The whole app — pipeline, UI, hardening, deploy — was built iteratively with Claude Code in a single agentic loop.
Examples
| Question | Papers | What StudyDiff finds |
|---|---|---|
| Do mouse models mimic human inflammation? | Seok 2013 vs Takao & Miyakawa 2015 | Same datasets, opposite conclusions — driven by the gene-selection strategy. |
| Does resveratrol activate SIRT1? | Howitz 2003 vs Beher 2009 | An assay artifact — the Fluor de Lys peptide substrate vs. native substrates. |
| Is the Treg lineage stable in vivo? | Zhou 2009 vs Rubtsov 2010 | A landmark T-cell controversy — stable vs. unstable, driven by the fate-mapping method. |
All three ship as offline demos (npm run demo / npm run demo -- resveratrol-sirt1 / npm run demo -- treg-stability).
Using it
- Web app (
npm run serve) — examples, PMID/DOI lookup, PDF upload, or paste; streams each pipeline step live, and exports a reproducible Markdown report with every value's verbatim supporting sentence. - MCP server (
npm run mcp) — lets Claude, or any agent, call the contradiction engine directly. See MCP server. - CLI —
node src/cli.mjs --q "Does resveratrol activate SIRT1?" 12939617 19843076 - Deploy — see DEPLOY.md (Railway + custom domain).
MCP server
StudyDiff's engine is also a Model Context Protocol server, so an agent can ask why two papers disagree as a tool call — the same grounded pipeline, no browser.
Published to the official MCP Registry as io.github.nickjlamb/studydiff, and to npm as studydiff-mcp.
| Tool | What it does |
|---|---|
compare_studies(paperA, paperB, question?) | The full pipeline on two papers. Each paper is {id} (PMID or DOI) or {citation, text}. Needs ANTHROPIC_API_KEY. |
compare_example(example) | Runs a cached worked example — no API key, no network. The quickest way to see the grounded output. |
list_examples() | Lists the built-in worked examples. |
It returns the same auditable report the web app exports: the verdict, the divergent design dimensions (unranked) and the identical ones (ruled out), every value with the verbatim sentence that supports it, the verification counts, and what evidence would resolve the disagreement. Ungrounded fields come back as not reported — never guessed — and the tool never picks a winner, never nominates a primary driver, and never reports a confidence it didn't compute.
Add it to Claude Code:
…or to Claude Desktop (claude_desktop_config.json):
ANTHROPIC_API_KEY is only needed for compare_studies. Leave it out and list_examples / compare_example still work — the cached examples run with no key and no network. (To run from a local clone instead, swap the command for node /absolute/path/to/studydiff/src/mcp.mjs.)
Then just ask:
Use studydiff to compare PMID 19633673 and 20929851 — why do they disagree?
No key handy? list_examples, then compare_example("treg-stability") runs entirely offline.
Project layout
Roadmap
Keyword search with a results picker, a source viewer that highlights each grounded quote in the original text, batch comparison, and an exportable report. Full list in ROADMAP.md.
Contributing
Contributions are welcome — see CONTRIBUTING.md for the setup and the invariants that keep the trust guarantee intact.
Provenance
StudyDiff was built for Anthropic's Built with Claude: Life Sciences hackathon (Builder track). All application code in this repository was written from scratch during the event. Grounding uses OpenGATE and PDF extraction uses unpdf, both as published dependencies. Retrieval uses public NCBI E-utilities; extraction uses the Claude API.
License
MIT © Nick Lamb