Thoth

Agentic systematic literature reviews — with every citation checked against the source.

Named for Thoth, ancient Egypt's ibis-headed god of writing and scribes.

Try the live demo · See a sample review · Public eval dashboard · Connect via MCP

What is Thoth?

Systematic literature reviews are slow to write — and when you ask an LLM to write one, it confidently invents citations and statistics that aren't in any paper.

Thoth does both halves and checks its own work. Give it a research question and it discovers relevant papers, reads them, drafts an evidence-grounded review — then runs a verification pass (cite_check) that compares every cited claim against the source paper and flags anything unsupported before you read the draft. The result is a review with a critic score, a citation-faithfulness percentage, and a per-claim audit you can trust.

It runs as a polished web app, a public eval dashboard, and an authenticated MCP server your AI assistant can call directly.

See it work

Claude.ai catches 6 fabricated citations in a real draft — using Thoth's audit:

Connected to Thoth via the official MCP Registry, Claude calls get_citation_audit on one deliberately-weak review (faithfulness 0.13 for that single review) and identifies all 6 unsupported claims — every one citing the same paper, with invented percentages that aren't in the source. This is cite_check doing its job: it's a single-review audit sample, not the golden-set aggregate (see /evals).

Every claim, scored against its source — the /showcase review (no login needed). The figures on this card (critic 4.2/5, faithfulness 75%, 8/8 citations checked, 2 unsupported) are this one review's scores — a worked example, not the aggregate:

Evaluated in public — /evals tracks citation recall / precision / faithfulness / coverage over an 18-question versioned golden set (7 of 18 populated at this commit), regenerated in CI and published with the last-run date, so a regression is a public, falsifiable signal:

You approve every step — three human-in-the-loop gates (review plan → review discovered papers → approve included papers); nothing runs unattended:

Key features

• 🔎 cite_check — verifiable citations. Every [paper_id] in the draft is scored against the cited paper and labelled supported / unsupported / unclear, so the LLM can't quietly hallucinate a citation. On the public golden set, the citations it does surface are accurate — citation precision 97%, recall 74% — and the verdict report is published per claim, not summarised away. This is the core differentiator: the citations are measured, not asserted.
• 🌐 Outbound web search (v2 — under active evaluation). An outbound discoverer → fetcher → screener path is wired across OpenAlex, arXiv, and Exa: it fetches open-access PDFs, OCRs them, and screens each against your plan, so you can run uploaded-only, hybrid, or fully autonomous discovery. The discovery and screening axes are v2 and still being calibrated — they're tracked openly on /evals (both currently at 0%) rather than shipped as a silent claim.
• 🔌 Authenticated, registered MCP server. OAuth 2.1 + PKCE + Dynamic Client Registration via Clerk, SHA-256 audit logging, rate limits — listed in the official MCP Registry. Most public MCP servers ship with no auth; this one doesn't.
• 📊 Public eval dashboard. Recall / precision / faithfulness / coverage over a versioned golden set, regenerated in CI and stamped with the last-run date, rendered at /evals — an eval regression is a public signal, not a hidden one.
• 💸 6 LLM providers, $0/mo by default. Swap providers with one env var; the Mistral free tier runs the whole thing, and the entire stack deploys on free tiers for $0/mo.

🚀 Quickstart

Try it now (nothing to install):

• Open the live demo → and build a review, or browse a finished one →.

Connect it to your AI assistant — paste this into claude.ai (Pro/Max), Claude Desktop, Cursor, or any MCP client (OAuth runs in your browser; no token to copy):

https://thoth-slr.vercel.app/api/mcp/mcp

Adding Thoth as a custom connector in claude.ai — OAuth runs in your browser (Clerk + DCR), no token to copy.

Run it locally:

git clone https://github.com/ahmedEid1/thoth.git && cd thoth
cp .env.example .env        # Clerk + Trigger.dev keys + MISTRAL_API_KEY
docker compose up -d        # postgres, minio, langfuse
pnpm install && pnpm prisma migrate dev
pnpm dev                    # Next.js on :3000
pnpm dev:trigger            # Trigger.dev worker (separate terminal)

Full setup, the agent pipeline, and the v2 flow: docs/architecture.md.

Proof

For engineers

Thoth is a LangGraph StateGraph driven by a Trigger.dev worker, with durable human-in-the-loop gates, a per-run cost cap, and exactly-once gate delivery. Next.js 16

• TypeScript (strict), Postgres + Prisma, Clerk auth (web + OAuth 2.1 for MCP), S3-compatible storage, Mistral OCR, Langfuse tracing.