Opengate MCP

Open Grounded AI Testing & Evaluation

Deterministic, gold-anchored verification for evidence-grounded AI — no LLM judge.

Quick start · Architecture · Surfaces · Examples · Roadmap · Contributing · Changelog

Evidence over plausibility. OpenGATE verifies AI systems that must justify every answer from source material — RAG pipelines, document-QA tools, legal and scientific assistants. It answers one question above all: can the system prove its answer from the evidence it was given?

The check is deterministic — no LLM-as-judge, no grader model, no six-point verdict scale. Required facts must be present, every number must trace back to the source, and when the context can't answer, the system must abstain rather than fabricate. Because it's pure logic, it's reproducible, free, and fast enough to run on every answer or gate on every commit.

As AI moves into high-stakes domains, evaluation is becoming as fundamental as automated testing is in traditional software. OpenGATE turns grounding failures into numbers you can track, and gates every prompt, model, or workflow change against a baseline — so reliability can't quietly regress.

Quick start (60 seconds)

No API key needed — the offline suite runs deterministic scorers against the bundled gold set:

npx @pharmatools/opengate           # run the offline evaluation suite
npx @pharmatools/opengate init      # scaffold gold cases + HTTP config + a GitHub Action

OpenGATE — 39 case(s), online=false, adapter=refcheckr

  ✓ citation-detection   PASS
      perClaim_exactSetRate      100.0%
      perClaim_jaccardMean       100.0%
      supportedStyle_accuracy    100.0%
  ⊘ grounding            SKIPPED — online scorer (pass --online)

Point opengate.http.json at your endpoint and add --online --ci to gate your own system. Full walkthrough: Getting Started.

One check, many surfaces

The same deterministic grounding logic ships wherever your stack lives:

Architecture

Scorers never talk to a system directly — they reach it through a small adapter, so the methodology travels and only the gold set changes. In the development loop it sits where CI sits: change a prompt, model, or pipeline; the regression gate diffs the new scorecard against the baseline — improved or held deploys, regressed fails the build.

Why not DeepEval?

Use both — evals measure, OpenGATE verifies. General-purpose frameworks like DeepEval and OpenAI Evals evaluate AI systems broadly, usually with an LLM judging the output. OpenGATE verifies the narrower, harder promise: that every answer is grounded in evidence:

• Provenance is first-class — does the cited passage actually exist, verbatim, in the source?
• No LLM judge — scores are deterministic checks against hand-labelled gold, so they're reproducible and free to run in CI; your judgment lives in the gold set, not a grader model's.
• Regression detection is first-class — every run is diffed against a per-adapter baseline; a drop fails the build.

Pair a general framework for broad quality metrics with OpenGATE to gate the grounding.

Core concepts

Gold cases — hand-labelled benchmark cases (datasets/cases/): source text, the claims that should be extracted, the sentences that should not be, and reference snippets with known-correct verdicts. Copy _template.json to add one; format in datasets/SCHEMA.md, labelling rules in datasets/LABELING-GUIDE.md.

Scorers — one module per metric family (src/scorers/):

Offline scorers run with no API key — fast enough for every commit. Online scorers exercise a live system through an adapter.

Scorecards — every run writes results/<timestamp>.json stamped with the git SHA, so any result is reproducible and auditable. Per-model runs carry a run_model label, turning the results directory into a measured comparison (accuracy × hallucination × latency × cost).

Regression gate — --baseline saves a reference; later runs print per-metric deltas (▲/▼ in percentage points) and --ci fails the build on any drop. Baselines are per-adapter, so one system's scorecard can't clobber another's.

HTML report — add --report (or opengate report) for a self-contained dashboard: pass/fail per scorer, deltas vs baseline, every named failure. One file, no server, no dependencies.

Evaluating your own system

An adapter is one file: two base exports — onlineAvailable(), onlineConfigHint() — plus at least one complete capability (e.g. grounding → answer()). Scorers check adapter.capabilities and skip cleanly across the boundary; adapters are validated at load with messages naming every missing export.

OPENGATE_ADAPTER=./adapters/my-system.mjs npm run eval:online

For REST-backed systems there's a no-code path: the bundled generic HTTP adapter reads endpoint paths and headers from opengate.http.json (${ENV} interpolation, built-in latency/token capture). Full contract and a minimal skeleton: ADAPTERS.md.

Examples

• Evaluating a NIM-powered RAG agent — builds a RAG agent on an NVIDIA NIM model and gates its answers' grounding with OpenGATE, deterministically and with no LLM judge. Includes a runnable Python notebook (opengate-grounding) and a Node adapter for the CI gate.

Proven in production

Four PharmaTools products run on OpenGATE in CI — four different capability shapes, one evaluation standard. Run against RefCheckr's gold set, OpenGATE:

• surfaced a silent parse-failure mode affecting ~50% of multi-claim verdicts, eliminated with enforced structured output (→ 0);
• halved passage hallucination (5.8% → 2.4%) by driving a measured production model change — a decision made on numbers, not reputation;
• holds claim extraction at 0.91 F1 with 0.93 recall on the committed baseline (run-to-run spread 0.86–0.94 — the splitter is an LLM), and is currently failing its own gate: 2 known non-claims leak into extraction on most runs, an open splitter issue the gate reports rather than rounds away.