# iris-eval/mcp-server [Health: Active]

**Category:** 📊 Monitoring  
**Repository:** https://github.com/iris-eval/mcp-server  
**GitHub Stars:** 9  
**npm Downloads (last month):** 2893  
**Views:** 3  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/iris-eval-mcp-server

## Description
MCP-native agent evaluation and observability server with trace logging, output quality evaluation, cost tracking, 12 built-in eval rules, real-time dashboard, and PII detection.

## Tools
Capabilities this server exposes over MCP:

- **log_trace** — Store one agent execution (input, output, tool calls, spans, cost, tokens) and get the trace_id later calls key on.

What it does. Writes one immutable trace to local storage; nothing is deduplicated. Only agent_name is required; store tool_calls, tools, cost and input/output so the rules can judge them. evaluate: true (with output) also scores it in this call, exactly as evaluate_output would. Full detail: iris://capabilities.

When not to use it. To score a stored trace (evaluate_output with its trace_id). To change one (delete_trace, then log again).

Returns. JSON: trace_id, status, cost_usd, cost_source, cost_estimate, evaluation.

Errors. IRIS_STORAGE_ERROR; IRIS_INVALID_ARGUMENT when evaluate is true without output. Unknown or malformed arguments are refused, naming the valid keys. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. evaluate_output — score the stored output; get_traces — query what was logged; delete_trace — remove one trace.
- **evaluate_output** — Score an output with deterministic rules: a ship verdict with its basis, per-rule evidence, what was unjudged.

What it does. eval_type picks a bundle or all (the default). A rule missing its input SKIPS, never passes: input is REQUIRED when eval_type="relevance"; tool_calls, tools, cost_usd, expected feed the rest. trace_id alone scores the stored trace. Full detail: iris://capabilities.

When not to use it. For semantic judgment (evaluate_with_llm_judge). As an input firewall: the rules read the output.

Returns. JSON: id, trace_id, reference_trace_id, verdict, coverage, provenance, run_id, interpretations, erased_at, eval_type, score, passed, rule_results, rules_evaluated, rules_skipped, insufficient_data, critical_failures, critical_skipped, categories, note, rules_changed.

Errors. IRIS_UNKNOWN_TRACE (nothing written); IRIS_STORAGE_ERROR. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. log_trace — record the run first; list_rules — what each rule needs.
- **get_traces** — Query stored traces with filters, pagination and sorting; optionally with the dashboard summary.

What it does. Read-only, local. Exact-match agent_name and framework, inclusive since and until, min_score and max_score on each trace's latest evaluation, and q: full-text search, ranked, with a snippet. limit is 1..1000 (default 50). A crossed range is refused, never returned as an empty page. Full detail: iris://capabilities.

When not to use it. To score a trace (evaluate_output) or create one (log_trace). As a live stream: poll with backoff.

Returns. JSON: untrusted, traces, total, limit, offset, summary, search.

Errors. IRIS_STORAGE_ERROR. Out-of-range bounds are refused, naming the values; no match is total 0, not an error. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. log_trace — record an execution; evaluate_output — score one output; delete_trace — remove one trace.
- **compare_runs** — Did this change make the agent worse? Compares two runs: worse, better, equivalent, or too little evidence to tell.

What it does. Pairs by case_key (McNemar exact), else Newcombe. A rule counts only where it ran; no `better` if the second run was judged on less. Refuses unlike runs unless force. Local, no model call. Full detail: iris://capabilities.

When not to use it. To score one output (evaluate_output). As a gate: it reports; blocking is your policy.

Returns. JSON: comparable, incomparable_because, forced, method, before, after, difference, paired, worse, better, improvement_withheld, call, smallest_detectable, equivalent_within, coverage, critical_rises, rules_tested, regressions, improvements, discordant, discordant_total, summary, dataset.

Errors. IRIS_INVALID_ARGUMENT (empty run id, no baseline, bad margin or dataset); IRIS_STORAGE_ERROR. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. log_trace — record into a run; get_traces — find a run's traces.
- **compare_traces** — How reliably does the agent answer the same question? Per-case pass rates, flaky cases, and an interval that respects repeats.

What it does. Groups evaluations by case_key (or session) and reports each case's pass rate with a Wilson interval, the flaky cases least reliable first, and an overall rate over cases, with an exact interval, rather than over pooled attempts. Deterministic, local, no model call. Full detail: iris://capabilities.

When not to use it. To compare two runs (compare_runs). To score an output (evaluate_output).

Returns. JSON: case_key, group_by, question, cases, attempts, overall, pooled, flaky_cases, by_case, summary.

Errors. IRIS_STORAGE_ERROR. No matching evaluations is cases 0, not an error. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. compare_runs — compare two runs against each other; log_trace — record an execution with a case_key; get_traces — read the traces themselves.
- **evaluate_runs** — Re-score every trace in a run under the current rules into a new run, so a rules change is compared, not overwritten.

What it does. Writes the verdicts into a NEW run; the source run is never modified. Traces already scored under the current ruleset are skipped, so a repeat call does no work. Local, no model call. Full detail: iris://capabilities.

When not to use it. To score a new execution (log_trace, then evaluate_output).

Returns. JSON: source_run, run, ruleset_hash, traces, evaluated, already_current, failed, passed, not_checked, summary, rules_changed.

Errors. IRIS_INVALID_ARGUMENT (empty run id, or the target run holds verdicts); IRIS_UNKNOWN_TRACE (source run empty); IRIS_STORAGE_ERROR. Unscorable traces are listed in failed. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. compare_runs — compare the two runs after; evaluate_output — re-score one trace; get_traces — read a run.
- **list_rules** — The rule inventory: every built-in rule with what it needs, its effective criticality and published accuracy, plus every deployed custom rule.

What it does. Read-only, no network. built_in is never narrowed by the filters; a rule whose needs are absent skips, never passes. Read critical and criticalSource before trusting passed: true. rules is the custom-rule store, filterable by eval_type and enabled_only. Full detail: iris://capabilities.

When not to use it. To add, remove or pause a rule (deploy_rule, delete_rule); built-in rules cannot be.

Returns. JSON: untrusted, rules, total, enabled_count, built_in, quarantined, plugins.

Errors. IRIS_INTERNAL_ERROR if the store cannot be read; a missing store is an empty list. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. deploy_rule — add a custom rule; delete_rule — remove, disable or re-enable one; evaluate_output — run the rules.
- **deploy_rule** — Deploy a custom rule that fires on every future evaluate_output call of its bundle — persisted, active immediately, audited.

What it does. Stores the rule and loads it into the running engine. eval_type says WHEN it fires (that bundle, and eval_type="all"); severity says what a failure does: low and medium lower the score, high and critical fail the verdict. definition.type picks the check, definition.config its keys. A taken name is refused unless replace is true. Full detail: iris://capabilities.

When not to use it. For a one-off check (custom_rules on evaluate_output). To pause a rule (delete_rule with enabled: false).

Returns. JSON: rule, replaced, warning.

Errors. IRIS_DUPLICATE_RULE; IRIS_INVALID_RULE_CONFIG, naming the field; IRIS_STORAGE_ERROR. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. list_rules — see what is deployed; delete_rule — remove or disable; evaluate_output — where it fires.
- **delete_rule** — Remove a deployed custom rule — or, with enabled, disable or re-enable it without removing it — effective on the next evaluate_output call.

What it does. Without enabled: deletes the rule and writes an audit entry; deleted is false when no rule has that id. With enabled: false stops it firing and keeps it stored, true re-enables it. Past evaluations are untouched. Full detail: iris://capabilities.

When not to use it. On built-in rules (they are not in the store). To replace a rule (deploy_rule with replace: true).

Returns. JSON: deleted, rule_id, toggled, enabled, rule.

Errors. IRIS_STORAGE_ERROR. A malformed rule_id is refused. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. deploy_rule — add or replace a rule; list_rules — find the id; evaluate_output — where the rule fires.
- **delete_trace** — Remove one stored trace by id; its spans go with it, and every evaluation linked to it keeps its verdict and loses its text.

What it does. Linked evaluations keep their verdicts and scores; their text is erased and erased_at is stamped. deleted is false when no trace has that id. Every deletion is audited at iris://audit. Full detail: iris://capabilities.

When not to use it. To expire data in bulk (retention.days). To change a trace: traces are immutable.

Returns. JSON: deleted, trace_id.

Errors. IRIS_STORAGE_ERROR. A malformed trace_id is refused. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. log_trace — store a trace; get_traces — find the trace to delete; delete_rule — the equivalent for custom rules.
- **evaluate_with_llm_judge** — Score an output with an LLM judge on your own key: a 0..1 score, a rationale, the spend.

What it does. Calls Anthropic or OpenAI with IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY; Iris never proxies. correctness needs expected, relevance input, faithfulness source_material. Worst-case cost is checked against max_cost_usd first. Full detail: iris://capabilities.

When not to use it. For length, keyword, PII or cost checks (evaluate_output, free).

Returns. JSON: id, reference_trace_id, score, passed, verdict, pass_threshold, self_reported_pass, disagreement, rationale, dimensions, model, provider, template, input_tokens, output_tokens, cost_usd, latency_ms, raw_response_id, warnings.

Errors. IRIS_JUDGE_NOT_ENABLED (recovery has the steps); IRIS_JUDGE_UNKNOWN_MODEL; IRIS_UNKNOWN_TRACE; IRIS_BUDGET_EXCEEDED; IRIS_PROVIDER_ERROR. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. evaluate_output — the free deterministic path; verify_citations — citation grounding.
- **verify_citations** — Is each citation in an output supported by its source? Extracts, fetches (opt-in, SSRF-guarded) and judges on your key.

What it does. Fetches only if the operator set IRIS_CITATION_ALLOW_FETCH=1, blocking private addresses. One judge call per resolved citation, within max_cost_usd_total. overall_score: supported / judged; null if none judged. Full detail: iris://capabilities.

When not to use it. With no citations (evaluate_output). Without IRIS_ANTHROPIC_API_KEY or IRIS_OPENAI_API_KEY.

Returns. JSON: id, reference_trace_id, overall_score, passed, verdict, total_unsupported, total_citations_found, total_resolved, total_judged, total_supported, total_cost_usd, warnings, citations.

Errors. IRIS_JUDGE_NOT_ENABLED, IRIS_JUDGE_UNKNOWN_MODEL, IRIS_UNKNOWN_TRACE (before any spend); IRIS_JUDGE_FAILED when every judge call failed. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. evaluate_with_llm_judge — semantic scoring; evaluate_output — the free deterministic path.

## Claude Desktop Quick Installation
Install path detected from listing signals. Uses `npx` (confidence: high):

```json
"mcpServers": {
  "mcp-server": {
    "command": "npx",
    "args": ["-y","@iris-eval/mcp-server"]
  }
}
```

## Documentation

## What iris-eval/mcp-server MCP server does

iris-eval/mcp-server MCP server provides local evaluation and observability for agent runs. It stores traces in SQLite on the machine running the server, then applies deterministic checks to agent outputs, tool calls, costs, PII, prompt-injection indicators, hallucination markers, and other evaluation conditions. Rules are inspectable and can be deployed, paused, edited, or removed.

The server also supports an optional LLM judge for semantic evaluations. Anthropic or OpenAI credentials can be supplied for that judge, which applies a per-evaluation cost limit. Citation verification can fetch pages referenced by an output, allowing cited claims to be checked against their sources.

## How it works

Agents can call MCP tools such as `log_trace`, `evaluate_output`, and `get_traces`. Additional tools expose rule listing and deployment, trace deletion, run and trace comparison, scoring, citation verification, and LLM-judge evaluation. The resulting verdicts and rule evidence are available through the local dashboard.

MCP calls are not an automatic interception layer: a model or application must request trace logging. For capture that should not depend on model behavior, an application can send traces to `POST /api/v1/traces` over HTTP. The endpoint can also run deterministic evaluations in the same request. It is served on the dashboard port, which defaults to port 6920, and is available only while the dashboard is running.

The dashboard presents failures and their evidence, supports viewing individual failures, and provides comparisons for runs over the same questions. All configured MCP clients use the shared Iris database, so traces can be viewed together across installations.

## Setup and configuration

Node.js 22.13 or newer is required. A quick health check is available with:

```bash
npx @iris-eval/mcp-server --self-test
```

Run the demo with `npx @iris-eval/mcp-server --demo`; it seeds a separate demo database and serves the dashboard at `http://localhost:6920`. Remove that data with `--demo-clear`.

For an MCP client, the installer can write the client’s configuration while preserving other configured servers:

```bash
npx -y @iris-eval/mcp-server install claude-code
```

Use `install --list` to see detected clients, or `install <client> --uninstall` to remove the configuration. A manual configuration can start the package with `npx -y @iris-eval/mcp-server`; adding `--dashboard` also starts the dashboard. The dashboard can be enabled with `IRIS_DASHBOARD=true`.

The LLM judge accepts an Anthropic or OpenAI API key through the corresponding Iris environment variable. Without those keys, deterministic evaluations remain available.

## Limitations and notes

Iris runs locally and does not send data elsewhere by default. Data leaves the machine when an OpenTelemetry endpoint, an LLM judge, citation checking, or a webhook is explicitly enabled. The LLM judge sends judged text to the selected provider; citation checks fetch cited pages. Webhook notifications contain identifiers, verdicts, and rule names rather than trace text.

The dashboard’s HTTP ingest endpoint is not available merely because the MCP transport uses HTTP; the dashboard must be started separately. The server’s MCP tools also do not automatically observe an agent’s activity, so applications requiring guaranteed capture should use the HTTP ingest endpoint or call the logging tools directly.

_Full upstream README: https://allmcps.com/mcp/iris-eval-mcp-server/readme_