Eval View

What hidai25/eval-view MCP server does

The hidai25/eval-view MCP server project documents EvalView, a regression-testing framework for AI agents. It treats an agent run as a snapshot: the recorded behavior includes the tools used, their parameters, their order, and the resulting output. Later runs are compared with that stored baseline so changes can be reviewed as passed checks, tool changes, or regressions.

This approach is intended for cases where an agent can continue returning successful HTTP responses while its behavior has changed. For example, a model or prompt update may cause a different tool to be called, a clarification step to be skipped, or output quality to decline. A new baseline can be accepted with another snapshot run when the changed behavior is intentional.

How it works

The main workflow has two commands. evalview snapshot records the current behavior of the configured agent. evalview check executes the tests again and compares the results with the saved baseline. The comparison covers the complete tool-call trajectory rather than only the final response.

The core tool and sequence comparison is deterministic and does not require an API key. An LLM judge can be added when output-quality scoring is needed, and semantic comparison using embeddings is optional. Running the agent itself may still call a model or provider and therefore may incur provider charges; disabling the judge does not disable those agent calls.

EvalView supports multi-variant baselines, with up to five valid paths for nondeterministic behavior. It can also run statistical or pass@k evaluations, record and replay cassettes, check for model drift, monitor production behavior with Slack alerts, and generate regression tests from incidents. These are documented as additional features beyond the basic snapshot and check loop.

Setup and configuration

Install the Python package with:

pip install evalview

A demonstration is available through evalview demo. For a configured agent, create a baseline first, then run checks after changes:

evalview snapshot
evalview check

The project can test agents exposed through an HTTP endpoint, such as evalview check --agent http://localhost:8000/invoke. It also provides a Python API; importing gate from evalview allows a test directory to be evaluated programmatically, with a Boolean pass result and per-test diffs available on the returned value.

Tools and capabilities

The documented EvalView capabilities include:

• Recording golden behavior snapshots.
• Comparing tool names, parameters, order, and outputs.
• Reporting changed tool trajectories and lower-quality results.
• Running checks as a CI merge gate.
• Posting pull-request comments containing diffs, cost and latency changes, and a pass/fail result through the supplied GitHub Actions example.
• Testing LangGraph, CrewAI, OpenAI, Claude, Mistral, Ollama, MCP-based agents, and arbitrary HTTP APIs.
• Running multi-turn tests, statistical evaluations, cassettes, drift canaries, production monitoring, and incident-based test generation.

Limitations and notes

The hidai25/eval-view MCP server README does not list MCP tool names, input schemas, transport settings, or compatibility instructions for Claude Desktop, Cursor, Windsurf, or Cline. Its concrete usage documentation describes the evalview CLI, Python library, HTTP agent endpoint, and CI action instead. Do not assume that installing the Python package alone exposes a standalone MCP transport.

The latest published release described in the README is version 0.8.1. Its OpenAI Assistants adapter still uses the Assistants API, while a Responses API migration is described as unreleased source. Users of openai-assistants therefore need to consult the migration guidance before testing. Other listed adapters are described as unaffected.

Provider credentials may be needed by the agent or optional evaluator, even though the deterministic comparison itself can run without an API key. Live provider checks can also involve paid API usage, while the local core checks use no paid inference calls.