redline

Catch prompt regressions before they ship.

Automatic eval suites from the prompt logs you already have.

redline turns real prompt-response logs into local regression tests. It selects representative cases, replays your changed prompt, and shows the behavioral diff before a bad prompt reaches users.

Website · Docs · MCP · MCP Registry · Security · License

Start Here

Install from PyPI:

python -m pip install redline-ai

Run the guided local app with the public proof loaded:

redline app --demo

This generates the public demo reports, opens the local product app, and shows the full import -> suite -> eval -> review workflow. The demo catches ten synthetic regressions without API keys, private logs, a cloud account, or an LLM judge.

Prefer terminal output first:

redline demo --public --compact

The demo writes JSON, Markdown, and self-contained HTML reports under .redline/demo.

Ask redline what to do next:

redline status --reports-dir .redline/demo/reports

status reads local config, suites, reports, history, and audit evidence, then prints the next command instead of leaving you to infer the workflow.

Open the guided local product app on existing reports:

redline app --reports-dir .redline/demo/reports

The app is a local, copy-command workflow: import logs, generate suites, run evals, review regressions, record history, and export CI/MCP setup without the browser executing shell commands.

On headless CI or remote shells, skip browser opening and use the printed HTML path:

redline app --reports-dir .redline/demo/reports --no-open --out .redline/app.html

Product Proof

redline has two proof paths: a fast first-run demo and a larger public-data dogfood run.

These screenshots are local artifacts from the 100-row internet dogfood run.

Dashboard	HTML report

What Is redline?

redline is an open-source, local-first eval tool for AI teams. It uses logs you already have: prompts, outputs, support tickets, traces, model responses, and production JSONL exports.

Instead of asking you to hand-write evals first, redline generates the first suite from real behavior. You can then run that suite every time a prompt, model, or runner changes.

No cloud account is required. No manual test writing is required. No LLM judge is required for the core regression signal. The package has zero runtime dependencies, which keeps installs fast and the default supply-chain surface small.

How It Works

redline gives you three primitives that cover the prompt-regression loop:

For a first pass on two local logs, use one command:

redline quick-check logs/baseline.jsonl logs/candidate.jsonl --open

It generates a temporary suite, writes JSON/Markdown/HTML reports plus a guided local app under .redline/quick-check, opens the focused HTML report, and prints the concrete behavioral diff. Use --open-app when you want the guided review workflow to open instead of only the focused report.

1. Logs

Start with prompt-response data you already have. Import JSONL, convert exports from tools like Langfuse or Helicone, capture OpenAI/Anthropic SDK calls, or add bounded FastAPI/ASGI middleware.

redline import downloaded.jsonl --detect
redline import downloaded.jsonl --auto-map --preview 3
redline import downloaded.jsonl --auto-map --out logs/baseline.jsonl
redline import downloaded.jsonl --input-field instruction --output-field response --preview 3
redline import downloaded.jsonl --input-field instruction --output-field response --out logs/baseline.jsonl
redline import langfuse-export.jsonl --preset langfuse --out logs/baseline.jsonl
redline suite logs/baseline.jsonl --out redline-suite.json
redline cases redline-suite.json

Use --detect when you do not know the field names. Use --preview when the export is new to you; it shows mapped, redacted sample rows without writing a baseline file.

Suite generation prints a readiness score and improvement suggestions. That score measures suite health, not model quality or candidate safety.

2. Suite

redline groups behavior into deterministic signatures and selects representative cases first. You can add pinned edge cases and explicit requirements when a scenario must never be missed.

redline cases redline-suite.json
redline suite add redline-suite.json --prompt "..." --response "..."

3. Eval

Replay a changed prompt or compare candidate outputs. redline names the behavior that broke: missing JSON keys, URLs, numbers, tables, code blocks, refusals, empty answers, or requirement failures.

redline eval --prompt prompts/v2.txt
redline diff redline-suite.json logs/candidate.jsonl

Product Promise

In under five minutes, on a real prompt log, redline should catch one regression you did not want to ship.

That promise is intentionally narrow. redline is not a hosted eval platform, a generic score, or a replacement for human judgment. It is the local safety loop between "I changed the prompt" and "this is safe enough to merge."

Real Workflow

Build a suite from baseline logs:

redline suite logs/baseline.jsonl --out redline-suite.json

Evaluate a changed prompt file through your configured runner:

redline eval --prompt prompts/v2.txt

Or compare candidate outputs you already generated:

redline diff redline-suite.json logs/candidate.jsonl

When redline finds a blocking change, it exits non-zero for CI and prints the reason:

REGRESSION case_004
- candidate missing JSON keys: owner, required_action
- candidate missing URL: https://example.com/policies/refunds

Confidence: HIGH | fix blocking cases before shipping

What redline Catches

redline is deterministic and local-first by default. Optional judge commands are available for ambiguous changed cases, but redline does not call a cloud model unless you explicitly configure that command.