LLM Preflight

Last reviewed: 2026-08-31 · As of: v2.10.0

Know whether an AI-generated LLM integration is safe before it reaches production. LLM Preflight is a local contract preflight for LLM integration changes: model, prompt, structured-output, and provider-call changes. It runs a small cross-provider preflight and compares validated output, response speed, tokens, and estimated cost.

Try it in 60 seconds

Create and run a deterministic local benchmark—no API key or network request:

python3 -m pip install llm-preflight
llm-preflight init
llm-preflight benchmark.json --no-save

From a source checkout:

python3 -m llm_preflight init
python3 -m llm_preflight benchmark.json --no-save

init never overwrites an existing config. It creates a mock benchmark so you can see the report and exit behavior before making a paid request. Its result is intentionally inconclusive (exit code 3): a local mock validates configuration and output handling, but cannot approve a live model.

Choose your path

• Validate a change. Compare an approved model, prompt, schema, or provider route with a candidate using the model-change guide.
• Review a new model. Discover metadata, deliberately probe a route, then prepare a bounded candidate smoke with the model-catalogue guide.
• Automate an established contract. Add the no-spend GitHub Marketplace Action or use CI and JSON output.
• Configure a coding agent. Start the local MCP server from a trusted repository with llm-preflight-mcp --workspace "$PWD", then use the MCP server guide for your client configuration.

Safety boundary

flowchart LR
    A[Integration change] --> B[No-spend validation\ndoctor, pricing, dry run]
    B --> C{Human reviews\nevidence and cost bound}
    C -->|Explicit approval| D[Bounded paid smoke]
    C -->|No approval or missing evidence| E[Inconclusive: fix or stop]
    D --> F[Local evidence for\nproduction approval]

LLM Preflight is local evidence, not production approval. It is not a hosted evaluation platform, tracing system, RAG framework, or public leaderboard. Its results apply to your account, network, prompts, and validation rules.

[!WARNING] Live benchmarks make paid API requests. Start with the no-key demo, preview the plan before a live run, and keep limits and repetitions small.

CLI, CI, and MCP

Works as a CLI, GitHub Action, and local MCP server. Every path starts with no-spend validation and planning; a live provider run remains an explicit, bounded human-approved step. See the GitHub Action guide or the MCP server guide.

For earlier releases, see the changelog.

Purpose

Mission: make every LLM integration change evidence-based before production.

Vision: AI-assisted software delivery where an agent can validate its LLM changes as routinely as it runs tests, while people retain control of spend and production approval.

Positioning: LLM Preflight is the fast, local, cross-provider contract preflight for AI-powered application changes. It is not a general evaluation, observability, or autonomous-deployment platform.

It is built for engineers and coding agents working on AI features: teams that need to check a real application contract against live model APIs before a model ID, prompt, parser, tool definition, or provider option ships. Read the north star and the AI implementation testing guide for the intended workflow and boundaries.

Common jobs

• Switch a model or provider. Run the bounded migration check, then add the contract test your feature needs.

• Check a prompt, schema, parser, or tool change. Define an explicit output contract before the smoke.

• Review a newly discovered model. Refresh metadata, then prepare—not run— a bounded candidate plan:

  bash

  Copy

  llm-preflight catalog refresh benchmarks/watch.json
  llm-preflight catalog prepare benchmarks/watch.json \
    --against benchmarks/approved.json --output benchmarks/candidates.json
  llm-preflight benchmarks/candidates.json --migration-check --dry-run
  

  Only explicitly approved, fully evidenced models proceed to paid work; see the model catalogue guide.

• Investigate a provider or price change. Run --doctor, --pricing-check, and a dry-run; report a suspected regression through the redacted issue forms.

• Automate a known contract. Use the no-spend GitHub Action or the CI guide with a saved baseline and --ci.

It measures deterministic test validity, end-to-end latency (p50/p95), time to first token, throughput when the stream is incremental and usage is available, token totals, and estimated cost. Result files retain request metadata and per-request observations for reproducibility.

"Deterministic" describes the validator, not the model: every response is checked against explicit structural rules — a regular expression, a JSON shape, an exact routing label — so the same response always produces the same verdict. The tool does not score semantic quality; that is your task-specific evaluation, and it stays out of scope on purpose.

What live evidence looks like

A completed preflight retains per-request observations and a machine-readable decision: contract validity, latency (including TTFT where observable), token usage, estimated cost, pricing evidence, and blocking warnings. The terminal summary is a convenience; automation should consume the saved JSON decision.

That evidence applies to your account, network, prompts, and validator at one time—not a universal model ranking. For a complete interactive example, see interactive runs.

First live run

Python 3.10+ is required. There are no third-party runtime dependencies: pip install llm-preflight installs this package and nothing else, and the CLI runs on the Python standard library alone. Development tools (pytest, ruff, mypy) are optional extras that never reach a production install.

cp benchmark.example.json benchmark.json
cp .env.example .env.production
# Edit benchmark.json and add only the provider keys you use.
python3 -m llm_preflight benchmark.json --dry-run
python3 -m llm_preflight benchmark.json

The CLI reads .env.production beside the config without overriding environment variables already set by your shell. Use --no-env-file or --env-file PATH when needed. Runs print a terminal report and, unless --no-save is used, write JSON and Markdown results under results/.

Install the command globally in a virtual environment if preferred:

python3 -m pip install llm-preflight
llm-preflight --init

Run --doctor and --dry-run before the final command. They make no generation requests; the final command is the paid work.

Change a model safely

This is the core workflow. Put your approved model and candidate model in one config, then run the small response-and-contract preflight:

llm-preflight benchmark.json --migration-check --dry-run
llm-preflight benchmark.json --migration-check

It sends three short representative cases to each selected model, once each. It answers: did the API work, did each response meet the basic contract, and how quickly did the provider start and finish responding? It is a cheap compatibility check, not a statistical performance conclusion.

When that passes, run the task-specific checks that match your application—for example exact-routing-check or structured-output-check—before approving a switch. Use custom contract tests to express the outputs your own feature must preserve.

Using a coding agent

Give an agent the same evidence you would use yourself: a reviewed config, an explicit output contract, and a dry run before paid work. Start with the recommended five-check suite:

# No generation request: inspect credentials, model selection, and paid-work plan.
llm-preflight benchmark.json --doctor --json
llm-preflight benchmark.json --tests agent-smoke --smoke --dry-run --json

# Paid run, only after reviewing the plan.
llm-preflight benchmark.json --tests agent-smoke --smoke --json --no-save