Quellgeist

First-line incident triage you can trust: ranked root-cause hypotheses where every claim cites a real evidence handle — and the agent abstains rather than guess.

Quellgeist is a model-agnostic AI agent for first-line production-incident triage. It runs a legible JSON-action ReAct loop over read-only tools (structured logs + recent deploys + metric time-series), then emits a structured Diagnosis: confidence-ranked root-cause hypotheses, each backed by a structured evidence handle (LogRef.id / CommitRef.sha / MetricRef.id) the agent actually saw — never free text. Two ideas set it apart:

• Cite-by-structured-handle. Evidence is a checkable handle, not a sentence, so a fabricated citation is measurable and deterministically rejected by a keyless fabrication check — not a matter of fuzzy string-matching.
• Abstain-over-hallucinate. A confidently-stated wrong cause is the worst possible answer, so "insufficient evidence" is a first-class outcome.

Status: Wave 4 complete — the fine-tune works. The DR-0020 QLoRA fine-tune of the local reasoner (Qwen3-4B, served via Ollama) took it from the base's 0/16 holdout to 12/16 — zero fabrication, zero speculative-filtering, and cheaper than the base — while beating a 31B frontier (Gemma-4-31B, 10/16) on the same holdout at $0, fully offline. Non-memorisation is triangulated three ways (fixtures ≈ holdout; core-fresh ≥ core-overlap; structure probe 7/10). Two honest limits: the resource_exhaustion class didn't transfer (0/N; the frontier passes it), and adversarial-abstention recall is 6/12 at the system level — a ceiling the 31B frontier shares (also 6/12), not a fine-tune regression. When this agent misses it's incomplete or too cautious, never confidently fabricating. See Status & roadmap · fine-tune case study.

Why it's different

What it is / what it's NOT

• It is: a first-line triage agent — ranked, evidence-cited root-cause hypotheses (or an honest abstention) from read-only logs/deploys/metrics, over a model-agnostic loop that runs on a hosted frontier model or a local 4B.
• It is NOT: an autonomous remediator (it never mutates prod — resolution verification is a deferred, cut-first wave); a production-hardened service (the demo is a deliberate toy); or a general-purpose agent. The holdout it's measured on is out-of-vocabulary but in-structure — not a claim about unseen incident shapes or real production data.

Quickstart (~30 seconds to a broken service + structured logs)

Requires uv and Python 3.12+.

See a real-shaped diagnosis in one keyless command (no model, no API key):

uv sync && uv run quellgeist diagnose --demo     # renders the demo incident's cited postmortem

Then run the full loop against the live toy service:

uv run uvicorn demo.app.main:app          # 1. start the toy service (leave running)

# --- in a second shell, from the repo root ---
uv run python -m demo.chaos.bad_deploy    # 2. inject a simulated bad deploy
curl -s localhost:8000/login              # 3. trip /login -> 500s + structured error logs
uv run quellgeist diagnose --show-trace   # 4. diagnose live (needs a model; see below)

uv run python -m demo.chaos.reset         # back to a green slate

The live step needs a reasoner — see Running the model. Without a key, quellgeist diagnose exits 1 with a one-line error + hint (never a traceback); --demo always works keyless and renders the same output shape deterministically from gold.

Architecture

A custom, legible loop is the orchestration layer; the three read-only tools are the evidence interface; the Diagnosis schema is the contract that the postmortem renderer and the eval judge both read.

flowchart TD
    trigger(["incident trigger - CLI"]) --> loop
    model["reasoner via LiteLLM<br/>(Gemini or local Qwen, swappable)"] -. "chat completion" .-> loop

    subgraph loopbox["model-agnostic JSON-action ReAct loop"]
      loop["run_loop()<br/>decide, call tool, observe, repeat"]
    end

    loop -- "query_logs" --> logs["logs tool<br/>structured JSONL, stable ids"]
    loop -- "get_recent_commits" --> commits["commits tool<br/>deploy_log.json, shas"]
    loop -- "query_metrics" --> metrics["metrics tool<br/>time-series, named series"]
    logs -- "rows + ids" --> loop
    commits -- "commits + shas" --> loop
    metrics -- "series + names" --> loop

    loop --> diag["Diagnosis (schema.py)<br/>ranked hypotheses citing<br/>LogRef.id / CommitRef.sha / MetricRef.id, or abstains"]
    diag --> pm["postmortem renderer<br/>deterministic Markdown"]
    diag --> judge["eval judge<br/>fixture scenarios, CI gate"]

All three tools are also exposed as MCP servers over stdio (python -m quellgeist.servers.logs_mcp, …commits_mcp, …metrics_mcp). The agent currently reuses the same tool functions in-process behind a ToolSpec registry; a stdio MCP-client path (the agent driving the servers over the wire) is on the roadmap (DR-0010).

Deep dive: docs/architecture.md walks the full pipeline (loop → tools → verifier → postmortem), a sequence diagram, the module map, and the cross-cutting design decisions.

The servers publish to the Official MCP Registry on each tagged release (see docs/publishing.md); once published each is runnable with uvx --from quellgeist quellgeist-logs-mcp (or …-commits-mcp / …-metrics-mcp).

Example session

Inject the bad deploy — it drops a marker that flips verify_token into a NoneType regression and writes a deploy_log.json whose offending commit landed just before the errors (illustrative stdout — the timestamp reflects when you run it; paths shown relative to the repo root):

$ uv run python -m demo.chaos.bad_deploy
injected bad deploy a1b2c3d (touched demo/app/auth.py) at 2026-06-24T12:22:43Z
  marker:     demo/.bad_deploy
  deploy log: demo/deploy_log.json
next: hit /login to generate the 500s, then `quellgeist diagnose`

With a reasoner configured, quellgeist diagnose reads the logs + deploys and emits a postmortem. The CI environment has no validated model key (DR-0012), so the diagnosis below is rendered from gold — built deterministically from the fixture's labelled cause and evidence handles via render_postmortem, not live model output:

# Incident Postmortem (rendered from gold)

## Root-cause hypotheses

### 1. Bad deploy a1b2c3d (10:01:50Z) refactored auth.py and introduced a NoneType error in verify_token; /login 500s begin ~20s later at 10:02:12Z.  (confidence: 1.00)

Evidence:
- log #2
- commit a1b2c3d

Reproduce that render yourself (no model needed):

uv run python - <<'PY'
from evals.scenarios.generator import load_scenario
from quellgeist.agent.schema import Diagnosis, Hypothesis
from quellgeist.output.postmortem import render_postmortem

s = load_scenario("evals/scenarios/fixtures/bad_deploy_0001.json")
gold = Diagnosis(hypotheses=[
    Hypothesis(cause=s.gold_cause, confidence=1.0, evidence=s.gold_evidence_refs)
])
print(render_postmortem(gold, title="Incident Postmortem (rendered from gold)"))
PY

The point isn't the prose — it's that log #2 and commit a1b2c3d are exact handles into the real signals, not paraphrases. A live run additionally fills in a one-line summary and suggested actions, and abstains outright when the evidence is too weak to name a confident cause.

Write the postmortem to a file with --out postmortem.md, or as a self-contained HTML page with --out postmortem.html (or --format html) — same deterministic render, no external assets.

Running the model