# LS-SIEM-LLP/qa-probe [Health: Active]

**Category:** 🔒 Delivery  
**Repository:** https://github.com/LS-SIEM-LLP/qa-probe  
**GitHub Stars:** 2  
**npm Downloads (last month):** 76  
**Views:** 2  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/ls-siem-llp-qa-probe

## Description
Probes your live API and classifies why each endpoint failed (root cause + evidence + calibrated confidence) over MCP, so your AI assistant debugs from evidence instead of guessing. Deterministic rules, no black box. Works with FastAPI, Express, Next.js, tRPC, and GraphQL. npm i -g qa-probe

## Tools
Capabilities this server exposes over MCP:

- **qa_probe_get_report** — "Show me all broken routes and their scores"
- **qa_probe_explain_failure** — "Why is /reports showing no data?"
- **qa_probe_probe_endpoint** — "Is GET /users returning data right now?"
- **qa_probe_get_graph** — "Which backend routes does /dashboard call?"
- **qa_probe_get_blast_radius** — "What pages break if the users endpoint goes down?"
- **qa_probe_suggest_fix** — "What should I do about feature_flag_disabled?"
- **qa_probe_run_analysis** — "Run a full QA check and give me the summary"
- **qa_probe_label** — "Mark this finding as expected / a real bug"

## Claude Desktop Quick Installation
Install path detected from listing signals. Uses `npx` (confidence: high):

```json
"mcpServers": {
  "qa-probe": {
    "command": "npx",
    "args": ["-y","qa-probe"],
    "env": {
      "QA_USER": "",
      "QA_PASS": ""
    }
  }
}
```

**Requires environment variables:** `QA_USER`, `QA_PASS` — the values above are empty placeholders; fill in real credentials before running (see the repository for what each one is for).

## Documentation & README

# qa-probe

**Find out exactly why your pages are blank - in under 5 minutes.**

qa-probe is a Node.js command-line tool that reads your React frontend, maps every
API call to its backend route, probes those endpoints live with real auth, and tells
you the **root cause** when something is wrong - not just a status code.

```
Overall score: 74/100

Root causes detected:
  empty_db:              5 route(s)   seed the database
  feature_flag_disabled: 3 route(s)   enable HAS_BILLING, HAS_REPORTS
  contract_mismatch:     1 route(s)   frontend calls /api/users/, backend serves /users

Route          Score   Status     Root cause
-----------    -----   --------   ----------------------
/dashboard     100     healthy
/users          80     healthy    empty_db
/billing         0     broken     feature_flag_disabled
/reports        50     degraded   empty_db
/settings       85     healthy
```

It works best on React apps backed by **FastAPI, Express, Next.js, GraphQL, or tRPC**
with an OpenAPI spec available. Built by the LightShield SIEM team and battle-tested
against a production SIEM.

---

## The problem it solves

Your dashboard loads. Nothing crashes. But every table is empty, every chart shows
zero, and you have no idea why. The cause is almost always one of:

- The frontend calls `/api/users/` but the backend route is `/api/users` (trailing slash)
- A feature flag (`HAS_BILLING=false`) silently disabled an entire router
- The database is connected but the table has no rows
- A backend refactor renamed a field and the frontend still reads the old name
- An endpoint is missing its auth guard, or leaks data without a token

qa-probe builds a source-aware map from frontend calls to live backend responses, then
explains each failure in one line with a fix hint. It complements Playwright (user
journeys) and Schemathesis (contract fuzzing) - it is the layer that answers
"why is this page blank?" first.

---

## Quick start

### 1. Install

```bash
npx qa-probe run        # use without installing
# or
npm install -g qa-probe
```

### 2. Create a config

`qa-probe.config.js` in your project root:

```js
module.exports = {
  baseUrl: 'http://localhost:8000',      // your backend
  frontendSrc: './frontend/src',          // your React source
  routerFile: './frontend/src/App.tsx',   // your router file

  auth: {
    type: 'bearer',
    loginUrl: '/auth/login',
    credentials: {
      username: process.env.QA_USER,       // never hardcode credentials
      password: process.env.QA_PASS,
    },
    tokenPath: 'access_token',
  },
};
```

### 3. Run it

```bash
QA_USER=tester QA_PASS=secret npx qa-probe run
```

qa-probe will parse your frontend, fetch `/openapi.json`, probe every endpoint with
real auth, and write a full report to `.qaprobe/` (`report.md`, `report.json`,
`ai-context.md`, `report.html`). A summary table prints to the terminal.

---

## Understanding the report

**Route status**

| Status   | Meaning |
|----------|---------|
| healthy  | endpoints respond, data present, schema valid |
| degraded | works but has an issue (empty data, slow response) |
| broken   | unreachable, 404, 5xx, or a security/logic failure |

**Common root causes**

| Root cause              | What happened | What to do |
|-------------------------|---------------|------------|
| `feature_flag_disabled` | router not registered (a `HAS_*` flag is false) | enable the flag and restart |
| `contract_mismatch`     | a similar route exists (slash/casing) | align the frontend path |
| `missing_route`         | 404, not in the OpenAPI spec | fix the typo or `include_router()` |
| `empty_db`              | 200 OK but empty array | seed the database |
| `precondition_required` | 428 - terms/onboarding/MFA gate not satisfied | clear the gate for the probe account |
| `auth_scope_mismatch`   | 401/403 - wrong role/scope | use a user with the required permissions |
| `schema_mismatch`       | a field was renamed/removed vs the spec | align frontend, backend, and spec |
| `auth_bypass`           | endpoint returns 200 without auth | add the missing auth guard |
| `assertion_failed`      | response violated a declared invariant | fix the response or the assertion |
| `server_error`          | 5xx | check backend logs |

**Every result is verifiable.** Each diagnosis carries its **evidence** (the request,
a snapshot of the response, timing) and a **confidence** level - `high`, `medium`, or
`none`. A `confidence: none` / `unknown` result is explicitly **not** a confirmed pass.

**Score:** each route is 0-100; the overall score is the average. Use
`--fail-under 80` to make CI fail when the score drops.

---

## Use cases

- **"Why is this page blank?"** - the everyday case. Get a root cause and a fix hint
  in one run instead of digging through network tabs and logs.
- **CI gate** - `qa-probe run --fail-under 80` blocks a deploy when pages regress.
- **Demo / staging sign-off** - confirm an environment actually works before a customer
  call, not just that it returns 200s.
- **Catch contract drift** - a backend refactor renames a field or moves a route;
  qa-probe flags the exact endpoint and frontend call affected.
- **Security smoke** - find endpoints missing an auth guard or leaking PII (opt-in,
  read-only, see Security checks).
- **Logic checks** - assert response invariants (valid enum values, non-negative
  counts, required fields) without writing a test suite.
- **AI-assisted debugging** - let Claude, Cursor, or Codex query the QA state and
  explain failures in plain English (see below).

---

## Use it with an AI assistant (Claude Code, Cursor, Codex)

qa-probe ships a **Model Context Protocol (MCP) server**. Point your AI assistant at
it and the assistant can read your QA data and explain failures directly - no manual
report-reading.

### Setup

Run qa-probe once to populate the cache:

```bash
npx qa-probe run
```

Then add the MCP server. For **Claude Code** / **Cursor**, add to `.mcp.json` at your
repo root:

```json
{
  "mcpServers": {
    "qa-probe": {
      "command": "npx",
      "args": ["qa-probe", "mcp"]
    }
  }
}
```

Restart the assistant. Now you can ask in plain English:

- "Why is /reports showing no data?"
- "Which pages break if the users endpoint goes down?"
- "Show me all broken routes."
- "Mark the empty /alerts result as expected." (records feedback, reapplied next run)

### Available MCP tools

| Tool | Ask it |
|------|--------|
| `qa_probe_get_report`     | "Show me all broken routes and their scores" |
| `qa_probe_explain_failure`| "Why is /reports showing no data?" |
| `qa_probe_probe_endpoint` | "Is GET /users returning data right now?" |
| `qa_probe_get_graph`      | "Which backend routes does /dashboard call?" |
| `qa_probe_get_blast_radius`| "What pages break if the users endpoint goes down?" |
| `qa_probe_suggest_fix`    | "What should I do about feature_flag_disabled?" |
| `qa_probe_run_analysis`   | "Run a full QA check and give me the summary" |
| `qa_probe_label`          | "Mark this finding as expected / a real bug" |

MCP output is sanitized - SQL errors, stack traces, and table names are redacted
before they reach the assistant. Raw data stays on disk. Results carry their
evidence and confidence, and `explain_failure` returns a plain-English `trust` note
when a finding is unverified, so the assistant does not report guesses as passes.

You can also simply open the `qa-probe` folder (or your project with `.qaprobe/`) in
an AI coding tool and ask it to read `ai-context.md` - a compact, LLM-oriented summary
written on every run.

---

## CLI commands

```
qa-probe run       [--config <path>] [--fail-under <N>]   analyze + probe + report
qa-probe analyze   [--config <path>]                       build the dependency graph
qa-probe probe     [--config <path>]                       probe all endpoints
qa-probe report    [--config <path>] [--fail-under <N>]   score, classify, write reports
qa-probe label <endpoint> <verdict> [-r <reason>]          record feedback (reapplied later)
qa-probe fix       [--config <path>] [--apply] [--pr]      generate remediation diffs
qa-probe mcp       [--config <path>]                       start the MCP server (stdio)
```

`--fail-under <N>` exits non-zero when the overall score is below N (a CI gate).

---

## Capabilities

- **Source-aware mapping** - Babel parses your frontend; detects axios/fetch calls and
  common hook patterns; extracts routes from React Router (JSX and `createBrowserRouter`)
  and TanStack Router.
- **Live probing with real auth** - bearer, cookie, api-key, or none. Concurrency-limited,
  with a hard per-request deadline so a streaming endpoint can never hang the run.
- **Root-cause classification** - around 25 causes, each with a fix hint.
- **ID chaining** - probes collections first, harvests a real id, then probes detail
  routes with it instead of a guessed value (kills unseeded-DB noise). On by default.
- **Adaptive baselines** - learns each endpoint's normal latency and row counts from
  recent runs and flags deviation from itself. No model; fully inspectable.
- **Feedback** - `qa-probe label` (or the MCP tool) records a verdict that is reapplied
  on future runs. Scoped labels auto-revoke if behavior changes, so they cannot hide a
  regression.
- **Security checks (opt-in)** - anonymous auth-bypass re-probe, PII scan, and a
  per-persona privilege-escalation matrix. GET-only; never issues a write.
- **Response assertions** - declare invariants (`in`, `type`, `gte`, `pattern`,
  `present`, `minItems`, ...) and qa-probe verifies them on every 2xx response.
- **Write-flows (opt-in, mutating)** - full create/read/update/delete chains with
  guaranteed cleanup. Off by default; for a disposable / test-tenant environment only.
- **SSE and WebSocket checks**, schema-drift detection, blast radius, regression diff,
  coverage, and HAR import for dynamic frontends.
- **OpenAPI auto-discovery** - tries common spec paths and supports a local spec file,
  so fewer apps fall back to headless mode.

---

## Configuration

The example above is enough to start. Key options (all optional unless noted):

```js
module.exports = {
  baseUrl: 'http://localhost:8000',      // required
  frontendApiPrefix: '/api',              // stripped when matching calls to routes
  framework: 'fastapi',                   // fastapi | express | nextjs | graphql | trpc | generic
  openApiUrl: '/openapi.json',            // auto-discovers fallbacks if missing
  openApiFile: null,                      // or load the spec from a local file
  frontendSrc: './frontend/src',
  routerFile: './frontend/src/App.tsx',

  auth: { type: 'bearer', loginUrl: '/auth/login',
          credentials: { username: process.env.QA_USER, password: process.env.QA_PASS },
          tokenPath: 'access_token' },

  probe: {
    concurrency: 5,
    timeoutMs: 10000,
    idDiscovery: true,                    // ID chaining (on by default)
    safePosts: [],                        // POST endpoints that are safe reads
    pathParamValues: { id: '1' },         // fallback values for {id} etc.
    ignoreHTTPSErrors: false,             // dev/staging only
  },

  // Opt-in extras:
  security: { enabled: true, piiAllow: ['email'] },
  assertions: { 'GET /alerts': [{ field: 'items[].severity', in: ['low','high','critical'] }] },
  writeFlows: { enabled: false, flows: [] },
  analyze: { har: { enabled: false, harFile: './traffic.har' } },

  output: { dir: '.qaprobe', formats: ['json', 'markdown', 'ai-context', 'html'] },
  seedCommand: 'npm run db:seed',         // shown in the empty_db fix hint
};
```

`qa-probe.config.js` is executed as JavaScript (like ESLint or Jest configs). Never
hardcode credentials - use environment variables.

---

## How it works

Three phases, each cached so you can re-run them independently.

1. **Analyze** - parse the frontend AST, extract routes, fetch the OpenAPI spec (or
   import a HAR), and build a dependency graph with blast radius.
2. **Probe** - authenticate once, then probe every discovered endpoint live: HTTP,
   SSE, WebSocket, with hard deadlines and ID chaining.
3. **Report** - classify root causes, apply baselines and feedback, score routes
   0-100, and write JSON / Markdown / ai-context / HTML reports.

---

## vs other tools

| Capability | qa-probe | Schemathesis | Playwright | Postman |
|---|:---:|:---:|:---:|:---:|
| Frontend to backend route mapping | yes | no | no | no |
| Root-cause labels with fix hints | yes | no | no | no |
| Live probe with real auth | yes | yes | yes | yes |
| Property-based contract fuzzing | no | yes | no | no |
| User-journey E2E | no | no | yes | no |
| MCP server for AI assistants | yes | no | no | no |
| CI exit-code gate | yes | yes | yes | yes |

Use Schemathesis for thorough contract fuzzing and Playwright for user journeys.
qa-probe maps and explains; reach for the others to go deeper on what it surfaces.

---

## Security and safe use

- Credentials are held in memory for the run and never written to any report.
- Config is executed as JavaScript - only run qa-probe with a config you trust.
- `ignoreHTTPSErrors: true` disables TLS verification; dev/staging only.
- Security re-probing is GET-only and never writes. Write-flows are off by default
  and clean up after themselves; use them only against a disposable environment.

To report a vulnerability, see [SECURITY.md](https://github.com/LS-SIEM-LLP/qa-probe/blob/HEAD/SECURITY.md).

---

## Known limitations

- Detects visible HTTP calls and common hook patterns. Generated SDK clients, GraphQL
  clients, tRPC routers, and custom service layers may need an adapter - or use HAR
  import to discover them from real traffic.
- Dynamic URLs built by arbitrary factory functions are not detected.
- Best results need an OpenAPI spec; headless mode loses the spec-dependent rules but
  still detects observed-shape drift.

---

## Contributing

See [CONTRIBUTING.md](https://github.com/LS-SIEM-LLP/qa-probe/blob/HEAD/CONTRIBUTING.md). Contributions require a
[Developer Certificate of Origin](https://developercertificate.org/) sign-off
(`git commit -s`).

---

## Branding and trademarks

"qa-probe", "LightShield", and "LS-SIEM" are trademarks of LS-SIEM LLP. The Apache
License does not grant permission to use these names or logos.

---

## License

Copyright (c) 2026 LS-SIEM LLP - created and maintained by the LightShield SIEM team.

Licensed under the Apache License, Version 2.0 - see [LICENSE](https://github.com/LS-SIEM-LLP/qa-probe/blob/HEAD/LICENSE) and
[NOTICE](https://github.com/LS-SIEM-LLP/qa-probe/blob/HEAD/NOTICE). You may use, modify, and redistribute qa-probe under the Apache-2.0
terms; copyright and the trademarks above remain with LS-SIEM LLP.