SpecterQA

AI personas walk your app so real users don't trip.

SpecterQA sends AI personas through your application — they look at the screen, decide what to do, and interact like real humans. No test scripts. No selectors. You describe personas and journeys in YAML, and SpecterQA handles the rest.

$ specterqa run -p myapp

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ SpecterQA Run                                                    ┃
┃ Product: myapp   Budget: $5.00   Viewport: 1280x720            ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

  ✓ Step 1/4: Navigate to homepage       PASS   3.2s   $0.0081
  ✓ Step 2/4: Click signup link          PASS   2.1s   $0.0043
  ✓ Step 3/4: Fill registration form     PASS   8.7s   $0.0312
  ✓ Step 4/4: Verify dashboard loads     PASS   4.5s   $0.0127

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ ALL TESTS PASSED                                                ┃
┃ Steps: 4/4   Findings: 0   Duration: 18.5s   Cost: $0.0563     ┃
┃ Run ID: GQA-RUN-20260222-143052-a1b2                            ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

What is this?

Traditional E2E tests are brittle. You write selectors, they break. You maintain scripts, they rot. SpecterQA takes a different approach: AI vision models look at your actual UI and navigate it the way a person would.

You define personas (who is using your app) and journeys (what they're trying to do). SpecterQA's engine takes a screenshot, sends it to a Claude vision model, gets back a decision ("click this button", "fill this field"), executes it via Playwright, takes another screenshot, and repeats until the goal is achieved or something goes wrong.

When something goes wrong, you get evidence: screenshots, UX observations, cost breakdowns, and findings categorized by severity.

Installation

SpecterQA is distributed via PyPI and requires Python 3.10 or later.

pip install specterqa

After installing, download the Playwright browser binaries:

specterqa install

For macOS native app testing and iOS Simulator support, install the optional native extra:

pip install specterqa[native]

For MCP server support (integrating SpecterQA as a tool in Claude Desktop, Cursor, or other MCP clients):

pip install specterqa[mcp]

You will also need an Anthropic API key to run tests:

export ANTHROPIC_API_KEY=sk-ant-...

To verify the installation:

specterqa --version
specterqa init       # scaffold a sample project
specterqa run -p demo

Quick Start

pip install specterqa
specterqa install          # downloads Playwright browsers
specterqa init             # scaffolds .specterqa/ with sample configs
specterqa run -p demo      # runs the sample journey

You'll need an Anthropic API key:

export ANTHROPIC_API_KEY=sk-ant-...

That's it. Three commands and an API key.

How It Works

The core loop is simple:

screenshot --> vision model --> action decision --> execute --> repeat

1. Screenshot -- Playwright captures the current page state as a PNG
2. Decide -- A Claude vision model receives the screenshot + persona context + goal, returns a structured JSON action (click, fill, navigate, scroll, keyboard, wait, done, or stuck)
3. Execute -- Playwright performs the action (click at coordinates, type text, navigate to URL, etc.)
4. Repeat -- Loop until the goal is achieved, the agent gets stuck, or the budget runs out

The persona's profile shapes how the AI behaves. A "tech-savvy developer" explores differently than a "frustrated first-time user." Persona patience, tech comfort, and frustrations all influence the system prompt.

Model routing keeps costs down. Simple actions (click, scroll) use Haiku. Complex actions (form filling, initial assessment) use Sonnet. You can also route simple actions to a local Ollama model (llava:13b) for zero API cost on straightforward navigation.

Features

• Persona-based testing -- Define AI users with backgrounds, goals, frustrations, and tech comfort levels. They don't just follow scripts; they react to what they see.
• Vision-powered -- No selectors, no DOM queries. The AI interprets screenshots like a human would. Catches visual/layout issues that selector-based tests miss entirely.
• YAML-configured -- Products, personas, and journeys are all YAML files. PMs can read them. No code to maintain.
• Budget enforcement -- Per-run, per-day, and per-month cost caps. The engine hard-stops if you hit the limit. No surprise bills.
• JUnit XML output -- Drop --junit-xml results.xml and plug it into any CI system.
• Tiered model routing -- Haiku for cheap navigation, Sonnet for complex reasoning, optional local Ollama for zero-cost simple actions.
• Multi-platform -- Web apps (via Playwright), macOS native apps (via Accessibility API + pyobjc), iOS Simulator (via simctl). Same YAML format, different runners.
• Evidence collection -- Every run produces screenshots, a findings report, cost breakdown, and a structured JSON result. Everything is saved to an evidence directory.
• Stuck detection -- If the AI repeats the same action or the UI stops changing, the engine escalates to a stronger model, then aborts if nothing works. No infinite loops.
• Template variables -- Use {{persona.credentials.email}} in your journey steps. Variables resolve from persona configs at runtime.
• Precondition checks -- Verify services are up before running tests. Fail fast with clear errors instead of wasting API calls.

Configuration

SpecterQA uses three types of YAML config files, all living in .specterqa/:

Product (products/myapp.yaml)

product:
  name: myapp
  display_name: "My Application"
  base_url: "http://localhost:3000"

  services:
    frontend:
      url: "http://localhost:3000"
      health_endpoint: /

  viewports:
    desktop:
      width: 1280
      height: 720
    mobile:
      width: 375
      height: 812

  cost_limits:
    per_run_usd: 5.00

Persona (personas/alex-developer.yaml)

persona:
  name: alex_developer
  display_name: "Alex Chen"
  role: "Full-Stack Developer"
  age: 28
  tech_comfort: high
  patience: medium
  preferred_device: desktop

  goals:
    - "Evaluate the app from a developer's perspective"
    - "Check for common UX anti-patterns"

  frustrations:
    - "Unclear error messages"
    - "Missing loading indicators"

  credentials:
    email: "alex@example.com"
    password: "TestPass123!"

Journey (journeys/onboarding.yaml)

scenario:
  id: onboarding-happy-path
  name: "Onboarding Happy Path"
  description: "New user signs up, completes onboarding, reaches dashboard."
  tags: [onboarding, critical_path, smoke]

  personas:
    - ref: alex_developer
      role: primary

  preconditions:
    - service: frontend
      check: /
      expected_status: 200

  steps:
    - id: visit_homepage
      mode: browser
      goal: "Navigate to the homepage and verify it loads"
      checkpoints:
        - type: text_present
          value: "Welcome"

    - id: navigate_signup
      mode: browser
      goal: "Find and click the signup link"

    - id: fill_signup_form
      mode: browser
      goal: "Complete the signup form with test credentials"

    - id: verify_dashboard
      mode: browser
      goal: "Verify signup succeeded and the dashboard loads"

See docs/configuration.md for the full reference.

YAML schema support

SpecterQA includes a JSON Schema for product YAML files at schemas/product.schema.json.

# yaml-language-server: $schema=../../schemas/product.schema.json

CI Integration

SpecterQA is built for CI. It runs headless by default and returns proper exit codes.

# Basic CI run
specterqa run -p myapp --junit-xml results.xml

# Smoke test (runs first scenario only, fast)
specterqa run -p myapp --level smoke --budget 2.00

# JSON output for programmatic consumption
specterqa run -p myapp --output json > results.json

Exit codes:

• 0 -- all tests passed
• 1 -- one or more tests failed
• 2 -- configuration error
• 3 -- infrastructure error (missing dependencies, API unreachable)

See docs/ci-integration.md for GitHub Actions, GitLab CI, and CircleCI examples.

Cost

SpecterQA uses Anthropic's Claude API. Every run costs money. Here's what to expect:

Typical costs per run:

• 3-step smoke test: ~$0.30-0.60
• 5-step standard journey: ~$0.50-1.50
• Complex 10-step journey with forms: ~$1.00-3.00

The default budget is $5.00 per run. The engine hard-stops if the budget is exceeded -- no silent overruns. You can set per-day and per-month caps too.