five46

An autonomous AI testing agent that verifies your app or API actually works while you're still building it — fully local, using your own LLM key.

You just changed something, and you want to know — right now, against the real running thing — whether it actually works, without first writing a test yourself. Give five46 a plain-English goal — "log in and confirm the dashboard loads," "create a user via POST, then confirm it via GET" — and an LLM, using your own OpenAI, Anthropic, Gemini, Groq, or AWS Bedrock key, drives your real app or real API, one real action at a time, and tells you honestly whether it worked, with a root-cause hypothesis if it didn't. Once it does, that exact run is captured as a real, standalone Playwright (or node:test) spec you keep — so the same check that helped you while you were building the feature becomes a permanent regression test afterward, with no five46 or LLM involved in ever running it again.

Status: early proof of concept, verified end-to-end against real live LLM keys across dozens of real-world sites and APIs.

If five46 is useful to you, a ⭐ on GitHub helps other people find it — much appreciated!

Why five46, and how it's different

Most testing tools assume you already have a suite to run. five46 is built for the moment before that — mid-feature, before a test exists at all. Point it at what you're building, describe the outcome you expect in plain English, and keep re-running it as you keep changing code; once it's solid, the run it just did becomes your regression test, not a separate thing you write afterward.

Most AI-driven test-generation tools also run in a cloud sandbox: your app's traffic, screenshots, and DOM leave your machine and go through a third-party service you don't control. five46 is the opposite bet — everything runs on your laptop, using a key you already pay for, and the only thing that ever leaves your machine is the text sent to your chosen LLM provider on each step (always disclosed, never hidden). If your organization can't adopt a cloud-hosted AI testing platform for compliance or trust reasons, this is built for exactly that constraint.

It's also not a black box: every run ends with a real .spec.ts/.test.mjs file you can read, diff, commit to your repo, and run in CI with plain npx playwright test — no vendor lock-in, no proprietary runner.

Features

• Bring your own key (BYOK) — OpenAI, Anthropic, Gemini, Groq, or AWS Bedrock. Your key, your usage, your cost.
• Fully local — no cloud sandbox, no tunneling for local dev servers. Nothing but the LLM calls ever leaves your machine.
• Browser and API testing — drive a real Chromium browser, or drive real HTTP requests directly, from the same agentic engine.
• Real, standalone output — every successful run writes a plain Playwright .spec.ts (or node:test script for API tests) you can re-run any time, with no five46 or LLM involved.
• Session reuse — log in once, capture the session, reuse it across runs without paying the LLM cost of logging in every time.
• Self-healing selectors — a stale selector gets one bounded, disclosed recovery attempt instead of just failing the step.
• Resilient generated specs — when a real, live check confirms Playwright's own getByRole() resolves uniquely to the exact element a step acted on, the generated spec prefers it over a positional CSS selector, since it's far more resistant to future DOM changes. Falls back to the always-correct selector automatically wherever that check can't be made — never changes what the live run itself does.
• Root-cause hypotheses — a failed assertion gets an LLM-generated hypothesis for what likely went wrong and what to check next.
• MCP server — expose five46_test/five46_api as tools an IDE-embedded AI assistant (Claude Code, Cursor, etc.) can call directly.
• Safe by default — API testing is read-only unless you explicitly unlock writes/deletes; destructive-looking browser clicks are blocked by default too.
• Flaky-test detection — --repeat N runs the same goal N times and reports whether the outcome/behavior actually stayed the same.
• Diffing — five46 diff compares two generated run files directly.
• Project management — five46.config.json + --project for reusable, named target defaults (url, session, safety flags).
• Video replay — --record-video records the whole session as a .webm.
• Structured planning — on by default, one extra upfront LLM call plans the whole goal, then most steps execute directly against the real page/response with no further live decision needed; --no-structured-plan opts back into the fully-adaptive, live-decision-every-step loop.
• Fast per-step decisions (--fast-steps, opt-in) — on Groq/Gemini, swaps in a genuinely faster model tier for the high-frequency per-step decision only; the upfront plan always uses your configured model. No effect on OpenAI/Anthropic/Bedrock, already at their fastest reliable tier. Opt-in, not default — see "Fast per-step decisions" below.
• Story mode (--story) — splits a raw, multi-AC user story into independent goals and runs them with bounded concurrency, reporting a clear pass/fail per acceptance criterion. See "Story mode" below.
• Cross-run action cache (--action-cache, opt-in) — skips the upfront planning LLM call on a repeat run of the same goal by reusing a previously-saved plan, falling back to a fresh plan automatically if the page has changed. See "Cross-run action cache" below.

five46 vs. cloud AI testing platforms

Not a knock on cloud platforms — it's a genuinely different tradeoff (their infra vs. your own key and your own machine), and the right choice depends on what your organization is allowed to send off-machine.

Installation

npm install -g five46

npm install --save-dev playwright @playwright/test   # one-time, if your project doesn't already have it
npx playwright install chromium                       # one-time, downloads the browser

Or run it without installing globally:

npx five46 test http://localhost:3000 --goal "log in and confirm the dashboard loads"

git clone https://github.com/sekharsdet/five46.git
cd five46
npm install
npm run build
node dist/cli.js test http://localhost:3000 --goal "..."

Configuration

One-time setup (same shape as gh auth login/aws configure):

five46 config

This prompts for an LLM provider + key, masking secret input, and saves it to ~/.five46/config.json (user-only file permissions). Or set environment variables instead — these always take priority over the saved config, which is useful for CI:

export FIVE46_LLM_PROVIDER=openai   # or: anthropic, gemini, groq, bedrock
export FIVE46_LLM_API_KEY=sk-...    # for bedrock, use your AWS region instead

Getting a key

Don't have a key yet? Pick whichever's easiest to get, or whichever you already use — five46 calls one small, cheap model per provider on every step (never a "flagship" model), so per-run cost is low regardless of which one you pick. If wall-clock speed is what you care about most, pick Groq — its whole differentiator is LPU-based inference hardware built specifically for fast token generation, meaningfully faster round-trips than typical GPU-hosted inference for an equivalent-size model. Since a run's time is dominated by LLM round-trip latency (not five46's own code), the provider you pick is the single biggest lever you control over how fast a run feels: