# nodebench [Health: Active]

**Category:** 🔎 Search & Data Extraction  
**Repository:** https://github.com/HomenShum/nodebench-ai  
**GitHub Stars:** 14  
**npm Downloads (last month):** 13  
**Views:** 0  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/nodebench

## Description
260 MCP tools across 49 domains. AI Flywheel, quality gates, research, web scraping.

## Claude Desktop Quick Installation
Install path detected from listing signals. Uses `npx` (confidence: high):

```json
"mcpServers": {
  "nodebench": {
    "command": "npx",
    "args": ["-y","nodebench-mcp"]
  }
}
```

## Documentation & README

# NodeBench AI

Entity intelligence for any company, market, or question.

**Live:** [nodebenchai.com](https://www.nodebenchai.com)  
**npm:** `npx nodebench-mcp` / `npx nodebench-mcp-power` / `npx nodebench-mcp-admin`  
**GitHub:** [HomenShum/nodebench-ai](https://github.com/HomenShum/nodebench-ai)

![NodeBench AI Home surface: research prompt with Quick answer / Deep research modes, and reusable public research cards below](.qa/evidence/2026-07-15-runtime-grounded-control-focus/baseline/exact-surfaces/home-desktop-light.png)

## Product

NodeBench is a research and reporting product built around five user-facing
surfaces:

- `Home` = start quickly
- `Reports` = reusable memory
- `Chat` = do the work
- `Inbox` = captures, nudges, alerts, automations, and unassigned review
- `Me` = operator context and control

Deep research opens in the separate Workspace surface at `nodebench.workspace`;
it is not a sixth tab in the operating app.

The core idea is simple:

users do not just need a chatbot that answers once.

They need a system that can:

- take a question, file, URL, or prior thread
- search and synthesize with sources
- turn the run into a reusable artifact
- watch for meaningful change later
- improve the next run from what it learned

## Pending Release Candidate: Runtime-Grounded Control Focus

This candidate is not yet merged or deployed. It narrows the reachable product
to controls backed by a real runtime capability and makes unavailable state
explicit:

- Agents keeps ask, explicit swarm launch, active work, approvals, sources,
  traces, receipts, exports, and provenance while removing projected metrics,
  synthetic fallbacks, duplicate actions, and local-only controls.
- Generated code remains copyable and exportable but never executes inside the
  signed-in app origin. Runtime cards accept successful structured tool output,
  not assistant prose, guessed search fields, or incomplete/error states.
- Canonical owner-scoped streaming remains; an unreferenced bearer-stream
  component and HTTP route were removed instead of retained as a second path.
- Home, Reports, Chat, Inbox, and Me use live runtime data, owner-scoped where
  private, or honest loading, empty, not-found, and unmeasured states instead of
  reachable product fixtures. Desktop and mobile now share the same Inbox and
  Me component trees; duplicate queue/profile renders and their hard-coded plan,
  usage, connector, and local-draft projections were deleted.
- Pipeline launches and schedules require an authenticated server identity,
  while guest history, detail, bundle, stream, and evaluation reads require an
  anonymous-session possession credential. Browser-supplied owner keys are no
  longer an authority boundary, and trusted cron or MCP work stays on internal
  contracts.
- Authenticated launches consume durable per-owner admission: four units per
  ten minutes and thirty per day, with composed runs costing two units. Server
  input bounds and per-owner schedule caps close the remaining cost-abuse path.
- Fresh pipeline attempts and recurring schedule occurrences are isolated by
  durable attempt identities and execution-generation fences, so retries can
  resume the same attempt without stale or overlapping workers rewriting it.
- Research distinguishes sources consulted from citations actually bound in
  the synthesis. Missing, malformed, or unbound evidence cannot be promoted to
  a verified claim, and chat no longer injects citation markers into prose.
- Operator telemetry and maintenance are deferred until deliberate disclosure
  and require server-confirmed operator access.
- Task sessions, traces, steps, evidence, approvals, swarms, and operational
  history are owner-scoped. The unreachable parallel timeline, kanban, hook,
  and public orchestrator were removed; due diligence retains only a bounded,
  owner-checked internal task substrate. Unused public due-diligence, investor,
  demo, and evaluation entrypoints were internalized or removed, and retained
  job, branch, memo, and catch-path writes require the exact owner chain. Raw
  orchestrator writes use internal contracts, MCP document and spreadsheet
  operations require exact object ownership, and a TRACE completion fails
  closed unless its receipt persists with an exact output hash.

Release evidence is intentionally pending. The source PR must merge through the
required CI gates first; a follow-up evidence-seal PR must then record the
canonical squash SHA, exact-revision checks, preview assertions, production
deployment revision, and direct production browser verification before this
section can be described as shipped.

## What Shipped

- five-surface web app across `Home`, `Reports`, `Chat`, `Inbox`, and `Me`
- separate deep-work Workspace shell at `nodebench.workspace`
- typed search and reporting pipeline
- hosted public research MCP for external apps and agents:
  `https://nodebench-mcp-unified.onrender.com?profile=public-research`
- Pi-AI pipeline lane on `Reports` with code-gen, design-gen, research,
  composed runs, schedules, streaming previews, eval scorecard, and MCP HTTP
  bridge
- live SSE streaming with saved runtime state
- Convex-backed product state for sessions, reports, entities, nudges, files,
  and related objects
- shared-context handoff and delegation plumbing
- local and deployed server runtime for search, streaming, voice, and shared
  context routes
- `nodebench-mcp`, `nodebench-mcp-power`, and `nodebench-mcp-admin`
  distribution lanes
- builder-facing Oracle, dogfood, eval, replay, and control-plane
  infrastructure

## Hosted Public Research MCP

NodeBench can be used as a public research memory and tool server from any
agent or app without forcing signup before the first useful result.

Use the hosted MCP endpoint:

```txt
https://nodebench-mcp-unified.onrender.com?profile=public-research
```

For Gmail/job-match style integrations, use the smaller profile:

```txt
https://nodebench-mcp-unified.onrender.com?profile=gmail-research
```

Public profiles are anonymous by default, but still metered. Responses include:

```txt
x-nodebench-request-id
x-nodebench-profile
x-nodebench-auth-mode
x-nodebench-account-key
```

Apps should send stable, non-sensitive client headers:

```http
x-nodebench-client: your-app-name
x-nodebench-client-version: 1.0.0
x-nodebench-client-id: stable-install-or-workspace-id
```

`x-nodebench-client-id` lets NodeBench attribute anonymous usage and estimated
costs without requiring a NodeBench token. Do not put private email text,
resume text, API keys, or user secrets in this header.

### Progressive Sign-In And Linking

The intended user flow is:

```text
First public dossier works without signup
  -> show sources, freshness, and confidence
  -> offer "Link NodeBench" after value is visible
  -> linked users get stable history, higher budgets, team usage, webhooks,
     token management, billing controls, and reusable private workspace context
```

Do not block public-source research behind login. Promote sign-in when the user
wants persistence, shared team memory, budget controls, private workspace
linking, or API/MCP tokens.

See [MCP_TOOL_PROFILES.md](./docs/guides/MCP_TOOL_PROFILES.md) for the full
profile list, tool catalog, account attribution, and cost tracking contract.

## Product At A Glance

```text
USER SURFACES
-------------
Home      -> start quickly
Reports   -> reusable report memory
Chat      -> answer, sources, trace, follow-ups
Inbox     -> captures, nudges, automations, alerts, unassigned items
Me        -> operator context, permissions, controls

BACKEND
-------
Convex tables and product state for sessions, reports, entities, nudges,
files, shared context, and evaluation artifacts

RUNTIME
-------
search pipeline
  -> answer packet
  -> saved report
  -> tracked entity / tracked theme / follow-up task
  -> nudge or prep brief
  -> resumed chat or reopened report

COMPOUNDING LOOP
----------------
question
  -> answer
  -> saved report
  -> watch item
  -> useful nudge
  -> better next run

DISTRIBUTION
------------
nodebenchai.com
nodebench.workspace
nodebench-mcp
nodebench-mcp-power
nodebench-mcp-admin
```

## Event Intelligence Serving Model

Event serving extends the same budgeted search route used by the main app and
MCP runtime. NodeBench treats search as a memory-building operation with a
budget. For events, NodeBench checks the event corpus and workspace memory
before live search, then persists useful results as entities, claims, sources,
and workspace context.

ScratchNode is the lightweight live-event sidecar for this model: a disposable
room that turns public chat and sourced `/ask` answers into a public wiki while
keeping attendee notes private. It complements Luma, Slack, Eventbrite, and
other event surfaces instead of replacing them; the explicit handoff to
NodeBench opens `https://nodebenchai.com/events/:eventSlug/private` with
private-note continuation context and no URL-borne `ownerKey`.

The event flow is:

```text
Before event
  -> build event corpus

During event
  -> capture messy notes instantly

After event
  -> turn captures into report, cards, follow-ups, and reusable memory
```

The product model is:

```text
ScratchNode sidecar room + public event corpus + private NodeBench continuation
```

Event corpus and capture data stay separated:

- `Shared event corpus` = public event info, speakers, sponsors, company pages,
  sessions, and public source cache.
- `Private captures` = what a user personally heard, wrote, recorded, or
  photographed; ScratchNode private notes never enter the public feed, public
  wiki, or public `/ask` cache.
- `Team/org memory` = shared only inside the fund, company, or workspace.
- `Event aggregate insights` = opt-in or anonymized only.

During the event, most captures should hit the event corpus first and avoid paid
search:

```text
voice memo / text / screenshot
  -> captureRouter
  -> active event corpus
  -> entity and claim extraction
  -> active event session attachment
  -> budget policy
  -> ack + next action
```

Example mobile ack:

```text
Saved to Ship Demo Day session
Detected 1 person | 1 company | 2 claims | 1 follow-up
Using event corpus | 0 paid calls
```

After the event, the report opens in `nodebench.workspace`:

```text
Brief      -> post-event memo
Cards      -> people, companies, products, themes
Notebook   -> raw notes, transcripts, screenshot OCR, cleaned notes
Sources    -> field notes, public evidence, verification status
Chat       -> follow-up questions and deeper refreshes
Map        -> graph view later
```

The canonical spec lives in
[EVENT_INTELLIGENCE_SERVING_MODEL.md](./docs/architecture/EVENT_INTELLIGENCE_SERVING_MODEL.md).
The ScratchNode/NodeBench privacy boundary lives in
[SCRATCHNODE_NODEBENCH_BOUNDARY.md](./docs/architecture/SCRATCHNODE_NODEBENCH_BOUNDARY.md).

## Why This Design

NodeBench is designed around a few product realities:

1. A useful answer should not disappear after one chat turn.
2. Saved work should become reusable memory, not a dead archive row.
3. The product should bring the user back only when something meaningful
   changes.
4. The system should gradually learn how the user works without forcing a heavy
   onboarding flow.
5. Operator context should improve future runs without turning the system into
   corporate-speak or fake-agreeable sludge.

That drives the current design:

- answer-first execution
- advisor mode by design via dynamic routing:
  - fast executive lane for routine work
  - deeper advisor lane for ambiguity, planning, and harder reasoning
  - similar in spirit to Claude Code's official `opusplan` split:
    stronger planning lane, cheaper execution lane
- saved artifacts as first-class objects
- visible sources and traceability
- a five-page loop instead of five unrelated tabs
- future `Harness v2` work focused on specification, operator context, and
  compounding behavior

Plain English:

```text
NodeBench should not spend the most expensive reasoning path on every request.
It should move fast by default, then go deeper when the task, evidence, or user
request justifies it.
```

The detailed implementation, verification, and evaluation plan for this mode
lives in:

- [HARNESS_V2_PROPOSAL.md](./docs/archive/2026-q1/architecture-superseded/HARNESS_V2_PROPOSAL.md) (archived)
- [HARNESS_V2_BUILD_PLAN.md](./docs/archive/2026-q1/architecture-superseded/HARNESS_V2_BUILD_PLAN.md) (archived)

## How The Five Pages Compound

NodeBench should not feel like five separate destinations.

The intended product behavior is:

```text
Home
  -> start quickly

Reports
  -> turn that artifact into reusable memory

Chat
  -> do the work
  -> create the first useful artifact

Inbox
  -> triage captures, nudges, automations, alerts, and unassigned items

Me
  -> improve how the next run is handled

Workspace
  -> open deep Brief / Cards / Notebook / Sources / Chat / Map work
  -> lives at nodebench.workspace, not in the operating tab bar

Next Home or Chat run
  -> starts with more context than before
```

The shortest version of the compounding loop is:

```text
question
  -> answer
  -> saved report
  -> watch item
  -> useful nudge
  -> better next run
```

Plain-English artifact flow:

```text
input
  -> answer packet
  -> saved report
  -> tracked entity / tracked theme / follow-up task
  -> nudge or prep brief
  -> resumed report or resumed chat
  -> user correction or confirmation
  -> updated operator context
  -> better next run
```

What each page contributes:

- `Home` starts the run with the least friction possible
- `Reports` turns those into a durable report the user can reopen, refresh, and
  reuse
- `Chat` creates the answer, sources, trace, entities, and next actions
- `Inbox` collects nudges, captures, automations, alerts, and unassigned items
- `Me` stores the operator context that improves the next answer
- `Workspace` owns recursive cards, notebook editing, source verification, and
  long-lived intelligence memory

## Current Legacy Infrastructure

NodeBench is not starting from zero. The repo already contains a substantial
legacy stack that works today.

Current legacy foundation:

- five-surface web product
- Convex-backed canonical data layer
- local and deployed server runtime
- harness v1 planning and execution path
- shared-context handoff and delegation support
- MCP distribution lanes
- builder-facing evaluation and control-plane systems

What that means:

- the problem is not missing architecture
- the problem is product behavior, workflow compression, and clearer
  cross-surface compounding

## Roadmap

The near-term goal is:

```text
keep the working legacy foundation
remove accidental complexity
add specification-aware operator context
ship one clear compounding workflow
```

Main tasks still to finish:

- [ ] make `Home -> Reports -> Chat -> Inbox -> Me` behave like one continuous
      workflow instead of five adjacent surfaces
- [ ] turn harness v1 into the clearer v2 shape described in
      [HARNESS_V2_PROPOSAL](docs/archive/2026-q1/architecture-superseded/HARNESS_V2_PROPOSAL.md)
- [ ] ship `Layer 0` operator context so the system can learn useful workflow
      patterns without forcing a heavy onboarding flow
- [ ] support permissioned transcript ingestion from NodeBench chats first, then
      optional external logs such as Claude Code JSONL transcripts for
      `nodebench-mcp`
- [ ] add style-drift guardrails so the system learns judgment and workflow
      without overfitting to corporate voice, filler, or sycophancy
- [ ] add anticipatory prep behavior so the system can prepare the user before
      important interactions, not only answer after the fact
- [ ] make saved reports behave like reusable memory, not storage
- [ ] keep `Nudges` as an Inbox section with at least one working daily trigger
- [ ] make `Me` clearly improve future runs by exposing what context is being
      used and why
- [ ] finish the `nodebench-mcp` v3 cut-and-split plan so default runtime,
      power runtime, and admin runtime are clearly separated
- [ ] instrument real latency, real cost, real artifact completion, and real
      reuse across both web and MCP flows
- [ ] keep README, runtime behavior, and exposed tool counts in sync so the
      public story matches the actual system
- [ ] keep dogfood, eval, and builder-control infrastructure as internal
      leverage instead of letting it leak into the main user-facing product

## Quick Start

### Web app

Open [nodebenchai.com](https://www.nodebenchai.com) and start in `Home`.

### MCP

```bash
# Claude Code
claude mcp add nodebench -- npx -y nodebench-mcp

# Claude Code power lane
claude mcp add nodebench-power -- npx -y nodebench-mcp-power

# Claude Code admin lane
claude mcp add nodebench-admin -- npx -y nodebench-mcp-admin

# Cursor
npx nodebench-mcp --preset cursor

# Generic MCP client
npx nodebench-mcp
```

### Local development

```bash
git clone https://github.com/HomenShum/nodebench-ai.git
cd nodebench-ai
npm install
cp .env.example .env.local

# Frontend + Convex + voice server
npm run dev

# Production build
npm run build
```

## Architecture

```text
nodebenchai.com (React + Vite + Tailwind)
    |
Convex Cloud (sessions, reports, entities, nudges, files, product state)
    |
server runtime + search pipeline + SSE
    |
answer packet
    |
saved report
    |
tracked entities / watch conditions / nudges
    |
future runs with better operator context
```

## Student Learning Lessons

The notebook and diligence stack in this repo are a good example of a common
product engineering tradeoff:

- the best user experience is one notebook that feels continuous
- the safest current runtime is still layered and block-addressable underneath

For NodeBench, that means:

- `founder` is a trait and diligence block, not a permanent sixth tab
- diligence should use one generic pipeline, not many narrow `*Identify.ts`
  features
- the runtime should stay `scratchpad-first -> structuring pass ->
  deterministic merge`
- user-owned prose should feel local-first and calm while typing
- live agent output should arrive as overlays or decorations first, not as
  direct document mutations
- accepted agent output should become frozen, user-owned notebook content
- provenance should stay available, but secondary to the reading and writing
  flow

Why the notebook does **not** use one giant live editor model yet:

- collaboration is more reliable when the system can address bounded sections
- provenance, evidence, and contribution logs need stable attachment points
- background agent updates should not compete with user keystrokes
- deterministic section-level merge is easier to reason about than whole-page
  mutation churn

The practical rule in this repo is:

```text
UX should feel monolithic.
Runtime should stay layered.
Typing should be local-first.
Agent output should be overlay-first.
Accepted output should become owned prose.
```

Current notebook refactor lessons:

- hide the block machinery from the reading path
- keep chrome quiet and move metadata to hover or focus
- isolate the notebook surface from page-level re-render churn
- favor one memoized notebook boundary over many inline object props
- treat live diligence as read-only reference overlay until the user accepts it
- when accepted, materialize a frozen notebook snapshot with explicit provenance
- anchor live overlays at the notebook surface, not inside the first editable row
- let Convex projection rows carry real source metadata so the UI is not forced to reconstruct trust state from prose alone
- use one generic projection producer for overlays: report save writes the same structured rows that page-load backfill and manual refresh re-run
- when moving beyond report-backed overlays, stream raw scratchpad only in a secondary rail and emit structured projection rows on checkpoint rather than dumping scratchpad prose into the notebook body
- if checkpoint structure comes from an LLM, keep it block-scoped and schema-bound: `scratchpad checkpoint -> JSON -> validation/repair -> deterministic fallback -> projection row`
- let the model structure intermediate JSON, but keep merge, persistence, and notebook ownership deterministic
- ship generic diligence primitives first, then block-specific renderers

For students reading the code, the most relevant docs are:

- [`AGENT_PIPELINE`](docs/architecture/AGENT_PIPELINE.md)
- [`DILIGENCE_BLOCKS`](docs/architecture/DILIGENCE_BLOCKS.md)
- [`SCRATCHPAD_PATTERN`](docs/architecture/SCRATCHPAD_PATTERN.md)
- [`PROSEMIRROR_DECORATIONS`](docs/architecture/PROSEMIRROR_DECORATIONS.md)
- [`SESSION_ARTIFACTS`](docs/architecture/SESSION_ARTIFACTS.md)

The live notebook refactor is deliberately incremental:

- current shipped slices make the notebook feel more continuous and reduce
  per-keystroke render churn
- current shipped slices also move live diligence into notebook-surface
  overlays instead of seeded block-like records and freeze accepted snapshots
- the end state is one notebook experience with layered internals, not a raw
  block UI and not a brittle giant document runtime

### Key tech

- Frontend: React, Vite, TypeScript, Tailwind CSS
- Backend: Convex
- Search: Linkup + Gemini extraction + grounding pipeline
- MCP server: Node.js + TypeScript
- Realtime runtime: SSE + Convex-backed persistence

## API Keys

Set these in `.env.local` for local work or in Convex / Vercel for deployed
environments.

| Key | Required | Purpose |
| --- | --- | --- |
| `GEMINI_API_KEY` | Yes | classification, extraction, synthesis |
| `LINKUP_API_KEY` | Recommended | web search and sourced answers |
| `VITE_CONVEX_URL` | Yes | Convex deployment URL |

## Codebase map

Top-3 levels, annotated. See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the
pipeline diagram and [`docs/architecture/README.md`](docs/architecture/README.md)
for the 13 canonical architecture docs.

```text
nodebench-ai/
├── README.md                   ← you are here
├── CONTRIBUTING.md             ← contribution bar
├── CLAUDE.md                   ← Claude Code conventions for this repo
├── AGENTS.md                   ← agent methodology + eval bench
├── LICENSE                     ← MIT
│
├── apps/
│   └── web/src/                ← React frontend (Vite)
│       ├── features/           ← feature-first (Home · Reports · Chat · Inbox · Me · Workspace · entities · agents · …)
│       │   └── <feature>/      ← views · components · hooks · lib · __tests__ (colocated)
│       ├── shared/             ← shared UI primitives, hooks, utils
│       ├── lib/                ← registry, analytics, error reporting
│       └── layouts/            ← shell + cockpit + public
│
├── workers/node/               ← Node runtime (Express + MCP gateway)
│   ├── pipeline/               ← agent harness runtime + diligence blocks
│   ├── routes/                 ← HTTP routes (search, harness, founder episodes)
│   ├── mcpGateway.ts           ← WebSocket MCP gateway
│   └── services/               ← shared services
│
├── backend/convex/             ← Convex backend
│   ├── domains/                ← domain folders (agents · product · research · founder · search · …)
│   ├── schema.ts               ← database schema (includes agentScratchpads)
│   └── crons/                  ← scheduled jobs
│
├── packages/
│   ├── mcp-local/              ← the published nodebench-mcp npm package (MIT)
│   ├── mcp-client/             ← typed client SDK
│   └── convex-mcp-nodebench/   ← Convex-side MCP auditor
│
├── .claude/
│   ├── README.md               ← map of the .claude/ layout
│   ├── rules/                  ← 31 modular rules with related_ cross-refs
│   ├── skills/                 ← reusable how-to procedures
│   ├── agents/                 ← subagent configs
│   └── commands/               ← custom slash commands
│
├── docs/
│   ├── README.md               ← docs tree map
│   ├── ARCHITECTURE.md         ← top-level pipeline diagram
│   ├── ONBOARDING.md           ← 30-minute new-contributor path
│   ├── architecture/           ← 13 canonical specs + plans/ + README index
│   ├── agents/                 ← agent docs + bootstrap configs
│   ├── guides/                 ← how-to for builders
│   ├── decisions/              ← ADRs
│   ├── changelog/              ← release notes
│   ├── product/                ← product decisions
│   ├── qa/                     ← QA protocols
│   └── archive/                ← superseded content, provenance-only
│
├── evals/
│   └── e2e/                    ← end-to-end eval suites
│
├── scripts/                    ← dogfood, eval harness, one-offs
└── public/                     ← static assets served by Vite + Vercel
```

## Related Docs

**Start here:** [`docs/ONBOARDING.md`](docs/ONBOARDING.md) · [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) · [`docs/architecture/README.md`](docs/architecture/README.md)

The 13 canonical architecture docs are organized in 4 tiers. See [`docs/architecture/README.md`](docs/architecture/README.md) for the indexed map:

- **Tier 1 (core pipeline):** [`AGENT_PIPELINE`](docs/architecture/AGENT_PIPELINE.md) · [`DILIGENCE_BLOCKS`](docs/architecture/DILIGENCE_BLOCKS.md) · [`USER_FEEDBACK_SECURITY`](docs/architecture/USER_FEEDBACK_SECURITY.md)
- **Tier 2 (sub-patterns):** [`SCRATCHPAD_PATTERN`](docs/architecture/SCRATCHPAD_PATTERN.md) · [`PROSEMIRROR_DECORATIONS`](docs/architecture/PROSEMIRROR_DECORATIONS.md) · [`AGENT_OBSERVABILITY`](docs/architecture/AGENT_OBSERVABILITY.md) · [`SESSION_ARTIFACTS`](docs/architecture/SESSION_ARTIFACTS.md)
- **Tier 3 (features):** [`FOUNDER_FEATURE`](docs/architecture/FOUNDER_FEATURE.md) · [`REPORTS_AND_ENTITIES`](docs/architecture/REPORTS_AND_ENTITIES.md) · [`AUTH_AND_SHARING`](docs/architecture/AUTH_AND_SHARING.md)
- **Tier 4 (cross-cutting):** [`MCP_INTEGRATION`](docs/architecture/MCP_INTEGRATION.md) · [`EVAL_AND_FLYWHEEL`](docs/architecture/EVAL_AND_FLYWHEEL.md) · [`DESIGN_SYSTEM`](docs/architecture/DESIGN_SYSTEM.md)

Active architecture addenda:
[`GRAPH_SEARCH_AGENT_CONTEXT`](docs/architecture/GRAPH_SEARCH_AGENT_CONTEXT.md)
captures the graph/search/agent-context strategy, exact product questions,
scale projection, node attention model, and human-vs-agent retrieval split.

Historical specs are preserved in [`docs/archive/2026-q1/`](docs/archive/2026-q1/INDEX.md).

## Production Readiness & Evaluation

NodeBench ships with a comprehensive evaluation harness that proves correctness
across 32+ scenarios, 9 user personas, and 9 feature categories. This is not
hand-wavy "it works" — it is measured, versioned, and reproducible.

### Latest Published Run Results

**Pi-AI pipeline cascade:** merged to `main` on 2026-04-30 at
`2a541037874c0f8c675ab393d5c08f50123cf6d2`.

| Lane | Result |
|------|--------|
| PR chain | #211 -> #212 -> #213 -> #214 -> #215 -> #216 all merged |
| Production surface | `https://www.nodebenchai.com/?surface=packets` |
| MCP bridge | `https://agile-caribou-964.convex.site/mcp/pipeline/*` behind `MCP_SECRET` |
| Code-gen run | `pipeline_mokobe4y_6n23be` succeeded, `verified`, 6 files, 32.9s, about `$0.001` |
| Research streaming | `pipeline_mokpvi1b_yoj8ot` completed with 4,317 streamed characters |
| Linkup research | `pipeline_mol2wj2j_2lgx2u` succeeded with 18 snippets across 5 sub-questions |
| Composed pipeline | `research_then_code` completed stage 1 research and stage 2 code-gen |
| Schedule workflow | once schedule swept by cron/manual sweep and auto-disabled after run |
| Design output | design-gen produced a PNG stored in Convex storage |
| UI launcher | DOM-submitted composed run updated the reactive run list |
| Pipeline scorecard | 41.7% verified, Brier 0.135 across 12 runs |

The implementation is mounted on the Reports surface:

- `PipelineLauncher`
- `PipelineSchedulesPanel`
- `PipelineEvalScorecard`
- `PipelineRunsPanel`
- `EntityFindingsPanel`

The detailed handoff is in
[`docs/handoff/PI_AI_PIPELINES_HANDOFF.md`](docs/handoff/PI_AI_PIPELINES_HANDOFF.md).

**Workflow-loop eval bank:** added on 2026-04-30 to test the full product loop,
not just answer text.

```text
query / capture
  -> memory search
  -> entity resolution
  -> report update
  -> notebook update
  -> graph edges
  -> sources / claims
  -> follow-up / export
```

| Eval bank | Result |
|-----------|--------|
| Total workflow cases | 124 |
| Minimum P0 suite | 30 cases |
| Coverage categories | 11 |
| Score dimensions | 12 |
| Validator | `apps/web/src/features/evaluation/data/nodebenchWorkflowEvalBank.test.ts` |
| Latest local check | `npx vitest run apps/web/src/features/evaluation/data/nodebenchWorkflowEvalBank.test.ts` -> 4/4 passed |

The eval bank lives in
[`apps/web/src/features/evaluation/data/nodebenchWorkflowEvalBank.ts`](apps/web/src/features/evaluation/data/nodebenchWorkflowEvalBank.ts).

### Two-Layer Judge Architecture

Every production run is evaluated by two independent systems:

**Layer 1: Deterministic Boolean Gates** (`server/pipeline/diligenceJudge.ts`)
- 10 strict pass/fail checks: tier validity, latency budget, token tracking,
  source capture, terminal status
- Verdicts: `verified` | `provisionally_verified` | `needs_review` | `failed`
- Zero LLM involvement — pure deterministic validation

**Layer 2: LLM Semantic Scoring** (`server/pipeline/diligenceLlmJudge.ts`)
- 5 dimensions scored [0,1]: prose quality, citation coherence, source
  credibility, tier appropriateness, overall semantic fit
- Prompt version tracking (`llmjudge-v1`) for cohort separation
- Bounded: 30s timeout, 512KB response cap, honest error reporting

This dual-layer approach means hallucinations and quality regressions are
caught by **two independent systems** before they reach users.

### Current Production Status

**Latest Full-Stack Eval:** `2026-04-23T06:46:53Z`

```text
Overall Pass Rate:     100% ✅
LLM Judge Average:     9.6/10 (target: ≥7 for production)
Dogfood Score:         100/100 (0 real issues)
Entity Resolution:     100% ✅
Factual Accuracy:      90.6% ✅
No Hallucinations:     90.6% ✅
Actionable Output:     100% ✅
Answer Control:        100% ✅ (all 8 dimensions)
Feature Breadth:       100% ✅ (31 scenarios)
Retention/Continuity:  4/4 passed ✅
```

All production gates **passing**:
- ✅ Expanded Feature Coverage Production Gate
- ✅ Answer Control Production Gate  
- ✅ Dogfood Production Gate
- ✅ Notebook Capacity Production Gate
- ✅ History Soak Production Gate

**Note:** The only outstanding item is p95 latency optimization (174s vs 90s
target) — a performance enhancement, not a correctness blocker. The system is
**production-ready for all quality scenarios**.

### Evaluation Coverage

**Capability Eval — 32 Persona Scenarios**

| Persona | Example Query | Status |
|---------|---------------|--------|
| JPM Startup Banker | "DISCO — worth reaching out? Fastest debrief" | ✅ 100% |
| Early Stage VC | "OpenAutoGLM — what's the wedge?" | ✅ 100% |
| CTO Tech Lead | "QuickJS — do I have exposure?" | ✅ 100% |
| Enterprise Exec | "Gemini 3 — procurement next step?" | ✅ 100% |
| Ecosystem Partner | "SoundCloud VPN — who benefits?" | ✅ 100% |
| Founder Strategy | "Salesforce Agentforce — counter-positioning?" | ✅ 100% |
| Academic R&D | "RyR2/Alzheimer's — literature anchor?" | ✅ 100% |
| Quant Analyst | "DISCO — extract funding signal" | ✅ 100% |
| Product Designer | "DISCO — schema-dense UI card JSON" | ✅ 100% |
| Sales Engineer | "DISCO — share-ready outbound summary" | ✅ 100% |

**Expanded Feature Breadth — 31 Scenarios**

| Category | Count | Pass Rate |
|----------|-------|-----------|
| Calendar | 3 | 100% ✅ |
| Disclosure | 4 | 100% ✅ |
| Document | 3 | 100% ✅ |
| Hybrid | 4 | 100% ✅ |
| Media | 3 | 100% ✅ |
| Skills | 4 | 100% ✅ |
| Spreadsheet | 3 | 100% ✅ |
| Tools | 4 | 100% ✅ |
| Web | 3 | 100% ✅ |

**Answer Control — 8 Dimensions**

- Entity resolution: 100% ✅
- Retrieval relevance: 100% ✅
- Claim support: 100% ✅
- Final response quality: 100% ✅
- Trajectory quality: 100% ✅
- Actionability: 100% ✅
- Artifact decision quality: 100% ✅
- Ambiguity recovery: 100% ✅

### How to Verify

Run the full production evaluation suite:

```bash
# Full 8-phase evaluation (typecheck → build → capability → expanded →
# answer-control → dogfood → notebook → history)
npm run eval

# Quick verification (3 scenarios)
npm run eval:quick-slice

# Individual lanes
npm run eval:capability      # 32 persona scenarios
npm run eval:feature-breadth # 31 feature scenarios  
npm run eval:retention       # Wiki continuity suite
```

All artifacts are versioned in `docs/architecture/benchmarks/`:
- `full-stack-eval-latest.md` — aggregate summary
- `comprehensive-eval-*.md` — capability results
- `expanded-eval-*.md` — feature breadth results
- `product-answer-control-eval-*.md` — answer control results

### What "Production Ready" Means Here

1. **Deterministic gates pass** — no regressions in core correctness
2. **LLM judge scores ≥7** — semantic quality validated by independent LLM
3. **Dogfood score ≥85** — internal usage shows no real issues
4. **All 32 persona scenarios pass** — diverse user types handled correctly
5. **All 31 feature scenarios pass** — broad surface area covered
6. **Retention/continuity passes** — long-term memory works
7. **Answer control 100%** — artifact decisions, ambiguity recovery solid

The system meets all of these. The only remaining work is latency optimization
— making fast answers even faster, not making broken answers work.

### Model Strategy

- **Primary:** `moonshotai/kimi-k2.6` (OpenRouter) — 100% capability pass
- **Fallback:** `gpt-5.4` — automatic retry on empty/missing debrief
- **Judge:** `kimi-k2.6` — 9.6/10 average across all scenarios

Kimi is the primary lane. GPT-5.4 remains the safety fallback until Kimi's
first-attempt stability improves, but both paths are production-tested.

## Product Suite

```text
NodeBench AI   = flagship user surface
nodebench-mcp  = workflow lane
Attrition.sh   = measured replay + optimization lane
```

Attrition is not a third flagship. It is the measurable optimization lane for
the same NodeBench workflow.

## License

MIT