DiffCI

Website · Test impact analysis guide · Open evidence study

Find test-selection opportunities in your CI before changing what it runs. DiffCI analyzes a commit's changes and dependency graph, then reports which test files it would select, why it falls back to a full run, and whether it can propose a test command. check also runs paired full and selected commands when it can infer them. The observe command and Action remain observation-only.

Try DiffCI

From a Git repository checkout, with Node.js 22.5+ and Git installed, run:

npx "@diffci.com/diffci@latest" check

check explains affected tests and, when it can infer safe commands, runs both the full and selected test commands to measure the difference. It sends nothing to DiffCI. Test commands can create files in the checkout. For analysis without test execution, use npx "@diffci.com/diffci@latest" observe --no-send.

To add instructions for coding agents, run:

npx "@diffci.com/diffci@latest" init

To also pin DiffCI as a development dependency and update the detected npm, pnpm, Yarn, or Bun lockfile, pass --install. Add --workflow for a separate non-blocking observation job:

npx "@diffci.com/diffci@latest" init --install --workflow

--install also adds diffci:check and diffci:observe package scripts. It preserves same-named scripts that the project already owns. After installation, CI or contributors can run npm run diffci:check or npm run diffci:observe without knowing the package name or version.

On Windows PowerShell, quote the package name:

npx "@diffci.com/diffci@latest" check

The copyable adoption kit includes an AGENTS.md instruction and maintainer PR text. AI-readable documentation is on Context7 CLI and Context7 Core. Use the GitHub Marketplace Action for a separate, non-blocking observation job. Required project CI remains authoritative.

One paired run is preliminary evidence. verify-savings can repeat comparisons, alternate arm order, record declared cache state, and run an explicit cache-preparation command before every arm. A result is labelled controlled only after at least three alternating, cache-prepared repetitions. On a full-validation fallback, check runs the full command once and reports 0% reduction.

Upgrade from 0.1.3: tests excluded by a source-only tsconfig.json could be discovered without their dependency edges, producing an incomplete selection. This is fixed in 0.1.4. Revalidate affected observations before using them as opportunity evidence; see the historical validation and release qualification.

The local default compares HEAD with its first parent; both commits must be available. For a specific comparison, add --base <base-sha> --head <head-sha>. DiffCI prints the selection, fallback reasons, and the path to a JSON report outside your checkout. REFUSED or ERROR is not a successful analysis; check the reported status even when the command exits successfully. See the support matrix for setup requirements and supported workloads. check runs inferred full and selected commands in the checkout and sends nothing by default. The commands may write generated files. Use observe --no-send for analysis without execution. See docs/ai-agents.md for Claude Code, Codex, Cursor, GitHub Copilot, and similar tools.

Measured example: a controlled Cal.com replay showed 44.2% net reduction in a job-equivalent install + pretest + test workload, including analysis overhead. This is one sandbox comparison, not Cal.com's production savings or a prediction for your repository. Read the timings and method.

Selection counts alone do not establish runtime savings. check reports a measured percentage only when both commands pass and the checked-out commit and worktree remain identical across both arms. The savings artifact embeds the base/head SHAs, observation SHA-256, commands, timings, and checkout snapshots. Snapshots use byte-level fingerprints for dirty files, manifests, lockfiles, and available resolved-dependency markers; the report also records the runner identity. A mismatch invalidates the comparison. observe does not execute tests.

For an advanced paired runtime check, you can still run observe first and then run verify-savings against the observation report. It compares your normal full command with DiffCI's proposed selected command and writes JSON plus Markdown evidence; see docs/npm-adoption.md.

For stronger evidence, run repeated measurements with an explicit cache preparation command:

npx "@diffci.com/diffci@latest" verify-savings \
  --repo . --full "npm test" --selected-from-report ../diffci-observation.json \
  --out ../diffci-savings.json --repetitions 3 --cache-state warm \
  --cache-prepare "node scripts/prepare-ci-cache.mjs"

DiffCI alternates full/selected order, reports min/median/max timings, and conservatively classifies stable selection misses, likely flakes, shared/pre-existing failures, infrastructure failures, and inconclusive results. Cache preparation is user-supplied because DiffCI must not delete repository or tool caches on its own.

Validate same-turn change specifications

When an agent or build tool supplies explicit specification IDs and logical targets, DiffCI can block two specifications from silently changing the same target:

{
  "specifications": [
    { "id": "SPEC-API", "logicalTarget": "com.example.Widget#run" },
    { "id": "SPEC-BEHAVIOR", "logicalTarget": "com.example.Widget#run" }
  ]
}

npx "@diffci.com/diffci@latest" validate-specs --file change-specifications.json

The command exits non-zero and names every conflicting specification ID. This surface is opt-in and does not infer semantic targets from source code; callers must provide stable logical target names.

Observe in GitHub Actions

Save this as .github/workflows/diffci.yml to add a dedicated, non-blocking observation job:

name: DiffCI observation
on: [push, pull_request]
permissions:
  contents: read
jobs:
  diffci:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: DiffCI/DiffCI.com@e1d7bab271c5d83899bda0034d70d3e34c10c1f7

Then check the workflow locally with npx "@diffci.com/diffci@latest" verify-workflow. Keep the observer out of required checks and other jobs' needs lists. The Action adds a job summary and a diffci-observation artifact to the run; it does not alter which tests your other jobs execute. The example pins release v0.2.11 to its qualified feature commit SHA for reproducibility.

The CLI sends no report with --no-send. The Action uploads a GitHub artifact by default; sending to DiffCI's hosted service requires an explicitly configured endpoint and token. Installation details · Seven-day pilot

Use DiffCI

The CLI bundles a pinned revision of the Core engine from GitHub. Users install only @diffci.com/diffci; the check command above uses it directly. Core performs Git analysis, dependency graphs, impact, path baseline, and selected-command planning. The report format and non-interfering GitHub Action remain in this repository. See docs/package-relationship.md for the source relationship. For evaluation results and their limits, start with docs/adoption-evidence.md.

Agent Adoption

Add DiffCI instructions to a repository:

npx "@diffci.com/diffci@latest" init

Then ask your coding agent to run:

npx "@diffci.com/diffci@latest" check

Agent-specific docs: Codex · Claude Code · Cursor · GitHub Copilot · Grok.

Live discovery files: llms.txt · AI agents.

Adoption materials: outreach copy · metrics · targets.

For native agent integrations, DiffCI provides a stateless, read-only HTTPS MCP endpoint for validation guidance:

https://diffci.com/mcp

Use the stdio MCP server when tools need to inspect the local checkout or run tests:

npx -p "@diffci.com/diffci@latest" diffci-mcp