slop-eval

Quickstart • CLI reference • Library API • MCP Server • Comparison • FAQ

Score AI-generated UI for genericness with an LLM judge, so a CI check catches the same "this looks like every other AI-built app" problem a human reviewer would flag on sight.

npx slop-eval-cli score --screenshot ./preview.png --json

No install step: npx fetches and runs the published npm package directly. Prefer Python? pip install slop-eval-cli gets you the same CLI as a genuine, independent port of the scoring logic.

Two distributions: npm and Python, both live

slop-eval-cli is live on both npm and PyPI (package slop_eval). The Python port is a genuine, independent implementation, built and tested (60/60 tests, verified in this pass) against the same rubric and Anthropic judge prompt as the TypeScript original. See python/README.md for Python-specific usage.

Why this exists, and what it isn't

Nutlope's Hallmark, a popular AI design skill with 21,000+ stars, has an open issue where a user says flatly: "all of it looks like slop." The maintainer closed it NOT_PLANNED. Separately, a contributor opened a PR against Hallmark titled "Add eval-driven quality harness for Hallmark outputs" that has sat open and unmerged for about two months as of this writing. Both are real and dated as of this writing. Neither proves the demand is large, only that the gap is real and currently unaddressed.

slop-eval is not the first tool in this space, and it doesn't try to be. Two real, free tools already sit nearby:

• Impeccable (pbakaus/impeccable, 54,000+ stars, Apache 2.0) ships a CLI that flags 59 specific visual tells of AI-generated UI (gradient palettes, glassmorphism, side-stripe borders, WCAG contrast violations), all enabled by default with no model call; a separate impeccable critique command adds further, opt-in LLM-based judgments on top. Core detection stays fast because it doesn't need a model for any of its default checks. It has grown well beyond a slop detector into a full design-language skill for Claude Code, Cursor, and Codex, with 23 commands total.
• aislop (MIT, 500+ stars) does the deterministic, rule-based equivalent for AI-generated code (not UI): 50+ regex/AST rules across 8 languages, no LLM in the runtime path, positioned exactly as a CI quality gate.

Neither does holistic, judgment-based UI scoring: "does this layout feel novel," "does this component choice feel considered," the kind of read a fixed rule can't easily encode. That's the gap slop-eval fills, built to compose with tools like Impeccable's rather than replace them.

Features

Verified directly against the code in this repo:

• Three rubric categories, each with mandatory cited evidence. src/rubric/v1.json scores layout novelty, visual-identity distinctiveness, and component-pattern novelty, 0-10 each. A finding with no specific citation is treated as a bug, not a valid score (see src/sources/RuleSource.ts).
• LLM judge via forced tool-call, returning structured JSON. LLMJudgeSource calls the Anthropic API with tool_choice locked to a submit_slop_scores schema: the response comes back as reliably structured JSON instead of a chat reply that has to be regexed apart.
• --json mode for CI and agents. Every run can emit a parseable { target, rubric, compositeScore, findings[], summary, disclaimer } object on stdout, on both success and error paths, so a script or agent never has to branch on shape to find an error string.
• Real exit-code contract. 0 success (no threshold, or score at/above --fail-below), 1 success but below threshold, 2 usage error or unrecoverable failure. Verified directly against the built CLI and the real npm/PyPI packages this session; see CLI reference.
• Content-hash caching. src/cache/judge-cache.ts hashes the input bytes and skips the API call entirely on a repeat run against unchanged input. That's a correctness guarantee as much as a cost saver: an unchanged PR can't flap a CI gate from LLM run-to-run variance.
• Composable RuleSource plugin interface. src/sources/RuleSource.ts is the boundary every scoring source implements. Today that's one real source (LLMJudgeSource) and one documented stub (ScreenshotDiffSource, honestly reported as not_scored until a real labeled corpus exists), so a future rule catalog or a second LLM provider slots in without touching the composite scorer.
• Screenshot input (real visual read) or --url fallback. --screenshot sends the actual rendered image to the judge. --url is a documented v0.1 limitation: no bundled headless browser, so it fetches raw HTML/text and the judge reasons over markup and copy instead of layout.
• GitHub Action that leads with the specific flag, then the score. action/action.yml posts a PR comment headed by the single most specific flagged finding, followed by the composite score, giving a reviewer the reasoning behind the number.
• Versioned, public rubric. Every score names the rubric version (v1 today) that produced it. Rubric changes ship as a new file, never a silent edit to an existing one.
• A real, agent-native library API alongside the CLI. Both distributions export a programmatic entry point (score_composite and friends in Python, runScore/scoreComposite in TypeScript) so an agent framework can call slop-eval in-process instead of shelling out. See Library API.

Quickstart

Requires Node.js 18+ (npm) or Python 3.9+ (PyPI), and an ANTHROPIC_API_KEY (BYO key; get one at console.anthropic.com).

The fastest path, no local clone or build needed, is the one-liner at the top of this README:

npx slop-eval-cli score --screenshot ./preview.png --json

Verified this session against the real published npm package, with a real PNG at ./preview.png and no ANTHROPIC_API_KEY set:

$ npx --yes slop-eval-cli@latest score --screenshot ./preview.png --json
{
  "error": "ANTHROPIC_API_KEY environment variable is not set.\nslop-eval calls the Anthropic API to run the LLM judge, and is BYO-key (bring your own key) -- there is no default or shared key baked into this tool. Set your key and try again:\n\n  export ANTHROPIC_API_KEY=\"sk-ant-...\"\n\nGet a key at https://console.anthropic.com/"
}
# exit code 2

To build from source instead:

git clone https://github.com/RudrenduPaul/slop-eval.git
cd slop-eval
npm install
npm run build

export ANTHROPIC_API_KEY="sk-ant-..."
./dist/cli.js score --screenshot ./test/fixtures/sample.png

For CI or agent consumption, add --json. --json always emits a valid JSON object on stdout, on both the success and error paths, and the --url/--screenshot mutual-exclusivity check is a good example of a real usage-error path you can rely on being parseable:

./dist/cli.js score --screenshot ./test/fixtures/sample.png --json

{
  "target": "./test/fixtures/sample.png",
  "rubric": "v1",
  "compositeScore": 62,
  "findings": [
    {
      "ruleId": "llm-judge.layout-novelty",
      "category": "Layout novelty",
      "score": 4,
      "evidence": "Matches a common hero + 3-card grid + footer CTA pattern.",
      "status": "flag"
    }
  ],
  "summary": { "pass": 1, "flagged": 1, "notScored": 1 },
  "disclaimer": "This score is a heuristic quality signal from an LLM judge, not a certification..."
}

CLI reference

Captured directly from ./dist/cli.js score --help on the built CLI this session, word for word:

Usage: slop-eval score [options]

Score a URL or screenshot for AI-UI genericness against a versioned rubric.

Note on --url mode (v0.1 limitation): this tool does not bundle a headless
browser. If --url is given, the raw HTML/text response is fetched and given to
the judge as a fallback input, instead of a rendered screenshot -- the judge
can reason about markup and copy, but not the actual visual layout. For the
stronger, layout-aware signal, render the page yourself and pass --screenshot.

Read the full README on GitHub →