MCP as a Judge ⚖️

mcp-name: io.github.OtherVibes/mcp-as-a-judge

MCP as a Judge acts as a validation layer between AI coding assistants and LLMs, helping ensure safer and higher-quality code.

MCP as a Judge is a behavioral MCP that strengthens AI coding assistants by requiring explicit LLM evaluations for:

• Research, system design, and planning
• Code changes, testing, and task-completion verification

It enforces evidence-based research, reuse over reinvention, and human-in-the-loop decisions.

If your IDE has rules/agents (Copilot, Cursor, Claude Code), keep using them—this Judge adds enforceable approval gates on plan, code diffs, and tests.

Key problems with AI coding assistants and LLMs

• Treat LLM output as ground truth; skip research and use outdated information
• Reinvent the wheel instead of reusing libraries and existing code
• Cut corners: code below engineering standards and weak tests
• Make unilateral decisions when requirements are ambiguous or plans change
• Security blind spots: missing input validation, injection risks/attack vectors, least‑privilege violations, and weak defensive programming

Vibe coding doesn’t have to be frustrating

What it enforces

• Evidence‑based research and reuse (best practices, libraries, existing code)
• Plan‑first delivery aligned to user requirements
• Human‑in‑the‑loop decisions for ambiguity and blockers
• Quality gates on code and tests (security, performance, maintainability)

Key capabilities

• Intelligent code evaluation via MCP sampling; enforces software‑engineering standards and flags security/performance/maintainability risks
• Comprehensive plan/design review: validates architecture, research depth, requirements fit, and implementation approach
• User‑driven decisions via MCP elicitation: clarifies requirements, resolves obstacles, and keeps choices transparent
• Security validation in system design and code changes

Tools and how they help

🚀 Quick Start

Requirements & Recommendations

MCP Client Prerequisites

MCP as a Judge is heavily dependent on MCP Sampling and MCP Elicitation features for its core functionality:

• MCP Sampling - Required for AI-powered code evaluation and judgment
• MCP Elicitation - Required for interactive user decision prompts

System Prerequisites

• Docker Desktop / Python 3.13+ - Required for running the MCP server

Supported AI Assistants

✅ Recommended setup: GitHub Copilot + VS Code — full MCP sampling; no API key needed.

⚠️ Critical: For assistants without full MCP sampling (Cursor, Claude Code, Augment, Qodo), you MUST set LLM_API_KEY. Without it, the server cannot evaluate plans or code. See LLM API Configuration.

💡 Tip: Prefer large context models (≥ 1M tokens) for better analysis and judgments.

If the MCP server isn’t auto‑used

For troubleshooting, visit the FAQs section.

🔧 MCP Configuration

Configure MCP as a Judge in your MCP-enabled client:

Method 1: Using Docker (Recommended)

One‑click install for VS Code (MCP)

Notes:

• VS Code controls the sampling model; select it via “MCP: List Servers → mcp-as-a-judge → Configure Model Access”.

1. Configure MCP Settings:

   Add this to your MCP client configuration file:

   config.json

   Copy

   {
     "command": "docker",
     "args": ["run", "--rm", "-i", "--pull=always", "ghcr.io/othervibes/mcp-as-a-judge:latest"],
     "env": {
       "LLM_API_KEY": "your-openai-api-key-here",
       "LLM_MODEL_NAME": "gpt-4o-mini"
     }
   }
   

   📝 Configuration Options (All Optional):

   • LLM_API_KEY: Optional for GitHub Copilot + VS Code (has built-in MCP sampling)
   • LLM_MODEL_NAME: Optional custom model (see Supported LLM Providers for defaults)
   • The --pull=always flag ensures you always get the latest version automatically

   Then manually update when needed:

   bash

   Copy

   # Pull the latest version
   docker pull ghcr.io/othervibes/mcp-as-a-judge:latest
   

Method 2: Using uv

1. Install the package:

   bash

   Copy

   uv tool install mcp-as-a-judge
   

2. Configure MCP Settings:

   The MCP server may be automatically detected by your MCP‑enabled client.

   📝 Notes:

   • No additional configuration needed for GitHub Copilot + VS Code (has built-in MCP sampling)
   • LLM_API_KEY is optional and can be set via environment variable if needed

3. To update to the latest version:

   bash

   Copy

   # Update MCP as a Judge to the latest version
   uv tool upgrade mcp-as-a-judge
   

Select a sampling model in VS Code

• Open Command Palette (Cmd/Ctrl+Shift+P) → “MCP: List Servers”
• Select the configured server “mcp-as-a-judge”
• Choose “Configure Model Access”
• Check your preferred model(s) to enable sampling

🔑 LLM API Configuration (Optional)

For AI assistants without full MCP sampling support you can configure an LLM API key as a fallback. This ensures MCP as a Judge works even when the client doesn't support MCP sampling.

• Set LLM_API_KEY (unified key). Vendor is auto-detected; optionally set LLM_MODEL_NAME to override the default.

Supported LLM Providers

Client-Specific Setup

Cursor

1. Open Cursor Settings:
   • Go to File → Preferences → Cursor Settings
   • Navigate to the MCP tab
   • Click + Add to add a new MCP server