CompletionKit

What CompletionKit does

The CompletionKit MCP server provides an MCP interface for evaluating prompts against real input data. A prompt is a versioned template containing {{variable}} placeholders, while a dataset is a CSV whose headers provide values for those placeholders. Each dataset row becomes an evaluation case.

A run sends the rendered prompts to a selected model and evaluates the responses with an LLM judge. The judge scores outputs from 1 to 5 against criteria defined for the evaluation. Prompt versions, run results, and score changes remain available for comparison, so developers can test changes to the prompt, model, temperature, or dataset.

When scores decline, CompletionKit can generate a proposed prompt improvement from the judge’s response-level feedback. The proposed change can be reviewed as a diff, applied as a new prompt version, and evaluated again.

How it works

The workflow starts by creating a prompt and uploading a CSV dataset with matching column names. A run executes the prompt for every row, records the generated outputs, and sends them to the configured judge. Results can then be compared with earlier prompt versions to identify improvements or regressions.

CompletionKit exposes the same underlying engine through its web interface, REST API, and MCP endpoint. The deployment can be hosted at completionkit.com, run as the bundled standalone application, or mounted as a Ruby on Rails engine. The standalone application requires Postgres, a web process, and a Solid Queue worker. The Rails engine requires a durable Active Job adapter such as Solid Queue, Sidekiq, or GoodJob for generation and judging jobs.

Setup and configuration

The CompletionKit MCP server can be used through the hosted product or deployed on your own infrastructure. For self-hosting, the repository’s standalone application is installed with Bundler, its migrations are run, and both the Rails server and job worker are started. A Rails integration uses the completion-kit gem and mounts the engine at /completion_kit.

Model providers include OpenAI, Anthropic, Ollama or another OpenAI-compatible local endpoint, OpenRouter, and Azure AI Foundry. Provider credentials may be supplied through environment variables, the generated initializer, or the web interface. Production deployments must configure Active Record encryption keys for stored provider credentials. Deployed environments also require authentication; basic authentication and a custom authentication strategy are supported.

Tools and capabilities

The CompletionKit MCP server supports capabilities centered on prompt evaluation, including:

• Running prompts over CSV-backed test cases
• Scoring generated responses with an LLM judge
• Comparing results across prompt versions and runs
• Testing different models, temperatures, and datasets
• Producing prompt-revision suggestions from judge feedback
• Accessing the evaluation system through MCP and REST

Specific MCP tool names are not provided in the supplied material.

Limitations and notes

The provided documentation does not list individual MCP tool names or define a client-specific setup procedure. Provider access depends on valid credentials or a reachable local model endpoint. Long-running generation and judging work is processed asynchronously, so production Rails deployments need a job adapter with durability and retries; the :async adapter is not suitable for production.

Rate limits apply separately to the REST API and MCP endpoint, as well as the web UI. Limits can be configured and are tracked through Rails cache. In production, provider credentials are encrypted at rest, and the required encryption keys must be configured before the application boots.