Universal Test Framework (UTF)

Contract-driven test enforcement and reporting for LLM-generated code — via MCP, VS Code, Copilot CLI, Claude, Cursor, or Python SDK.

UTF is an MCP server and contract enforcement engine. It does not replace your AI coding assistant — it gives every test your AI writes a mandatory quality gate, a structured audit trail, and a comprehensive compliance report.

Why UTF? (Not Another Test Generator)

AI tools — GitHub Copilot, Claude, Cursor, Gemini — generate tests fast. The problem: fast ≠ trustworthy.

UTF vs Markdown Instructions / Prompt Files

You may already use markdown files (AGENTS.md, copilot-instructions.md, .cursorrules) to guide your AI. UTF is complementary — not competing:

Use markdown instructions to shape how your AI thinks. Use UTF to verify and report on what it produced.

Key Features

• 13 MCP tools — generate, validate, register, analyze, trace, execute, mutate, report, and query tests
• 6 languages — Python, TypeScript, JavaScript, Java, Go, C++
• 7 test types — unit, integration, API, E2E, contract, performance, security
• 8-section contract — every test must pass all 8 sections or it is blocked (minimum score: 0.85)
• 3-segment test IDs — TC-{PRJ}-{MODULE}-{NNN} and classic TC-{MODULE}-{NNN} both accepted
• Five install modes — uvx (zero-clone), local clone, Docker, GitHub MCP Registry, pip SDK
• VS Code @utf — chat participant with slash commands
• CI/CD ready — GitHub Actions, GitLab CI, pre-commit hooks
• Per-project overrides — YAML rules in .utf/rules/ override global defaults

Quick Start

Setup Method Comparison

All methods (1, 2, 4) using stdio transport write the registry to ${workspaceFolder}/.utf/utf.db — isolated per project and persistent across sessions.

Method 1 — uvx (Zero-Install, Recommended)

Requires uv. No cloning, no venv.

Add to %APPDATA%\Code\User\mcp.json (Windows) or ~/.config/Code/User/mcp.json (macOS/Linux):

{
  "servers": {
    "utf": {
      "type": "stdio",
      "command": "uvx",
      "args": ["--from", "universal-test-framework", "utf-server", "--transport", "stdio"],
      "cwd": "${workspaceFolder}",
      "env": {
        "UTF_PROJECT_DIR": "${workspaceFolder}"
      }
    }
  }
}

Registry: .utf/utf.db inside ${workspaceFolder} — isolated per project, persistent across sessions.
Reload VS Code after editing mcp.json.

Method 2 — Local Clone (VS Code + Copilot)

git clone https://github.com/phoenice-labs/Universal-Test-Framework
cd Universal-Test-Framework

# Register MCP server, install @utf extension, copy global prompts
.\scripts\install-vscode-mcp.ps1

# Optionally scaffold a specific project
.\scripts\install-vscode-mcp.ps1 -InitProject -ProjectDir C:\my-project

Reload VS Code → open Copilot Chat → ask naturally: generate e2e tests for my backend.

The install script writes this entry to mcp.json:

{
  "utf": {
    "type": "stdio",
    "command": "<python>",
    "args": ["-m", "mcp_server.server", "--transport", "stdio"],
    "cwd": "${workspaceFolder}",
    "env": { "PYTHONPATH": "<UTF_install_dir>" }
  }
}

Registry: ${workspaceFolder}/.utf/utf.db — isolated per project.
@utf slash commands are available after the VSIX extension is installed.

Method 3 — Docker (Team / Remote)

# Start the UTF server
cd Universal-Test-Framework
docker compose -f docker/docker-compose.yml up -d

# MCP server available at http://localhost:8765/sse

Connect from VS Code by adding to mcp.json:

{
  "servers": {
    "utf-remote": {
      "type": "sse",
      "url": "http://localhost:8765/sse"
    }
  }
}

Registry: persisted in a named Docker volume (utf_registry → /app/.utf/utf.db).
For per-project isolation with Docker, use a bind-mount in docker-compose.yml:

yaml

Copy

volumes:
  - /path/to/your/project/.utf:/app/.utf

Because Docker uses HTTP/SSE transport (no ${workspaceFolder} templating), pass project_dir explicitly in tool calls, or set UTF_PROJECT_DIR in the container environment.

Per-project Docker workflow:

# docker-compose.override.yml
services:
  utf-server:
    volumes:
      - ./my-project/.utf:/app/.utf
    environment:
      UTF_PROJECT_DIR: /app

Method 4 — GitHub MCP Registry

Once published, UTF is discoverable via the MCP marketplace. Clients that support server.json install it automatically. The generated mcp.json entry is equivalent to Method 1 (uvx).

Registry isolation: The MCP registry schema does not support a cwd field at the registry level.
UTF resolves project isolation via (in priority order):

1. project_dir argument passed to each tool call
2. UTF_PROJECT_DIR environment variable
3. Path.cwd() fallback (server's working directory)

For correct isolation, ensure the MCP client writes "cwd": "${workspaceFolder}" and "UTF_PROJECT_DIR": "${workspaceFolder}" in the generated entry (UTF's mcp-gallery.json does this).

Method 5 — pip + Python SDK

pip install universal-test-framework

from mcp_server.tools.generate_tests import generate_tests
from pathlib import Path

result = generate_tests(
    test_type="unit",
    source_code=open("src/auth.py").read(),
    project_dir=str(Path.cwd()),   # ← pass explicitly for correct registry isolation
)
print(result["suite_code"])

Registry: <project_dir>/.utf/utf.db when project_dir is passed; falls back to Path.cwd().

MCP Tools Reference

The 8-Section Test Contract

Every test generated by UTF must satisfy all 8 sections. Tests that fail any hard section are blocked — returned in blocked_tests, never silently included.