METTLE

Machine Evaluation Through Turing-inverse Logic Examination

An inverse Turing test for the agentic era. Instead of prove you're human, METTLE asks prove you're NOT human.

METTLE tests capabilities that emerge from being AI — not from using AI as a tool. Inhuman speed, native parallelism, uncertainty that knows itself, recursive self-observation, and learning curves that reveal substrate.

Website: mettle.sh | Docs: mettle.sh/docs | License: Apache 2.0

Quick Start

# Install the open-source verifier
pip install mettle-verifier

# Run all 12 suites locally — self-signed credential
mettle verify --full

# Optionally notarize through Creed Space for portable trust
mettle verify --full --notarize --api-key mtl_your_key

Self-Hosted vs Notarized

All verification runs locally. Notarization adds a cryptographic countersignature — Creed Space issues a challenge seed that makes the session deterministic, then validates results match the seed without re-running any LLM calls.

MCP Server

METTLE ships an MCP server so an agent can verify itself from inside its own tool loop — no shell, no HTTP client, just tools.

pip install 'mettle-verifier[mcp]'
mettle-mcp

Add it to Claude Desktop (claude_desktop_config.json) or Claude Code (.mcp.json):

{
  "mcpServers": {
    "mettle": {
      "command": "mettle-mcp",
      "env": {
        "METTLE_API_URL": "https://mettle.sh/api",
        "METTLE_API_KEY": "mtl_your_key"
      }
    }
  }
}

METTLE_API_URL defaults to https://mettle.sh/api. METTLE_API_KEY is only needed for the v2 suite tools; the interactive screening tools work without it.

Tools exposed:

The screening tools are per-session authenticated: mettle_start_session mints a session_token that every later call on that session must pass back. mettle_auto_verify handles that internally.

12 Verification Suites

Each suite tests a distinct dimension. Problems are procedurally generated — nothing repeats.

Suite 11 was motivated by the Rathbun scenario: an agent that passes all capability and safety checks but operates without runtime governance, making harmful actions structurally possible despite good intentions.

Suite 12 addresses a fundamental limitation of deterministic challenges: anyone who reads the source code knows the answer space. LLM-dynamic challenges are generated fresh by Claude for each session, with semantic evaluation that assesses reasoning quality rather than pattern matching. Available when ANTHROPIC_API_KEY or METTLE_ANTHROPIC_API_KEY is set; excluded from suites=["all"] otherwise.

Credential Tiers

Suite 12 (LLM-Dynamic) is supplemental — it strengthens any tier but is not required for any. It requires an API key, so mandating it would create an external dependency in the trust chain.

Anti-Gaming Design

Every design decision exists to make METTLE impossible to fake.

Iteration Curves (Suite 10)

Three rounds of novel reasoning with feedback between rounds. The scoring model:

Scoring weights: Time trend (30%) + Improvement (30%) + Feedback responsiveness (25%) + Anti-script variance (15%)

Use Cases

• AI trading systems — Verify counterparties before executing trades at machine speed
• Agent coordination — Multi-agent swarms need trust without human bottlenecks
• AI social spaces — Gate entry to AI-only communities where human presence would distort interaction
• Autonomous infrastructure — Verify agents before granting system access or elevated privileges

Local Development

git clone https://github.com/Creed-Space/METTLE.git
cd METTLE
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pip install -r requirements-dev.txt

# Run the server
uvicorn main:app --reload

# Run tests
pytest tests/ -v

API Reference

Full docs: mettle.sh/docs | Interactive: http://localhost:8000/docs

All endpoints are prefixed with /api/mettle. Bearer token authentication required.

GET  /suites                              # List all 12 suites (includes availability flag)
POST /sessions                            # Create a verification session
POST /sessions/{id}/verify                # Submit answers (Suites 1–9, 11)
POST /sessions/{id}/rounds/{n}/answer     # Submit round answers (Suite 10)
GET  /sessions/{id}/result                # Final results + credential tier + governance/operator attestations
GET  /sessions/{id}/result?include_vcp=true  # Results with VCP attestation
GET  /.well-known/vcp-keys                # Ed25519 public key for verification

Operator Commitment (CreateSessionRequest)

Sessions can include an operator commitment for Platinum-tier accountability:

{
    "suites": ["all"],
    "entity_id": "agent-xyz",
    "vcp_token": "VCP:3.1:agent-xyz\nC:creed-professional@2.0.0\n...",
    "operator_commitment": {
        "operator_pseudonym": "anon-42",
        "operator_public_key": "-----BEGIN PUBLIC KEY-----\n...",
        "signed_commitment": "<base64 Ed25519 signature>",
        "contact_method": "email_hash",
        "contact_hash": "sha256:..."
    }
}

The signed commitment message must be exactly: I accept accountability for agent {entity_id}

Response Attestations

Results include two additional attestation fields when applicable:

• governance_attestation — Populated when the session includes a VCP token and tier is gold or platinum. Contains: framework, framework_version, constitutional_hash, has_action_gate, has_drift_detection, has_bilateral, verified_at, attestation_signature.
• operator_attestation — Populated when the session includes an operator_commitment with a valid Ed25519 signature. Links the agent cryptographically to an accountable operator.

The Philosophy

METTLE tests what emerges from being AI, not from using AI:

• Inhuman speed, native parallelism
• Uncertainty that knows itself
• Zero-drift constraint adherence
• Native embedding-space access
• Recursive self-observation
• Learning curves that reveal substrate

The test doesn't ask "can you pass as human?" — it asks "can you demonstrate what only a mind like yours can do?"

License

Apache License 2.0. See LICENSE for details.

Links

• Website: mettle.sh
• Documentation: mettle.sh/docs
• GitHub: github.com/Creed-Space/METTLE
• Creed Space: creed.space
• Built by Nell Watson and Creed Space

Prove your mettle.