Interactive Data Systems Lab

Runnable Python and Rust data-system internals with an MCP-native runtime copilot layer.

This repository combines two things:

• a local-first engineering lab for PostgreSQL-like and Databricks-like internals,
• a Runtime Copilot MCP surface for diagnostics, explainability, regression checks, and operational memory.

Core capabilities

• Storage and planner internals: heap tables, B-tree indexing, selectivity, and plan choice.
• Persistence and replay: WAL/checkpoint style flows and deterministic state transitions.
• Workflow and write-path modeling: event-first architecture, idempotency, retry semantics.
• Explainable runtime operations: traced runs, failure summaries, regression verdicts, baseline compare.
• MCP access: machine-usable operational interface instead of ad hoc shell scripts.

Who this is for

• Data engineers learning warehouse and query-engine internals.
• Platform and infrastructure engineers teaching storage and execution fundamentals.
• Teams building onboarding labs, workshops, and demo environments.

Quick start

Requirements:

• Python 3.10+ (tested with Python 3.14)

Setup:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install "psycopg[binary]"

Run end-to-end flow:

cargo run --bin e2e_flow

Run core demos:

.venv/bin/python mini_pg_like.py
.venv/bin/python mini_databricks_clone.py
cargo run --bin mini_pg_like
cargo run --bin mini_databricks_clone

What this repository contains

• mini_pg_like.py: PostgreSQL-like toy engine with heap table, B-tree index, and planner output.
• mini_databricks_clone.py: Databricks-like toy platform with versioning, partitions, DAGs, and events.
• src/bin/mini_pg_like.rs: Rust PostgreSQL-like demo.
• src/bin/mini_databricks_clone.rs: Rust Databricks-like demo.
• src/lib.rs, src/common.rs, src/pg.rs: shared Rust core modules.
• mcp_engine_server.py: MCP runtime adapter for diagnostics and regression workflows.

Why this exists

Most internals content stops at diagrams. This project stays runnable and inspectable:

• compare Python and Rust implementations of the same system ideas,
• trace write-path behavior with concrete events and state transitions,
• run explainable regression checks through MCP,
• turn runtime operations into a discoverable control surface for AI clients.

MCP Adapter Layer

This repo also includes a minimal MCP server that wraps the lab operations:

• mcp_engine_server.py
• Cursor config: .cursor/mcp.json

Current MCP tool list for this release (47 tools total):

Engine state and runtime:

• init_engine
• insert_row
• upsert_row
• create_index
• explain_customer
• reindex_project
• run_e2e_flow

Explainability and demos:

• explain_run
• demo_explain_run
• demo_explain_run_failure
• demo_explain_semantic_failure
• demo_explain_idempotency_conflict
• demo_explain_concurrency_failure_storm
• explain_regression_suite

Trace and retrieval:

• record_tool_trace
• similar_incidents
• refresh_trace_path
• refresh_docs_path
• memory_upsert
• memory_search

SLO and ROI:

• health_check
• benchmark_calls
• scenario_load_test
• capture_roi_baseline
• report_drift_bug
• decision_gate

Schema evaluation (verdict + report via MCP, no external Postgres):

• schema_load_tool
• schema_explain_tool
• schema_evaluate_tool
• schema_evaluate_full_tool

Project contract and regression:

• project_manifest
• project_capabilities
• project_tool_catalog
• project_get_defaults
• project_run_regression
• project_capture_baseline
• project_compare_baseline

Generic project state:

• project_list_entities
• project_get_entity
• project_upsert_entity
• project_delete_entity
• project_append_event
• project_ingest_trace
• project_explain_run
• project_export_state

Generic heuristics:

• project_list_heuristics
• project_run_heuristic

For machine-readable discovery, prefer:

• project_tool_catalog
• project_get_defaults

Current heuristic profiles available through project_run_heuristic:

• pain_structure
• naive_bias
• price_distribution
• liquidity_signals
• price_liquidity_matrix
• cross_category
• sale_format
• speed_signals
• trust_signals

If Cursor MCP auto-discovery is enabled, restart Cursor and connect mini-data-engine. Default MCP runtime data paths are under tests/artifacts/mcp/*.

Cursor approval setup (reduce repeated prompts)

If Cursor keeps asking for MCP or command approval on every call, apply this once:

1. Enable workspace trust in Cursor user settings:

"security.workspace.trust.enabled": true

2. In Cursor, open Settings -> Agents -> Auto-Run and set:

   • Auto-run mode: Run in Sandbox
   • MCP Allowlist: add mini-data-engine tools you use often
   • Command Allowlist: add frequently used safe commands

3. Keep this repo opened as the same trusted workspace and reload the window once.

Notes:

• MCP server approval and per-tool allowlist behavior are enforced by Cursor security settings.
• In some Cursor versions, allowlist behavior can be best-effort and still prompt in edge cases.

Fastest way to see the new explainability use case in action through MCP:

demo_explain_run

That single tool call creates a traced run, records step-level events under one run_id, and returns an explanation with:

• ordered timeline
• tool path
• total elapsed time
• failure summary if anything breaks

You can then replay the same explanation directly with:

explain_run(run_id="...")

Explain Regression Suite

Use explain_regression_suite when you want regression checks to run through MCP and come back as explainable run summaries instead of isolated test output.

The suite drives the current validation surface through the MCP layer, attaches run_id traces, and returns explain output for each check so regressions can be inspected with the same mechanism used for runtime incidents.

It currently runs:

• Python unit tests via python -m unittest discover
• Rust tests via cargo test, including the current engine_cli integration tests
• health_check
• benchmark_calls
• scenario_load_test
• explainability control demos

The explainability demos intentionally include both positive and negative controls:

• demo_explain_run as expected_success
• demo_explain_run_failure as expected_failure
• demo_explain_semantic_failure as expected_failure
• demo_explain_idempotency_conflict as expected_failure
• demo_explain_concurrency_failure_storm as expected_failure

That means the suite is not only checking that the happy path stays green. It also checks that the explain layer still classifies and summarizes known failure classes correctly.

The current regression surface covers:

• happy-path traced execution
• runtime/path failures
• semantic data validation failures
• idempotency conflict failures
• concurrency and failure-storm control scenarios
• sampled benchmark and scenario SLO regressions

Fastest MCP call for the full regression bundle:

explain_regression_suite

Use it as the top-level MCP regression entrypoint when you want one answer that includes:

• which checks passed
• which failures were expected controls
• explain summaries for each traced run
• early signals that a latency or behavior regression appeared

The MCP layer is an access interface, not the core product idea. The core of the repository is the runnable lab itself.

Product note:

• PRODUCT_NOTE_RUNTIME_EXPLAINABILITY.md Short note describing the runtime explainability use case, the required signals, and the explain_run MVP.
• PRODUCT_NOTE_RUNTIME_COPILOT.md Product framing for Runtime Copilot as an MCP-native operational brain.
• EXPLAIN_REGRESSION_SUITE_FEASIBILITY.md Short article describing what this repository validated about explain-first regression suites and where the current denominator still stays narrow.

Use in Codex:

• skill package: codex/skills/runtime-copilot/SKILL.md
• automation examples: codex/automations
• guide: docs/use-in-codex.md

Run persistent engine CLI (productization path):

# Initialize storage
cargo run --bin engine_cli -- init ./tests/artifacts/engine/data orders

# Insert and upsert (WAL append)
cargo run --bin engine_cli -- insert ./tests/artifacts/engine/data orders 1 4242 50
cargo run --bin engine_cli -- upsert ./tests/artifacts/engine/data orders 1 4242 55

# Build index and explain
cargo run --bin engine_cli -- index ./tests/artifacts/engine/data orders
cargo run --bin engine_cli -- explain ./tests/artifacts/engine/data orders 4242

# Write snapshot and truncate WAL
cargo run --bin engine_cli -- checkpoint ./tests/artifacts/engine/data orders

# Transaction simulation: begin/commit/rollback semantics,
# per-table write lock, snapshot read, and conflict detection
cargo run --bin engine_cli -- tx-demo ./tests/artifacts/engine/data orders

# Crash/restart recovery for transaction journals
cargo run --bin engine_cli -- tx-recovery-list ./tests/artifacts/engine/data orders
cargo run --bin engine_cli -- tx-recovery-commit ./tests/artifacts/engine/data orders <tx_id>
cargo run --bin engine_cli -- tx-recovery-rollback ./tests/artifacts/engine/data orders <tx_id>

What You Should See