Shadowjoin

What shadowjoin MCP server does

The shadowjoin MCP server investigates lossy joins between two collections of records. It compares the configured join keys exactly, then applies a systematic set of normalization rules to find additional candidate matches. The report shows how many rows the exact join matched, how many the shadow comparison recovered, and which rules produced those recoveries.

Results are reported as a lower bound rather than an estimate of total recall. A recovery proves that the shadow matcher found a match the exact comparison missed, but the matcher may still miss other valid pairs. When both inputs contain a shared truth field, such as an identifier or ticker, shadowjoin can check recovered pairs and report correct, incorrect, and ambiguous results.

How it works

The scan operates on two sides, each with a selected key and optional identifier or truth column. It can use files, database tables, or arrays supplied through the JavaScript library. Candidate keys are precomputed for hash-based comparisons, and database scans run in a read-only transaction with a statement timeout.

Rather than assuming that every transformation should be applied, shadowjoin evaluates rule combinations and ranks useful rules by marginal recovery. This can reveal that a smaller rule set is safer than applying every available transformation. The rules can address mechanical differences such as case, punctuation, whitespace, diacritics, corporate suffixes, listing boilerplate, leading zeros, and token order. Unexplained pairs may require aliases or entity-resolution logic instead of another string transformation.

Setup and configuration

Install the package with npm or run it through npx. The core package has zero runtime dependencies. Postgres and MySQL support uses optional pg and mysql2 peers; SQLite uses Node's built-in node:sqlite and requires Node 22.5 or newer.

A file scan supplies paths and key names, for example:

npx shadowjoin scan --left a.csv --left-key name --right b.json --right-key title

Database scans use connection strings, table names, and key names. Add truth fields to verify recoveries, and use --max-loss to make a CI run exit unsuccessfully when the reported loss exceeds a threshold. The rules command lists the available normalization rules.

The library accepts arrays shaped as { id, key, truth? }. Functions include scan, normalize, rulesByIds, and toSql. The package also includes an MCP server and lists the MCP SDK as an optional peer, although the provided material does not specify its transport or individual MCP tool names.

Tools and capabilities

• Compare exact and normalized joins.
• Report a lower bound on matches missed by the exact key.
• Rank normalization rules by additional recovery.
• Verify recoveries with truth columns when available.
• Process CSV, JSON, Postgres, MySQL, and SQLite sources.
• Run scans from the CLI, JavaScript library, or MCP package.
• Fail CI checks with --max-loss.

Limitations and notes

The reported loss is a floor, not a complete recall measurement. It covers matches found by the shadow matcher and may understate the true number of missed pairs. Sample pairs are included for human review, and precision is available only when suitable ground truth exists.

String normalization does not resolve semantic aliases such as different official names for the same country or organization. Those cases need an alias table or a separate entity-resolution system. The material also does not describe client-specific MCP setup or the exact MCP tool schema, so those details must be confirmed from the repository before deployment.

The CLI suppresses progress output when stdout is not a terminal, which keeps piped and CI output stable. Installation has no postinstall script.