mdcx

Convert a document collection to verified Markdown, package it into a single encrypted file, and query it from an agent through the Model Context Protocol.

Contents

• Overview
• Requirements
• Installation
• Quick start
• Conversion
• Packaging and querying
• Sent and received
• Working incrementally
• What the corpus knows about words
• Something to keep that is not text to search
• Writing often
• MCP server
• When the client goes away
• Reaching a word the transcription got wrong
• Language support
• Cross-language retrieval
• Portable paths
• Signing
• Encryption
• Limitations
• Tests
• Contributing
• Security
• Releases
• Authorship
• Citation
• Licence

Overview

mdcx converts a collection of documents to Markdown, verifies each conversion against its original, packages the corpus with its index and provenance into a single encrypted file, and serves that file to agents over the Model Context Protocol.

It addresses one constraint. An agent asked a question about a document collection must either receive the documents in its context window, which is bounded in size and billed per token, or query a component that holds an index and returns only the passages that bear on the question. mdcx implements the second. Three properties distinguish it from an extraction script:

• Fidelity is measured, not assumed. Every conversion is checked against the text the original exposes, read by a library independent of the engine that produced the conversion, and the coverage achieved is recorded per file.
• The corpus is a single encrypted artefact. Passages, index and provenance are held in one AES-256-GCM file whose header can be read without the key.
• Every passage carries its source. An answer can be cited against a document and a location rather than recalled.

Pipeline

Conversion. Each document is attempted by the least expensive engine capable of reading it and escalated only where that engine falls short: direct text extraction, then a pass that recovers the tables a page draws, then full layout analysis. Documents exposing no text are read by optical character recognition. Content the selected engine omitted is appended verbatim rather than reported as lost.

Over the collection used during development — 99 documents, 1,144,553 reference tokens — 594 tokens were not recovered, a coverage of 99.948%. Of the 95 documents that expose text, 70 were recovered in full and none fell below 99.5%. The remaining four are scanned drawings holding no text in the file; they are marked unverifiable, as no text original exists to measure them against.

Packaging. The corpus, its search index and the provenance of every passage are written to a single .mdcx file. The development collection produced 3.9 MB from 8.8 MB of Markdown. A growing collection is not rebuilt from the start: vectors already computed are reused, and a corpus exceeding what can be decrypted into memory is held as several packages queried as one.

Retrieval. A query returns the passages that answer it, each with its source document and its position in the ranking. Word matching and dense retrieval are merged by reciprocal rank, so a query reaches a document whether it shares that document's vocabulary or only its subject, including where the two are written in different languages. Over a corpus of 136 documents in 34 languages, the merged engines rank the expected document first for 135 of the 136 queries. Where no document in the corpus is about the question, the reply states this rather than presenting its nearest passage as an answer.

Measured cost

One query over the development collection — 99 documents, 180 MB — counted with the cl100k_base tokenizer:

The 435 model tokens comprise 20 for the question, 274 for the retrieved passage and 141 for the answer.

Reading the originals costs the whole collection because a PDF is a binary format: absent prior conversion there is no way to determine which of the 99 documents holds the answer, so all of them are extracted and read.

This is a single measurement, not an average, and the saving depends on how much text an answer requires. The work is not eliminated but relocated, from the context window, which is billed and finite, to local processing, which is neither. The local column rises for that reason.

Requirements

Python 3.11 or later. No other component is required to query a package.

The floor is 3.11 because a package is held as one SQLite database and serialised in memory to be encrypted, and sqlite3 gained the call that does so in that version. Earlier interpreters were declared supported and were not: neither building a package nor opening one worked there. Conversion and cross-language retrieval each add dependencies, listed under Installation.

Installation

Querying and conversion are separated because their requirements differ by two orders of magnitude.

Conversion accounts for the heavy dependencies. A recipient who only queries an .mdcx file installs neither Docling nor PyTorch.

The multilingual extra is required for queries that cross languages. Most of its size is the embedding model, downloaded once on first use. A single-language corpus does not require it.

The tables extra covers what a page does not draw. Tables in printed material are usually found from the rules drawn around them, which costs nothing and needs no extra; borderless ones — a screenshot of a spreadsheet, a layout held together by alignment — are read by a small model that reports where the rows and columns run. It reads the shape only: the words still come from the text layer of the document, so a cell cannot hold anything the page does not say. Without it those pages are read by Docling instead, which is slower but already present in the convert extra.

If the machine has a CUDA card

Install mdcx[all-gpu] rather than mdcx[all], and install onnxruntime-gpu yourself.

onnxruntime and onnxruntime-gpu are two distributions publishing the same module, so they cannot coexist: whichever pip wrote last wins, and it is usually the CPU one. An extra that pins the CPU build therefore removes CUDA from an environment that had prepared it, on every upgrade — measured on three consecutive releases, in two environments each time, with no error, nothing in any log, and optical recognition simply costing tens of times more. all-gpu is all without that pin.

pip cannot express "either of these distributions", so this cannot be settled by declaration alone. mdcx-convert therefore checks at startup: when the machine has a card and the runtime does not offer it, it says so and gives the repair. The check on its own is one line:

python -c "import onnxruntime as o; assert 'CUDAExecutionProvider' in o.get_available_providers(), 'OCR WITHOUT THE CARD'"

Quick start

pip install "mdcx[convert]"

mdcx-convert --input ./Documents --output ./Documents_md
mdcx pack --output ./Documents_md --target corpus.mdcx --key "passphrase"
mdcx search corpus.mdcx "where is the storage temperature stated" --key "passphrase"

Conversion

mdcx-convert --input ./Documents --output ./Documents_md

The output mirrors the input directory structure, adds a global index, and records for each file the coverage achieved against its original.

Supported formats

PDF, EPUB, Word, Excel, PowerPoint, HTML, Markdown, CSV and plain text.

The format of a file is determined from its first bytes rather than from its extension. Repositories are known to serve EPUB files from URLs ending in .pdf and declaring application/pdf, where only the content identifies the format correctly. Routing such a file by extension sends it to a reader that cannot open it, and the resulting failure is indistinguishable from a damaged document.

Plain text carries no signature, so its extension determines the format. A file whose content identifies no known format is skipped rather than assumed.

How much of the machine it uses