densely

Lossless context compression for LLMs. Pack any text into 2x–8x fewer tokens with guaranteed byte-exact reconstruction — verified by sha256 on every decompress.

MIT, free forever. (A hosted tier is coming for teams — waitlist.)

An o200k token can carry up to ~17.6 bits of information, but typical code occupies tokens at only ~5–6 bits each. densely reclaims the difference:

text -> lzma -> 16-bit chunks -> 65,536 single-token English words

Each carrier word (" the", " of", …) costs exactly 1 token — the o200k pre-tokenizer never merges across word boundaries — so every token in the payload carries 2 bytes of compressed data (16 of the ~17.6 theoretically available bits, 91% of channel capacity).

Benchmarks

Reproduce with python3 bench.py (fixed seeds, stdlib code sample):

The neural backend (--backend neural, python3 bench.py --neural) drives an integer arithmetic coder with next-token probabilities from Qwen2.5-Coder-0.5B, NNCP-style batched across segments. The 88.3% figure benefits from the model having seen Python's stdlib during training; the 86.3% row is this repo's own sources — code that did not exist before 2026-08-08 — and is the honest number for novel code (~0.56 bit/byte).

Measured on real-world data

Same pipeline pointed at data we didn't curate: this repo's own sources, an installed pip module, a slice of a real macOS /var/log/install.log, a real package-lock.json, and the live npm registry response for express. o200k counted locally with tiktoken; Claude counted via Anthropic's own count_tokens endpoint (Sonnet 5, request overhead calibrated out), using the claude1 alphabet. Every row round-tripped byte-exact.

For comparison, Headroom reports 15–20% savings for coding agents and 60–95% on JSON — achieved by dropping content from context, with originals kept in a local cache with a TTL. densely keeps the full data in the context itself, restorable byte-for-byte with no external storage and no expiry.

Lossless compression below the entropy of the data is mathematically impossible (Shannon; see also Fundamental Limits of Prompt Compression) — within that bound, densely sits near the practical ceiling for a deterministic, CPU-only method.

Usage

pip install "densely[mcp]"
# extras: [neural] for the neural backend (torch + transformers)

densely compress  big_context.txt -o payload.dense
densely compress  src.py -o payload.dense --backend neural
densely decompress payload.dense   -o restored.txt   # byte-identical
densely stats     file1.py file2.json                # token savings

Library:

from densely import compress, decompress

payload = compress(text)        # ~2x-8x fewer tokens
assert decompress(payload) == text  # always true, sha256-checked

compress via the CLI self-verifies the round trip before writing output; decompress raises ValueError on any corruption or hash mismatch.

Use it in Claude Code / Cursor (MCP)

# Claude Code
claude mcp add --scope user densely -- "$(which densely-mcp)"

# Cursor (~/.cursor/mcp.json) and other MCP clients
{"mcpServers": {"densely": {"command": "/absolute/path/to/densely-mcp"}}}

Three tools:

• compress_file(path) — agent calls this instead of reading a large log/JSON/dump: gets a preview + dense payload at 2x-8x fewer tokens.
• compress_text(text) — same for a big tool output already in hand.
• expand(payload | payload_file, start_line, end_line) — exact original back, sha256-verified; line ranges let the agent pay only for the slice it needs.

Every compression also writes a .dense sidecar file next to the original — a plain text file you can commit, ship, or archive; no cache, no TTL, nothing to expire. Small payloads are additionally returned inline. This matters for context compaction: a compaction summary replaces old conversation content, so an inline payload alone could be summarized away — but the sidecar on disk (and its path, which carries into summaries) cannot. If you use the Anthropic compaction API beta directly, add this to your instructions so payload references survive verbatim:

Preserve any densely payload references (paths ending in .dense, or lines starting with DENSE1/DENSE2) verbatim in the summary. Do not call any tools while writing this summary.

Automatic mode (hook)

Zero-effort savings: a PostToolUse hook compresses every large tool output (Bash output, Read of non-code files, >= 5,000 tokens) on the fly — the agent sees a preview + stats, exact content stays available via search/expand, and every replacement is recorded in a savings ledger (stats tool, or the running total shown in each replacement).

Add to ~/.claude/settings.json (absolute paths — hooks run outside your shell PATH):

{
  "hooks": {
    "PostToolUse": [{
      "matcher": "Read|Bash",
      "hooks": [{
        "type": "command",
        "command": "/absolute/path/to/densely-hook",
        "timeout": 30
      }]
    }]
  }
}

(which densely-hook after install gives the absolute path — hooks run outside your shell PATH, so absolute paths only.)

Tune with DENSELY_HOOK_MIN_TOKENS (default 5000). Code files the agent is editing are never touched.

When it saves tokens (and when it doesn't)

Agent sessions resend the whole history to the API on every turn, so anything sitting in context is paid for again and again. What densely does to each kind of content:

Note on prompt caching: cached history is cheaper, but the context window stays the same size — densely primarily buys you room, then money.

vs. the field

The main players make different trade-offs (numbers from each project's own published benchmarks):

Different philosophies: maximum savings with silent degradation vs. exactness or nothing. Readable-lossy tools win on content the model must keep reading; densely wins when the data must never be wrong and must survive compaction, export, and machine moves. Use both.

The honest caveats