memtrust

Agent memory backends each publish their own benchmark numbers, on different tests, measured different ways. memtrust runs the same evals against all four and publishes the raw logs. Run against the vendors, not by them.

pip install memtrust-cli
memtrust run --backends mempalace,mem0,zep,openviking --eval all

(For contributing to this repo instead of just running it, see Development -- pip install -e ".[dev]" from a clone.)

Contents: Why this exists · What it does · Commands · How this differs · Contradiction detection · Compression fidelity · Temporal-KG boundary · The landscape · Benchmarks · GitHub Actions usage · Self-host · Install · Hosted layer · Backend coverage · Development · FAQ · License · Success stories

Why this exists

If you've compared agent-memory backends recently, you've probably noticed each one leads with a different accuracy number, on a different benchmark, measured a different way. MemPalace's own community already flagged the problem in public. Issue #27 on the MemPalace repository, opened April 7, 2026 and still open, documents that a headline 100% LongMemEval figure, measured with Haiku reranking, wasn't reproducible from the repository's own benchmark scripts and was pulled from the README as unverifiable. A separate 96.6% figure people cite everywhere turns out to be mostly ChromaDB's default embeddings doing the work in raw mode, not MemPalace's own architecture. A "lossless" compression claim (the "AAAK" mode) drops the same LongMemEval score from 96.6% to 84.2% in practice, a 12.4 percentage point gap. Two internal pull requests attempting to fix the reporting problem, #433 and #729, were both closed without merging on April 12, 2026 -- #729 within seven minutes of being opened. As of this writing, the issue has 232 thumbs-up reactions and 39 comments.

None of that means MemPalace, or any other backend, doesn't work. It means nobody outside the vendor had run the same test, the same way, against every option, and published the raw logs.

memtrust does that. It runs LongMemEval, LoCoMo, and a growing set of evals built specifically for this project -- 17 of them as of this writing, all registered in the CLI's --eval flag. The two that matter most for understanding what this project is actually for: contradiction detection, because neither LongMemEval nor LoCoMo tests the question that actually matters once a memory system sits underneath a production agent -- what happens when a new fact contradicts an old one? Does the backend flag the conflict? Silently overwrite the old fact with no audit trail? Serve whichever version it happens to retrieve first? None of the four backends this project tracks publish a number for that. And compression/round-trip fidelity, built to directly test claims like the "lossless" one above: it stores content, retrieves it, and scores literal reconstruction fidelity rather than semantic accuracy, per operating mode a backend exposes (see MemoryBackendAdapter.supported_modes) -- the mechanism that would let a contributor with live MemPalace credentials actually reproduce the 12.4-point compressed-mode accuracy drop mempalace/mempalace#27 documents, instead of just citing it. Neither has been run against a live MemPalace instance as of this writing -- both have, however, been run against a live self-hosted mem0ai install; see "Benchmarks" below. The other evals -- ranking quality, crash recovery, extraction quality, embedding drift, scale/volume stress, lock contention, stats accuracy, orphan cleanup, result consistency, migration rollback, filter injection, resource-sync safety, and temporal-KG boundary detection -- each grew out of a specific real bug report against one of the four tracked backends; see "Success stories" below for the full list.

Where this stands right now, in one place: live benchmark results for one backend (mem0_direct, self-hosted mem0ai), including a real bug this project's own attempt to get those numbers surfaced in mem0's default configuration; and 197 real GitHub issues and PRs filed against MemPalace, Mem0, Zep/Graphiti, and OpenViking independently root-caused against this codebase -- 55 (28%) PASS, 16 (8%) PARTIAL, 42 (21%) a genuine capability gap, 84 (43%) not applicable, every verdict re-verified by a reviewer independent of whoever built the fix.

What it does

Every command below was actually run against this repo, with zero vendor API keys configured, to produce the output shown. Nothing here is simulated.

$ memtrust run --backends mempalace,mem0,zep,openviking --eval all
memtrust 0.3.4 -- run_id=mt_2026-08-04T061759Z
Backends: mempalace, mem0, zep, openviking   Evals: longmemeval, locomo, contradiction,
resource_sync_safety, compression, ranking_quality, scale_stress, embedding_drift, crash_recovery,
extraction_quality, migration_rollback, filter_injection, lock_contention, stats_accuracy,
orphan_cleanup, result_consistency, temporal_kg_boundary

mempalace: SKIPPED (not configured) -- mempalace is not configured: environment variable
MEMPALACE_STORAGE_PATH is not set. Skipping this backend. See docs/methodology.md for setup
instructions.
mem0: SKIPPED (not configured) -- mem0 is not configured: environment variable MEM0_API_KEY is not
set. Skipping this backend. See docs/methodology.md for setup instructions.
zep: SKIPPED (not configured) -- zep is not configured: environment variable ZEP_API_KEY is not set.
Skipping this backend. See docs/methodology.md for setup instructions.
openviking: SKIPPED (not configured) -- openviking is not configured: environment variable
OPENVIKING_API_KEY is not set. Skipping this backend. See docs/methodology.md for setup
instructions.

Cost: $0.00 (no LLM-judged evals ran -- structural evals only, or judge not configured)

Full report: memtrust-report-2026-08-03.json

That's the real, reproducible behavior of a fresh clone with no credentials: every backend reports SKIPPED, the command exits cleanly, and a valid JSON report is still written. memtrust --version now correctly prints the installed version, matching pip show memtrust-cli. Earlier releases printed 0.0.0+unknown even when properly installed, because src/memtrust/__init__.py read importlib.metadata.version("memtrust") while the installed distribution is actually named memtrust-cli -- kept in the FAQ below for the record rather than deleted, since silently erasing a bug the moment it's fixed is exactly the kind of curation this project exists to push back on in other people's benchmarks. Set the relevant environment variable for any backend you want to actually test (MEM0_API_KEY, ZEP_API_KEY, OPENVIKING_API_KEY, MEMPALACE_STORAGE_PATH) and that backend runs for real against its live API instead of being skipped.

The eval logic itself is proven offline, against the bundled synthetic fixtures and, for several adapters, the real installed vendor packages with only the network boundary mocked, by the test suite: