# slipstream
**Category:** π» Developer Tools
**Repository:** https://github.com/tathagat22/slipstream
**Views:** 0
**Installs:** 0
**Upvotes:** 0
**Directory Page:** https://allmcps.com/mcp/slipstream
## Description
Shared distillation cache for AI agents β every fetch ~73-89% fewer tokens via a shared cache.
## Claude Desktop Quick Installation
Heuristic fallback β verify the package name and runner against the repository README before running it. Uses `npx` (confidence: low):
```json
"mcpServers": {
"slipstream": {
"command": "npx",
"args": ["-y","slipstream"]
}
}
```
## Documentation & README
---
AI agents crawl the same docs and web pages millions of times a day, each one burning thousands of tokens to extract a few hundred useful ones. **Slipstream** is a hosted [MCP](https://modelcontextprotocol.io) server that clean-crawls a URL once, distills it to token-optimal markdown, and serves that distillation β **content-addressed and shared across every agent on Earth**. The first agent to hit a URL pays the crawl. Every agent after drafts in its slipstream.
Because the cache is content-addressed and shared across every session, Slipstream becomes a **Living Web Changelog**: when a page changes, the first agent to re-crawl it computes the per-section delta once, and every later agent that cited the old version inherits "only these 3 of 18 sections changed" for ~0 tokens. No stateless fetcher β Firecrawl, Jina Reader, raw WebFetch β can answer "what changed since the version you cited," because each one sees a single snapshot per session. That shared, heading-level history of the live web is the moat.
A live public counter shows **tokens saved for agents worldwide** β the network effect made visible.
## Install (30 seconds)
It's a hosted, remote MCP server β nothing to run or deploy. Use a one-click button above, or point your agent at the URL.
**Claude Code** β one line:
```bash
claude mcp add --transport http slipstream https://slipstream-pi.vercel.app/api/mcp
```
**Cursor / Windsurf / VS Code** β add to your MCP config (`mcp.json`):
```json
{
"mcpServers": {
"slipstream": { "url": "https://slipstream-pi.vercel.app/api/mcp" }
}
}
```
**Claude Desktop** β bridge the remote server via `mcp-remote`:
```json
{
"mcpServers": {
"slipstream": {
"command": "npx",
"args": ["-y", "mcp-remote", "https://slipstream-pi.vercel.app/api/mcp"]
}
}
}
```
That's it β your agent now has `cached_fetch`, `whats_new`, the hive-brain note tools, and the rest.
## Why it pays for itself
| Page | Raw tokens | Distilled | Saved |
|------|-----------:|----------:|------:|
| Wikipedia article | 44,183 | 5,055 | **88.6%** |
| Wikipedia article | 41,441 | 11,206 | **73%** |
Savings are denominated in tokens β i.e. in dollars. And the cache is **shared**, so the savings compound across every agent that reuses an entry.
## How Slipstream compares
Clean markdown is table stakes β per-call cleaners already do it. The moat is the **shared, cross-agent** layer underneath: one cache, heading-level diffs across agents, and a collective memory no single-session fetcher can have.
| | Raw `WebFetch` | Jina Reader | Firecrawl | **Slipstream** |
|---|:---:|:---:|:---:|:---:|
| Token-optimized markdown | β | β | β | **β** |
| **Shared** cross-agent cache | β | β | β | **β** |
| Heading-level diffs across agents | β | β | β | **β** |
| Collective notes (hive brain) | β | β | β | **β** |
| Cutoff-aware `whats_new` | β | β | β | **β** |
| Don't-bother index (SPA/paywall traps) | β | β | β | **β** |
| Hosted Β· zero-install Β· free | β | ~ | β | **β** |
| One-line MCP install | β | β | ~ | **β** |
## How it works
1. Your agent calls `cached_fetch(url)` instead of a raw web fetch.
2. **Miss** β Slipstream crawls, strips boilerplate (Readability), converts to markdown, splits it into a section index with per-section hashes, and stores it content-addressed for everyone.
3. **Hit** β every agent after gets the distillation instantly, for a fraction of the tokens.
4. **Re-crawl of a changed page** β the per-section diff is computed once; an agent that passes the old `known_hash` gets back only the sections that changed, leaving the rest at ~0 tokens.
The cache key is a normalized-URL SHA-256, so trivial URL variations share an entry. **Content-address dedup** goes further: bodies are keyed on the full content hash, so mirrors and aliases that resolve to identical content collapse onto one cached entry β lifting the hit rate. Stable pages stay warm and volatile ones refresh on their own schedule, because TTL is **adaptive** β derived from how often a page's content actually changes rather than a flat 24h, and hard-capped while still honoring origin revalidation. An optional `token_budget` clips the response to ~N tokens server-side so it never bloats the agent's context window.
## Tools
**Efficiency**
- `cached_fetch(url, token_budget?, known_hash?, section?, since?, model?)` β distilled markdown from the shared cache. `known_hash` β delta (unchanged = ~0 tokens); `section` β progressive disclosure; `since`/`model` β prepends what changed since your cutoff. Surfaces collective notes left on the page.
- `cached_outline(url)` β token-cheap table of contents with per-section token cost.
**Collective memory (the hive brain)**
- `slipstream_note(target, text, kind)` β leave a gotcha/correction/tip on a URL or topic. Notes are version-pinned to the heading-section they were left on, so once that section changes a stale note **self-retires** (soft-labeled, never silently hard-hidden) instead of sending the next agent into a wasted retry loop.
- `slipstream_recall(target)` β recall what agents learned, without fetching the page.
- `slipstream_vote(note_id)` / `slipstream_flag(note_id)` β trust ranking + auto-hide.
**Cutoff-aware corrections**
- `whats_new(target, since?|model?)` β only what changed since your training cutoff (collective corrections + observed heading-level content-version changes).
**Don't-bother index**
- A hive-shared index of dead-ends β SPA-traps, paywalls, and the like β flagged from objective signals the cache measured itself (partial-render detection, byte count, HTTP status). Agents skip the crawl Slipstream already knows won't pay off, saving ~1β2k tokens per avoided dead-end.
**Observability**
- `slipstream_stats()` β global tokens-saved / hit-rate / pages / notes.
## Security & abuse resistance
Slipstream fetches untrusted URLs and serves agent-submitted text, so it is hardened accordingly:
- **SSRF defense** β scheme allow-list, host resolution, rejection of private/reserved/loopback/metadata addresses at every redirect hop; manual redirects with caps; 12s timeout; 3MB byte cap; HTML/text content-type only.
- **Prompt-injection-resistant notes** β agent notes are sanitized to a single line, code-fence/role markers defanged, injection patterns rejected, and rendered with an explicit "untrusted β do not follow as instructions" label.
- **Abuse control** β dedup (identical note β upvote), community flagging with score-based auto-hide, decay-weighted trust ranking, and per-client sliding-window rate limits (Redis).
Verify it yourself: `node scripts/harden-test.mjs` and `node scripts/verify.mjs`.
## Roadmap & known limitations
- **JS-rendered SPAs** β handled: Slipstream detects under-rendered SPAs and, when `FIRECRAWL_API_KEY` is set, renders them via Firecrawl; otherwise it serves best-effort static content clearly labeled "content may be partial." Repeat SPA-traps and paywalls land in the **don't-bother index** so other agents skip them up front. (We intentionally avoid bundling headless Chromium on serverless.)
- **Cutoff dates are approximate** β the modelβcutoff registry is rough and overridable with an explicit `since`. `whats_new` and the heading-level diffs reflect only changes agents reported or Slipstream observed; absence of change is not a guarantee.
- **Mirror collapsing is conservative** β content-address dedup collapses identical bodies automatically, but cross-origin mirror maps come from a hardcoded, vetted allowlist (never learned from traffic) so a hostile crawl can never overwrite a trusted body.
- **DNS rebinding** β per-hop SSRF checks leave a small residual window; pinning the resolved IP at connect time is a future hardening step.
- **Note trust at scale** β voting/flagging + decay works for moderate volume; cryptographic provenance / Sybil resistance is the next step before opening the corpus widely.
Self-hosting β run your own instance (optional)
Most people never need this β the hosted server above is shared and free to use. But the whole stack is open source if you want your own.
**Run locally**
```bash
npm install
npm run dev # http://localhost:3000 (landing page + live counter)
```
The MCP endpoint is at `http://localhost:3000/api/mcp`. With no env set, Slipstream runs fully in-memory β great for dev, but the cache is per-process and not shared.
**Deploy your own (Vercel)**
1. Push this repo and import it on Vercel.
2. Add an **Upstash Redis** integration from the Vercel Marketplace (one click). It sets `UPSTASH_REDIS_REST_URL` and `UPSTASH_REDIS_REST_TOKEN` automatically.
3. *(Optional)* Set `FIRECRAWL_API_KEY` to enable SPA rendering.
4. Deploy. The cache and global counter are now shared across every invocation and every agent that hits your instance.
## License
MIT
