# superfetch [Health: Active]
**Category:** ๐ป Developer Tools
**Repository:** https://github.com/j0hanz/super-fetch-mcp-server
**GitHub Stars:** 0
**Views:** 0
**Installs:** 0
**Upvotes:** 0
**Directory Page:** https://allmcps.com/mcp/superfetch
## Description
Intelligent web content fetcher MCP server that converts HTML to clean, AI-readable JSONL format
## Claude Desktop Quick Installation
Install path detected from listing signals. Uses `npx` (confidence: high):
```json
"mcpServers": {
"superfetch": {
"command": "npx",
"args": ["-y","args"]
}
}
```
## Documentation & README
# Fetch URL MCP Server
[](https://www.npmjs.com/package/%40j0hanz%2Ffetch-url-mcp) [](#contributing-and-license)
[](https://insiders.vscode.dev/redirect/mcp/install?name=fetch-url&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D) [](https://insiders.vscode.dev/redirect/mcp/install?name=fetch-url&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D&quality=insiders) [](https://vs-open.link/mcp-install?%7B%22fetch-url-mcp%22%3A%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D%7D)
[](https://lmstudio.ai/install-mcp?name=fetch-url&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBqMGhhbnovZmV0Y2gtdXJsLW1jcEBsYXRlc3QiXX0%3D) [](https://cursor.com/en/install-mcp?name=fetch-url&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBqMGhhbnovZmV0Y2gtdXJsLW1jcEBsYXRlc3QiXX0%3D)
An MCP server that fetches web pages and converts them to clean, readable Markdown.
## Overview
This server takes a URL, fetches the page, and strips away everything you don't need โ navigation, sidebars, banners, scripts โ leaving just the main content as Markdown. It's perfect for feeding into LLMs, giving them the distilled essence of a page without the noise. It also recognizes GitHub, GitLab, Bitbucket, and Gist URLs and rewrites them to fetch the raw content directly.
By default it runs over stdio. Pass `--http` if you need a proper HTTP endpoint with auth, rate limiting, TLS, and session support.
## Key Features
- **HTML to Markdown** โ Turns any public web page into clean, readable Markdown with metadata like `title`, `url`, `contentSize`, and `truncated`.
- **Smart URL handling** โ Recognizes GitHub, GitLab, Bitbucket, and Gist page URLs and rewrites them to raw-content endpoints before fetching.
- **Task mode** โ Big or slow pages can run as async MCP tasks with progress updates, instead of blocking. In HTTP mode, tasks are bound to the authenticated caller rather than a single MCP session, so they can be resumed after reconnecting as the same authenticated subject. Polling task state exposes task summaries; numeric progress remains out-of-band via `notifications/progress`.
- **Self-documenting** โ Includes an `internal://instructions` resource and a `get-help` prompt so clients know how to use it.
- **HTTP mode** โ Optionally serves over Streamable HTTP with host/origin validation, bearer or OAuth auth, rate limiting, health checks, and TLS.
## Web Client
A browser-based client is available if you want to use the server without any MCP setup.
**[Live app](https://fetch-url-client.vercel.app)** ยท [Source code](https://github.com/j0hanz/fetch-url)
## Requirements
- **Node.js** >= 24
- **Docker** (optional) โ only needed if you want to run the container image
## Quick Start
Add this to your MCP client config:
```json
{
"mcpServers": {
"fetch-url-mcp": {
"command": "npx",
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
}
}
}
```
## Client Configuration
Install in VS Code
[](https://insiders.vscode.dev/redirect/mcp/install?name=fetch-url&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D)
Add to `.vscode/mcp.json`:
```json
{
"servers": {
"fetch-url-mcp": {
"type": "stdio",
"command": "npx",
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
}
}
}
```
Or install via CLI:
```sh
code --add-mcp '{"name":"fetch-url-mcp","command":"npx","args":["-y","@j0hanz/fetch-url-mcp@latest"]}'
```
For more info, see [VS Code MCP docs](https://code.visualstudio.com/docs/copilot/chat/mcp-servers).
Install in VS Code Insiders
[](https://insiders.vscode.dev/redirect/mcp/install?name=fetch-url&config=%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D&quality=insiders)
Add to `.vscode/mcp.json`:
```json
{
"servers": {
"fetch-url-mcp": {
"type": "stdio",
"command": "npx",
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
}
}
}
```
Or install via CLI:
```sh
code-insiders --add-mcp '{"name":"fetch-url-mcp","command":"npx","args":["-y","@j0hanz/fetch-url-mcp@latest"]}'
```
For more info, see [VS Code Insiders MCP docs](https://code.visualstudio.com/docs/copilot/chat/mcp-servers).
Install in Cursor
[](https://cursor.com/en/install-mcp?name=fetch-url&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBqMGhhbnovZmV0Y2gtdXJsLW1jcEBsYXRlc3QiXX0%3D)
Add to `~/.cursor/mcp.json`:
```json
{
"mcpServers": {
"fetch-url-mcp": {
"command": "npx",
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
}
}
}
```
For more info, see [Cursor MCP docs](https://docs.cursor.com/context/model-context-protocol).
Install in Visual Studio
[](https://vs-open.link/mcp-install?%7B%22fetch-url-mcp%22%3A%7B%22command%22%3A%22npx%22%2C%22args%22%3A%5B%22-y%22%2C%22%40j0hanz%2Ffetch-url-mcp%40latest%22%5D%7D%7D)
For solution-scoped setup, add this to `.mcp.json` at the solution root:
```json
{
"servers": {
"fetch-url-mcp": {
"type": "stdio",
"command": "npx",
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
}
}
}
```
For more info, see [Visual Studio MCP docs](https://learn.microsoft.com/en-us/visualstudio/ide/mcp-servers).
Install in Goose
Add to `~/.config/goose/config.yaml` on macOS/Linux or `%APPDATA%\Block\goose\config\config.yaml` on Windows:
```yaml
extensions:
fetch-url-mcp:
name: fetch-url-mcp
cmd: npx
args: ['-y', '@j0hanz/fetch-url-mcp@latest']
enabled: true
type: stdio
timeout: 300
```
For more info, see [Goose extension docs](https://block.github.io/goose/docs/getting-started/using-extensions/).
Install in LM Studio
[](https://lmstudio.ai/install-mcp?name=fetch-url&config=eyJjb21tYW5kIjoibnB4IiwiYXJncyI6WyIteSIsIkBqMGhhbnovZmV0Y2gtdXJsLW1jcEBsYXRlc3QiXX0%3D)
Add to `~/.lmstudio/mcp.json` on macOS/Linux or `%USERPROFILE%/.lmstudio/mcp.json` on Windows:
```json
{
"mcpServers": {
"fetch-url-mcp": {
"command": "npx",
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
}
}
}
```
For more info, see [LM Studio MCP docs](https://lmstudio.ai/docs/basics/mcp).
Install in Claude Desktop
Add to `claude_desktop_config.json`:
```json
{
"mcpServers": {
"fetch-url-mcp": {
"command": "npx",
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
}
}
}
```
For more info, see [Claude Desktop MCP docs](https://modelcontextprotocol.io/quickstart/user).
Install in Claude Code
Use the CLI:
```sh
claude mcp add fetch-url-mcp -- npx -y @j0hanz/fetch-url-mcp@latest
```
For project-scoped config, Claude Code writes `.mcp.json` with:
```json
{
"mcpServers": {
"fetch-url-mcp": {
"command": "npx",
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"],
"env": {}
}
}
}
```
For more info, see [Claude Code MCP docs](https://docs.anthropic.com/en/docs/claude-code/mcp).
Install in Windsurf
Add to `~/.codeium/windsurf/mcp_config.json`:
```json
{
"mcpServers": {
"fetch-url-mcp": {
"command": "npx",
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
}
}
}
```
For more info, see [Windsurf MCP docs](https://docs.windsurf.com/windsurf/cascade/mcp).
Install in Amp
Add to `~/.config/amp/settings.json` on macOS/Linux, `%USERPROFILE%\.config\amp\settings.json` on Windows, or `.amp/settings.json` for workspace-scoped config:
```json
{
"amp.mcpServers": {
"fetch-url-mcp": {
"command": "npx",
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
}
}
}
```
Or install via CLI:
```sh
amp mcp add fetch-url-mcp -- npx -y @j0hanz/fetch-url-mcp@latest
```
For more info, see [Amp docs](https://ampcode.com/manual).
Install in Cline
Open the MCP Servers panel, choose `Configure MCP Servers`, and add this to `cline_mcp_settings.json`:
```json
{
"mcpServers": {
"fetch-url-mcp": {
"command": "npx",
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
}
}
}
```
For more info, see [Cline MCP docs](https://docs.cline.bot/mcp/configuring-mcp-servers).
Install in Codex CLI
Use the CLI:
```sh
codex mcp add fetch-url-mcp -- npx -y @j0hanz/fetch-url-mcp@latest
```
Or add this to `~/.codex/config.toml` or project-scoped `.codex/config.toml`:
```toml
[mcp_servers.fetch-url-mcp]
command = "npx"
args = ["-y", "@j0hanz/fetch-url-mcp@latest"]
```
For more info, see [Codex MCP docs](https://developers.openai.com/codex/mcp/).
Install in GitHub Copilot
Add to `.vscode/mcp.json`:
```json
{
"servers": {
"fetch-url-mcp": {
"type": "stdio",
"command": "npx",
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
}
}
}
```
For more info, see [GitHub Copilot MCP docs](https://code.visualstudio.com/docs/copilot/chat/mcp-servers).
Install in Warp
Open `Personal > MCP Servers` in Warp, choose `+ Add`, and either add a CLI server with:
- `command`: `npx`
- `args`: `["-y", "@j0hanz/fetch-url-mcp@latest"]`
Or paste this JSON snippet when using Warp's multi-server import flow:
```json
{
"mcpServers": {
"fetch-url-mcp": {
"command": "npx",
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
}
}
}
```
For more info, see [Warp MCP docs](https://docs.warp.dev/features/warp-ai/mcp).
Install in Kiro
Use Kiro's MCP Servers panel or the `Add to Kiro` install flow. Kiro stores workspace-scoped MCP config in `.kiro/settings/mcp.json` and user-scoped config in `~/.kiro/settings/mcp.json`.
For this server, use:
- `command`: `npx`
- `args`: `["-y", "@j0hanz/fetch-url-mcp@latest"]`
For more info, see [Kiro MCP docs](https://kiro.dev/blog/unlock-your-development-productivity-with-kiro-and-mcp/).
Install in Gemini CLI
Add to `~/.gemini/settings.json`:
```json
{
"mcpServers": {
"fetch-url-mcp": {
"command": "npx",
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
}
}
}
```
For more info, see [Gemini CLI MCP docs](https://google-gemini.github.io/gemini-cli/docs/tools/mcp-server.html).
Install in Zed
Add to `~/.config/zed/settings.json`:
```json
{
"context_servers": {
"fetch-url-mcp": {
"command": "npx",
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"],
"env": {}
}
}
}
```
For more info, see [Zed MCP docs](https://zed.dev/docs/ai/mcp).
Install in Augment
Use the Augment Settings panel and either add the server manually or choose `Import from JSON`:
```json
{
"mcpServers": {
"fetch-url-mcp": {
"command": "npx",
"args": ["-y", "@j0hanz/fetch-url-mcp@latest"]
}
}
}
```
For more info, see [Augment MCP docs](https://docs.augmentcode.com/setup-augment/mcp).
Install in Roo Code
Use Roo Code's MCP Servers UI or marketplace flow.
For this server, use:
- `command`: `npx`
- `args`: `["-y", "@j0hanz/fetch-url-mcp@latest"]`
For more info, see [Roo Code docs](https://docs.roocode.com/).
Install in Kilo Code
Use Kilo Code's MCP Servers UI or marketplace flow.
For this server, use:
- `command`: `npx`
- `args`: `["-y", "@j0hanz/fetch-url-mcp@latest"]`
For more info, see [Kilo Code docs](https://kilocode.ai/docs).
## Use Cases
- **Documentation for LLMs** โ Grab a docs page, blog post, or reference article as Markdown and pass it straight into a context window.
- **Repository content** โ Hand it a GitHub, GitLab, or Bitbucket URL and it resolves the raw content endpoint. Works with Gists too.
- **Slow or large pages** โ Task mode lets big fetches run in the background while sending monotonic progress updates back to the client, while `tasks/get` exposes the latest task summary fields such as `statusMessage`, `createdAt`, `lastUpdatedAt`, `ttl`, and `pollInterval`.
## Architecture
```text
[MCP Client]
โโ stdio -> `src/index.ts` -> `startStdioServer()` -> `createMcpServer()`
โโ HTTP (`--http`) -> `src/index.ts` -> `startHttpServer()` -> HTTP dispatcher
โโ `GET /health`
โโ `GET /.well-known/oauth-protected-resource`
โโ `GET /.well-known/oauth-protected-resource/mcp`
โโ `POST|GET|DELETE /mcp`
`createMcpServer()`
โโ registers tool: `fetch-url`
โโ registers prompt: `get-help`
โโ registers resources:
โ - `internal://instructions`
โโ enables capabilities: logging, resources, prompts, tasks
โโ installs task handlers, log-level handling, and shutdown cleanup
`fetch-url` execution
โโ validate input with `fetchUrlInputSchema`
โโ normalize URL and block local/private targets unless allowed
โโ rewrite supported code-host URLs to raw endpoints when possible
โโ fetch content via the shared pipeline
โโ transform HTML into Markdown in the transform worker path
โโ validate `structuredContent` with `fetchUrlOutputSchema`
```
### Request Lifecycle
```text
[Client] -- initialize {protocolVersion, capabilities} --> [Server]
[Server] -- {protocolVersion, capabilities, serverInfo} --> [Client]
[Client] -- notifications/initialized --> [Server]
[Client] -- tools/call {name, arguments} --> [Server]
[Server] -- {content: [{type, text}], structuredContent?, isError?} --> [Client]
```
## MCP Surface
### Tools
#### `fetch-url`
Takes a URL and returns Markdown. Read-only, with no JavaScript execution. Supports running as a background MCP task for large or slow pages. In task mode, `tasks/get` and `tasks/list` expose task summaries including `status`, `statusMessage`, `createdAt`, `lastUpdatedAt`, `ttl`, and `pollInterval`; numeric progress remains out-of-band via `notifications/progress`.
| Parameter | Type | Required | Description |
| --------- | -------- | -------- | --------------------------- |
| `url` | `string` | yes | Target URL. Max 2048 chars. |
You get text content back by default. If output validation passes, the response also includes `structuredContent` with typed fields: `url`, `resolvedUrl`, `finalUrl`, `title`, `metadata`, `markdown`, `fetchedAt`, `contentSize`, and `truncated`. A `true` value for `truncated` means the content hit a server-side size limit.
To opt into progress updates, include `_meta.progressToken` in the tool call. The token may be a string or number, and the server may emit monotonic `notifications/progress` updates while the fetch runs.
To run the tool in task mode, include `params.task = { ttl?: }`. `tasks/result` waits until the task reaches a terminal status and then returns the stored output or a terminal error payload for cancelled or failed tasks. Task summaries and final results include `_meta["io.modelcontextprotocol/related-task"] = { "taskId": "" }`.
```json
{
"method": "tools/call",
"params": {
"name": "fetch-url",
"arguments": {
"url": "https://example.com/docs"
},
"task": {
"ttl": 300000,
"pollInterval": 1000
},
"_meta": {
"progressToken": 7
}
}
}
```
```text
1. [Client] -- tools/call {name: "fetch-url", arguments} --> [Server]
2. [Server] -- dispatch("fetch-url") --> [src/tools/fetch-url.ts]
3. [Handler] -- validate(fetchUrlInputSchema) --> normalize / fetch / transform
4. [Handler] -- validate(fetchUrlOutputSchema) --> assemble content + structuredContent
5. [Server] -- result or tool error --> [Client]
```
### Resources
| Resource | URI | MIME Type | Description |
| ---------------------------- | ------------------------- | --------------- | -------------------------------------------- |
| `fetch-url-mcp-instructions` | `internal://instructions` | `text/markdown` | Guidance for using the Fetch URL MCP server. |
### Prompts
| Prompt | Arguments | Description |
| ---------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `get-help` | `topic?` | Return Fetch URL server instructions: workflows, task mode, and error handling. Optional values: `capabilities`, `workflows`, `constraints`, `errors`. |
## MCP Capabilities
| Capability | Status | Notes |
| ---------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| completions | confirmed | Advertised in `createServerCapabilities()`. |
| logging | confirmed | Advertised in `createServerCapabilities()`. |
| resources | confirmed | Static instruction resource is registered during server startup. Subscription and list-changed support are not advertised. |
| prompts | confirmed | `get-help` is registered during server startup. |
| tasks | confirmed | Advertised in `createServerCapabilities()` and backed by registered task handlers plus optional tool task support. |
| progress notifications | confirmed | Opt-in via `_meta.progressToken`. Tool execution reports monotonic `notifications/progress` updates during fetch and transform stages. |
| task status updates | confirmed | `notifications/tasks/status` is emitted when task status changes and `TASKS_STATUS_NOTIFICATIONS=true`. |
### Tool Annotations
| Annotation | Value |
| ----------------- | ------- |
| `readOnlyHint` | `true` |
| `destructiveHint` | `false` |
| `idempotentHint` | `true` |
| `openWorldHint` | `true` |
### Structured Output
The tool declares an `outputSchema` and includes `structuredContent` in the response when validation passes. Clients that support structured output get typed data directly; the rest use the text fallback.
## Configuration
All configuration is through environment variables. For basic stdio usage, nothing needs to be set.
### HTTP Server
| Variable | Default | Notes |
| ------------------------------------- | ----------- | ---------------------------------------------------------------------- |
| `HOST` | `127.0.0.1` | Bind address. Non-loopback bindings also require `ALLOW_REMOTE=true`. |
| `PORT` | `3000` | Listening port for `--http`. |
| `ALLOW_REMOTE` | `false` | Must be enabled to bind to a non-loopback interface. |
| `ALLOWED_HOSTS` | empty | Additional allowed `Host` and `Origin` values. |
| `SERVER_MAX_CONNECTIONS` | `0` | Optional connection cap. |
| `SERVER_TRUST_PROXY` | `false` | Trust `X-Forwarded-For` / `Forwarded` for client IP resolution. |
| `SERVER_HEADERS_TIMEOUT_MS` | unset | Optional Node server tuning. |
| `SERVER_REQUEST_TIMEOUT_MS` | unset | Optional Node server tuning. |
| `SERVER_KEEP_ALIVE_TIMEOUT_MS` | unset | Optional keep-alive tuning. |
| `SERVER_KEEP_ALIVE_TIMEOUT_BUFFER_MS` | unset | Optional keep-alive tuning buffer. |
| `SERVER_MAX_HEADERS_COUNT` | unset | Optional header count limit. |
| `SERVER_BLOCK_PRIVATE_CONNECTIONS` | `false` | Enables inbound private-network protections when not trusting a proxy. |
### Authentication & OAuth
| Variable | Default | Notes |
| ------------------------- | ------- | ----------------------------------------------------------- |
| `ACCESS_TOKENS` | unset | Comma- or space-separated static bearer tokens. |
| `API_KEY` | unset | Alternate static token source for header auth. |
| `OAUTH_ISSUER_URL` | unset | Enables OAuth mode when combined with the other OAuth URLs. |
| `OAUTH_AUTHORIZATION_URL` | unset | Optional explicit authorization endpoint. |
| `OAUTH_TOKEN_URL` | unset | Optional explicit token endpoint. |
| `OAUTH_REVOCATION_URL` | unset | Optional OAuth revocation endpoint. |
| `OAUTH_REGISTRATION_URL` | unset | Optional OAuth dynamic client registration endpoint. |
| `OAUTH_INTROSPECTION_URL` | unset | Required for OAuth token introspection. |
| `OAUTH_REQUIRED_SCOPES` | empty | Required scopes enforced after auth. |
| `OAUTH_CLIENT_ID` | unset | Optional introspection client ID. |
| `OAUTH_CLIENT_SECRET` | unset | Optional introspection client secret. |
### TLS
| Variable | Default | Notes |
| ---------------------- | ------- | ----------------------------------------------------------- |
| `SERVER_TLS_KEY_FILE` | unset | Enable HTTPS when set together with `SERVER_TLS_CERT_FILE`. |
| `SERVER_TLS_CERT_FILE` | unset | TLS certificate path. |
| `SERVER_TLS_CA_FILE` | unset | Optional custom CA bundle. |
### Fetching
| Variable | Default | Notes |
| ------------------- | ------------------------- | -------------------------------------------------- |
| `ALLOW_LOCAL_FETCH` | `false` | Allows loopback and private-network fetch targets. |
| `FETCH_TIMEOUT_MS` | `15000` | Network fetch timeout in milliseconds. |
| `USER_AGENT` | `fetch-url-mcp/` | Override the outbound user agent string. |
### Tool Output
| Variable | Default | Notes |
| -------------------------- | ------- | ---------------------------------------------- |
| `MAX_INLINE_CONTENT_CHARS` | `0` | `0` means no explicit inline truncation limit. |
### Tasks
| Variable | Default | Notes |
| ---------------------------- | ------- | ------------------------------------------------------------------------------------ |
| `TASKS_MAX_TOTAL` | `5000` | Total retained task capacity, including completed/cancelled tasks until they expire. |
| `TASKS_MAX_PER_OWNER` | `1000` | Per-owner retained task cap, clamped to the total cap. |
| `TASKS_STATUS_NOTIFICATIONS` | `false` | Enables status notifications for tasks. |
| `TASKS_REQUIRE_INTERCEPTION` | `true` | Requires interception for task-capable tool execution. |
### Transform Workers
| Variable | Default | Notes |
| ------------------------------------------ | --------- | ------------------------------------- |
| `TRANSFORM_CANCEL_ACK_TIMEOUT_MS` | `200` | Cancellation acknowledgement timeout. |
| `TRANSFORM_WORKER_MODE` | `threads` | Worker execution mode. |
| `TRANSFORM_WORKER_MAX_OLD_GENERATION_MB` | unset | Optional worker memory limit. |
| `TRANSFORM_WORKER_MAX_YOUNG_GENERATION_MB` | unset | Optional worker memory limit. |
| `TRANSFORM_WORKER_CODE_RANGE_MB` | unset | Optional worker memory limit. |
| `TRANSFORM_WORKER_STACK_MB` | unset | Optional worker stack size. |
### Content Cleanup
| Variable | Default | Notes |
| ------------------------------------- | -------------- | ------------------------------------------ |
| `FETCH_URL_MCP_EXTRA_NOISE_TOKENS` | empty | Extra noise-removal tokens. |
| `FETCH_URL_MCP_EXTRA_NOISE_SELECTORS` | empty | Extra DOM selectors for noise removal. |
| `FETCH_URL_MCP_LOCALE` | system default | Locale override for extraction heuristics. |
| `MARKDOWN_HEADING_KEYWORDS` | built-in list | Override heading keywords used by cleanup. |
### Logging
| Variable | Default | Notes |
| ------------ | ------- | ------------------------------------ |
| `LOG_LEVEL` | `info` | `debug`, `info`, `warn`, or `error`. |
| `LOG_FORMAT` | `text` | Set to `json` for structured logs. |
## HTTP Endpoints
| Method | Path | Auth | Purpose |
| -------- | ------------------------------------------- | ------------------------------------------ | ------------------------------------------------------- |
| `GET` | `/health` | no, unless `?verbose=1` on a remote server | Basic health response, with optional diagnostics. |
| `GET` | `/.well-known/oauth-protected-resource` | no | OAuth protected-resource metadata. |
| `GET` | `/.well-known/oauth-protected-resource/mcp` | no | OAuth protected-resource metadata for the MCP endpoint. |
| `POST` | `/mcp` | yes | Session initialization and JSON-RPC requests. |
| `GET` | `/mcp` | yes | Session-bound server-to-client stream handling. |
| `DELETE` | `/mcp` | yes | Session shutdown. |
## Security
| Control | Status | Notes |
| -------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Host and origin validation | implemented | HTTP requests are rejected unless `Host` and `Origin` match the allowlist built from loopback, the configured host, and `ALLOWED_HOSTS`. |
| Authentication | implemented | HTTP mode supports static bearer tokens locally or OAuth token introspection; remote bindings require OAuth. |
| Protocol version checks | implemented | Session-bound MCP HTTP requests validate `MCP-Protocol-Version` and pin it to the negotiated session version. |
| Rate limiting | implemented | Requests pass through the HTTP rate limiter before route dispatch. Enable `SERVER_TRUST_PROXY=true` behind a trusted reverse proxy so limits apply to the forwarded client IP instead of the proxy hop. |
| Outbound SSRF protections | implemented | Local/private IPs, metadata endpoints, and `.local`/`.internal` hosts are blocked unless `ALLOW_LOCAL_FETCH=true`. |
| TLS | optional | HTTPS is enabled when both TLS key and certificate files are configured. |
| Stdio logging safety | implemented | Server logs are written to stderr, not stdout, so stdio MCP traffic stays clean. |
## Development
### Essential Commands
| Command | Description |
| -------------------- | ------------------------------------------------- |
| `npm run build` | Clean, compile TypeScript, copy assets. |
| `npm run dev` | Watch mode TypeScript compilation. |
| `npm run dev:run` | Run the server with `--watch` and `.env` support. |
| `npm start` | Start the compiled server. |
| `npm test` | Run the full test suite. |
| `npm run lint` | Lint with ESLint. |
| `npm run lint:fix` | Auto-fix lint issues. |
| `npm run type-check` | Type-check source and tests. |
| `npm run format` | Format with Prettier. |
| `npm run inspector` | Build and launch MCP Inspector. |
All npm scripts
| Script | Command |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------- |
| `clean` | `node scripts/tasks.mjs clean` |
| `build` | `node scripts/tasks.mjs build` |
| `copy:assets` | `node scripts/tasks.mjs copy:assets` |
| `prepare` | `npm run build` |
| `dev` | `tsc --watch --preserveWatchOutput` |
| `dev:run` | `node --env-file=.env --watch dist/index.js` |
| `start` | `node dist/index.js` |
| `format` | `prettier --write .` |
| `type-check` | `node scripts/tasks.mjs type-check` |
| `type-check:src` | `node node_modules/typescript/bin/tsc -p tsconfig.json --noEmit` |
| `type-check:tests` | `node node_modules/typescript/bin/tsc -p tsconfig.test.json --noEmit` |
| `type-check:diagnostics` | `tsc --noEmit --extendedDiagnostics` |
| `type-check:trace` | `node -e "require('fs').rmSync('.ts-trace',{recursive:true,force:true})" && tsc --noEmit --generateTrace .ts-trace` |
| `lint` | `eslint .` |
| `lint:tests` | `eslint src/__tests__` |
| `lint:fix` | `eslint . --fix` |
| `test` | `node scripts/tasks.mjs test` |
| `test:fast` | `node --test --import tsx/esm src/__tests__/**/*.test.ts node-tests/**/*.test.ts` |
| `test:coverage` | `node scripts/tasks.mjs test --coverage` |
| `knip` | `knip` |
| `knip:fix` | `knip --fix` |
| `inspector` | `npm run build && npx -y @modelcontextprotocol/inspector node dist/index.js --stdio` |
| `prepublishOnly` | `npm run lint && npm run type-check && npm run build` |
## Build and Release
- `npm run prepublishOnly` runs lint, type-check, and build as a single release gate.
- CI workflows are under `.github/workflows/`.
- `Dockerfile` and `docker-compose.yml` are included for containerized runs.
- Published on npm as [`@j0hanz/fetch-url-mcp`](https://www.npmjs.com/package/@j0hanz/fetch-url-mcp).
## Troubleshooting
| Symptom | Likely Cause | Fix |
| --------------------------------------------------- | ----------------------------------- | ----------------------------------------------------------------------------------- |
| Server output mixes with MCP traffic on stdio | Logs going to stdout | Ensure all logging writes to stderr; the server does this by default. |
| HTTP mode returns `403` | Host/origin mismatch | Add the domain to `ALLOWED_HOSTS` or verify loopback bindings. |
| HTTP mode rate limits every request from your proxy | `SERVER_TRUST_PROXY` not enabled | Enable `SERVER_TRUST_PROXY=true` when the server is behind a trusted reverse proxy. |
| HTTP mode returns `401` | Missing or invalid token | Set `ACCESS_TOKENS` or configure OAuth env vars for remote bindings. |
| Fetch returns private-IP error | SSRF protections blocked the target | Set `ALLOW_LOCAL_FETCH=true` if the target is intentionally local. |
| `truncated: true` in response | Content exceeded inline limits | Increase `MAX_INLINE_CONTENT_CHARS` or accept truncated output. |
| Transform timeout or worker crash | Large or complex HTML | Tune `TRANSFORM_WORKER_MAX_OLD_GENERATION_MB` or increase `FETCH_TIMEOUT_MS`. |
| Client config not working | Wrong config format for the client | Check the matching `` block above โ config keys vary by client. |
## Credits
| Dependency | Registry |
| ------------------------------------------------------------------------------------------ | -------- |
| [@modelcontextprotocol/server](https://www.npmjs.com/package/@modelcontextprotocol/server) | npm |
| [@modelcontextprotocol/node](https://www.npmjs.com/package/@modelcontextprotocol/node) | npm |
| [@mozilla/readability](https://www.npmjs.com/package/@mozilla/readability) | npm |
| [linkedom](https://www.npmjs.com/package/linkedom) | npm |
| [node-html-markdown](https://www.npmjs.com/package/node-html-markdown) | npm |
| [undici](https://www.npmjs.com/package/undici) | npm |
| [zod](https://www.npmjs.com/package/zod) | npm |
## Contributing and License
Pull requests welcome. Please make sure these pass before submitting:
1. `npm run lint` and `npm run type-check`
2. `npm test`
3. `npm run format`
## License
MIT License. See [LICENSE](https://github.com/j0hanz/super-fetch-mcp-server/blob/HEAD/LICENSE) for details.