# Atlas Web Compiler & Scraper [Health: Active]

**Category:** 🔎 Search & Data Extraction  
**Repository:** https://github.com/Atlas-Compiler/Atlas  
**GitHub Stars:** 1  
**npm Downloads (last month):** 855  
**Views:** 1  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/atlas-web-compiler-scraper

## Description
Deterministic Zero-LLM Web Compiler, Scraper, Crawler, and Site Mapper for AI Agents.

## Tools
Capabilities this server exposes over MCP:

- **atlas_scrape** — Quickly extract clean, readable text, markdown, and links from a web page using Atlas's fast synchronous parser. Strips cookie banners, navigation clutter, and ads. Best for articles, documentation pages, and general web reading.
- **atlas_compile** — Compile a public web page into clean, deterministic, LLM-optimized markdown or structured AST nodes using the full 12-pass Atlas compiler. Strips boilerplate and generates section hierarchies. Best for complex documentation, API references, and dense technical articles.
- **atlas_map** — Discover the URL topology, sitemap structure, and hierarchy of a website without compiling individual pages. Use before crawling to identify relevant documentation paths and section boundaries.
- **atlas_batch_compile** — Compile a bounded list of explicit URLs in parallel. Returns clean markdown for each page within a consolidated budget. Maximum 20 URLs per batch call.
- **atlas_crawl** — Traverse and compile a bounded subsection of a website (such as an API reference or documentation folder) starting from a seed URL. Returns consolidated page summaries and navigable resource URIs.

## Claude Desktop Quick Installation
Install path detected from listing signals. Uses `npx` (confidence: high):

```json
"mcpServers": {
  "atlas-web-compiler-scraper": {
    "command": "npx",
    "args": ["-y","@atlascompiler/mcp"],
    "env": {
      "ATLAS_API_KEY": ""
    }
  }
}
```

**Requires environment variables:** `ATLAS_API_KEY` — the values above are empty placeholders; fill in real credentials before running (see the repository for what each one is for).

## Documentation

## What Atlas Web Compiler & Scraper MCP server does

Atlas Web Compiler & Scraper MCP server gives MCP-compatible agents access to Atlas web retrieval and transformation operations. It is intended for turning webpages, single-page applications, and documentation sites into data that an agent can inspect without handling the original page markup directly.

The service applies heuristic, multi-pass processing rather than an LLM-based extraction step. Its output can include token-reduced Markdown, structured AST data, metadata, tables, code fences, equations, and outbound links. The repository describes the pipeline as deterministic, so repeated requests are designed to produce reproducible transformations.

## How it works

The MCP package acts as the agent-facing interface to Atlas. Requests are sent to the Atlas service using an API key, while the MCP layer makes the supported operations available as tools to an assistant or coding environment.

The available operations are:

- `compile` performs a deeper AST-oriented transformation and extracts a structural content graph.
- `scrape` retrieves page content as Markdown for lower-latency agent interactions.
- `map` discovers reachable URLs across a domain and returns a ranked topology.
- `crawl` runs bounded, multi-page crawls with path glob filtering.
- `batch` retrieves up to 500 URLs in parallel within one job and supports unified status webhooks.

The Atlas Web Compiler & Scraper MCP server is useful when an agent needs documentation lookup, page analysis, domain discovery, or multi-page collection. The README reports sub-1.5-second latency for the scraping operation and describes execution across more than 300 edge locations, but actual results depend on the requested operation and target site.

## Setup and configuration

Install and initialize the MCP package with:

```bash
npx -y @atlascompiler/mcp init
```

The interactive initializer requests an Atlas API key, checks connectivity, and configures installed Claude Desktop, Cursor, VS Code, and Windsurf editors. The README says the initializer avoids placing tokens in process arguments. To start the stdio server without the wizard, run:

```bash
npx -y @atlascompiler/mcp
```

An Atlas API key is required. The same service can also be accessed outside MCP through the documented JavaScript, Python, and Go SDKs or through the REST API, but those alternatives are separate from the MCP server configuration.

## Limitations and notes

The tool depends on access to the Atlas service and therefore requires an API key and network connectivity. It is not presented as a browser automation environment or a general-purpose page interaction tool; its documented role is compiling, scraping, mapping, crawling, and batching web content.

The pipeline is heuristic and deterministic. This avoids an LLM extraction stage, but the available material does not promise that every site, dynamic application, or unusual page structure will be interpreted perfectly. Crawls are bounded, and batch jobs support up to 500 URLs per job. The repository is licensed under MIT.

The Atlas Web Compiler & Scraper MCP server is one part of a larger repository that also contains SDKs, agent skills, examples, and the MCP package.

_Full upstream README: https://allmcps.com/mcp/atlas-web-compiler-scraper/readme_