Atlas Web Compiler & Scraper README

Atlas

The web compiler that turns webpages into clean, structured and token efficient data for AI agents and developers.

Transforms messy web pages, documentation, and articles into token-efficient Markdown, structured ASTs, and domain topologies with sub-1.5s latency.

What Atlas Does (Before & After)

⚡ Why Atlas?

Traditional web scraping returns megabytes of raw, unstructured HTML bloated with navigation bars, cookie banners, tracking scripts, and presentational CSS. Feeding raw HTML into Large Language Models wastes 70% to 90% of your context window, escalates inference costs, and introduces reasoning hallucinations.

Atlas compiles web content into clean, token-efficient markdown at the edge—before it reaches your models:

• 🧠 Deterministic by Design: Rule-based AST heuristic passes produce identical, reliable output for identical inputs. Zero prompt drift, zero hallucinated content, and zero LLM cost during compilation.
• 📉 70–90% Token Reduction: Eliminates boilerplate chrome, banners, and layout bloat while faithfully preserving headings, tables, code fences, equations, and semantic links.
• 🤖 Built for AI Agents: Native Model Context Protocol (MCP) server for instant integration into Claude Desktop, Cursor, VS Code, and Windsurf, plus native SDKs (TypeScript, Python, Go) for custom agent loops.
• ⚡ Sub-1.5s Edge Execution: Powered by Cloudflare Workers across 300+ edge locations with built-in DNS-aware SSRF protection.
• 🔁 Safe Retries & Idempotency: Built-in request deduplication ensures retrying failed or timed-out requests never burns duplicate compute credits or double-executes jobs.
• 🌐 Engineered for Real-World Content: High-fidelity extraction tailored for documentation engines (Mintlify, Docusaurus, GitBook), technical blogs, research publications, and modern SSR/SSG sites.

The result: High-signal, token-efficient input for AI agents and RAG pipelines—without spending LLM tokens just to clean up HTML.

📦 What’s in This Repository

This monorepo houses the public ecosystem and developer tools for Atlas:

🚀 Quickstarts

🔑 Authentication & API Key Setup

1. Get your API Key: Sign in to the Atlas Dashboard to generate your free API key (atlas_...).

2. Set your Environment Variable (Recommended): All Atlas SDKs, MCP servers, and CLI tools automatically detect ATLAS_API_KEY:

   server.tsCopy

   # macOS / Linux (add to ~/.bashrc or ~/.zshrc)
   export ATLAS_API_KEY="atlas_your_api_key_here"
   
   # Windows PowerShell
   $env:ATLAS_API_KEY = "atlas_your_api_key_here"
   

[!TIP] Prefer not to set environment variables? You can pass your API key directly into any SDK client constructor as shown below.

Direct HTTP / cURL

# Instant Real-Time Scrape (Sub-1.5s, clean markdown output)
curl -X POST https://api.atlas-compiler.com/v1/scrape \
  -H "Authorization: Bearer ${ATLAS_API_KEY:-atlas_your_api_key_here}" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

# Deep Semantic Compilation (includes full AST, link graph, and idempotency key)
curl -X POST https://api.atlas-compiler.com/v1/compile \
  -H "Authorization: Bearer ${ATLAS_API_KEY:-atlas_your_api_key_here}" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

AI Agents & MCP Server

Connect Claude Desktop, Cursor, VS Code, or Windsurf directly to Atlas with the 1-click interactive wizard:

npx -y @atlascompiler/mcp init

The wizard prompts for your Atlas API key, validates connectivity, and automatically configures your installed editors.

To run the Stdio server directly:

npx -y @atlascompiler/mcp

For Remote / Cloud AI Agents (Streamable HTTP):

• Endpoint: https://api.atlas-compiler.com/mcp
• Headers: Authorization: Bearer atlas_your_api_key_here

TypeScript / JavaScript

Install the official type-safe client library:

npm install @atlascompiler/sdk
# or: pnpm add @atlascompiler/sdk / yarn add @atlascompiler/sdk

import { AtlasClient } from '@atlascompiler/sdk';

// Option A: Explicit manual API key
const client = new AtlasClient({
  apiKey: 'atlas_your_api_key_here',
});

// Option B: Automatically loaded from process.env.ATLAS_API_KEY
// const client = new AtlasClient();

// Compile any web page into clean, structured Markdown
const response = await client.compile({
  url: 'https://docs.github.com/en/rest',
});

if (response.status === 200) {
  console.log('Title:', response.value.metadata.title);
  console.log('Markdown:\n', response.value.markdown);
}

Python

Install the official Python client library:

pip install atlascompiler

from atlascompiler import AtlasClient

# Option A: Explicit manual API key
client = AtlasClient("atlas_your_api_key_here")

# Option B: Automatically loaded from os.environ["ATLAS_API_KEY"]
# client = AtlasClient()

response = client.compile({"url": "https://example.com"})

if response.status == 200:
    print("Title:", response.metadata.get("title"))
    print("Markdown:\n", response.markdown)

Async support is built-in via AsyncAtlasClient.

Go

Install the official Go module:

go get go.atlas-compiler.com/sdk

package main

import (
	"context"
	"fmt"
	"log"

	atlas "go.atlas-compiler.com/sdk"
)

func main() {
	// Option A: Explicit manual API key
	client := atlas.New("atlas_your_api_key_here")

	// Option B: Automatically loaded from os.Getenv("ATLAS_API_KEY")
	// client := atlas.New("")

	res, _, err := client.Compile(context.Background(), atlas.CompileUrlRequest{
		Url: "https://example.com",
	})
	if err != nil {
		log.Fatal(err)
	}

	if res.Response200 != nil {
		fmt.Println("Markdown:\n", res.Response200.Markdown)
	}
}

🛠 Supported Operations

Atlas exposes 5 core operations via REST, SDKs, and MCP:

Synchronous Operations (Real-Time)

• scrape (POST /v1/scrape): Real-time Markdown, raw text, and link extraction for interactive agent loops (sub-1.5s).
• compile (POST /v1/compile): Deep 12-pass AST compilation with structural content graph extraction, Table of Contents, and link resolution.
• map (POST /v1/map): Rapid topological spider that crawls sitemaps and discovers reachable URLs across a domain without fetching bodies.

Asynchronous Operations (Queued Jobs)

• crawl (POST /v1/crawls): Bounded, asynchronous multi-page website crawls with path glob filtering and progress tracking (202 Accepted).
• batch (POST /v1/batches): High-concurrency parallel retrieval for up to 500 URLs per job with unified status webhooks (202 Accepted).

💬 Community & Support

• Discord: Join our active developer community at discord.gg/z9ktRhuudq for help, agent prompts, and feature discussions.
• GitHub Discussions: Share recipes, ask questions, or propose ideas at Discussions.
• Bug Reports: Open an issue using our Bug Report Template.
• Security Inquiries: Email support@atlas-compiler.com for private disclosure.

📜 Contributing & License

Contributions are welcome! Please read our Contributing Guide and Code of Conduct.

Licensed under the MIT License.