# olostep/olostep-mcp-server [Health: Active]

**Category:** 🔎 Search & Data Extraction  
**Repository:** https://github.com/olostep/olostep-mcp-server  
**GitHub Stars:** 25  
**npm Downloads (last month):** 869  
**Views:** 2  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/olostep-olostep-mcp-server

## Description
API to search, extract and structure web data. Web scraping, AI-powered answers with citations, batch processing (10k URLs), and autonomous site crawling.

## Tools
Capabilities this server exposes over MCP:

- **create_map** — Get a LIST of URLs on a website (URL discovery only — does NOT scrape content). Use when the user wants a list of links: 'show me all URLs on this site', 'map this website', or when you want to surface candidate URLs to the user before scraping a subset. Prefer `create_crawl` if the goal is to scrape the whole site — it discovers AND scrapes in one workflow. Use this only when the URL list itself is the deliverable.
- **create_crawl** — **PREFERRED tool for crawling a website.** Use this whenever the user says 'crawl', 'scrape the whole site', 'get all pages from a site', or wants multiple pages from a single domain. This is the CORRECT tool for any whole-site scraping task. **Do NOT use `batch_scrape_urls` for crawling** - that tool is only for when you already have a specific list of unrelated URLs from different domains. Starts an ASYNC crawl that autonomously discovers and scrapes pages by following links from a start URL. Returns a crawl_id - the crawl runs in the background. You MUST then call `get_crawl_results` with the returned crawl_id to poll status and retrieve the scraped pages. Do NOT call `get_batch_results` with a crawl_id - crawls and batches are separate resources.
- **get_crawl_results** — Retrieve the status and scraped pages for a crawl job. Pass the crawl_id returned by create_crawl. If the crawl is still in_progress, returns the current status so you can call again later (poll every ~10 seconds). Once completed, returns the list of discovered pages with their scraped content in the requested formats. This is the REQUIRED companion to create_crawl — create_crawl only kicks off the async job, this tool is how you actually get the content.
- **batch_scrape_urls** — Scrape a SPECIFIC, KNOWN list of URLs (typically from different domains). **Do NOT use this for crawling a website** - if the user wants to scrape a whole site or 'crawl' a domain, use `create_crawl` instead. Use this only when you already have an explicit list of URLs to scrape (e.g., user provides a CSV of URLs, or you need to scrape unrelated pages). Returns a batch_id immediately. Use `get_batch_results` with the batch_id to fetch the scraped content once the batch completes (~5–8 min). Set `wait_for_completion_seconds` to poll automatically.
- **get_batch_results** — Retrieve the status and scraped content for a batch job. Pass the batch_id returned by batch_scrape_urls. If the batch is completed, returns the scraped content for each URL. If still in_progress, returns the current status so you can call again later.
- **answers** — Answer a factual question using web search, optionally shaped into a flat JSON object of fields (returned with sources and citations). Best for a bounded, factual answer (e.g. a company's founding year, a product's current price). It is NOT reliable for enumerating a live list from a page (e.g. 'the latest N blog posts with titles and dates'). For that, use create_map or get_webpage_content on the page and read the results instead.
- **search_web** — Search the web for a given query and return structured results (non-AI, parser-based).
- **scrape_website** — Extract content from a single URL. Supports multiple formats and JavaScript rendering.
- **get_webpage_content** — Retrieve content of a webpage in markdown
- **get_website_urls** — Search and retrieve relevant URLs from a website (URL discovery only - does NOT scrape content). Use this only when the user wants a *filtered list of links* matching a search query. **Do NOT use this as a precursor to scraping** - if the user wants to scrape/crawl a site, use `create_crawl` directly.

## Claude Desktop Quick Installation
Install path detected from listing signals. Uses `npx` (confidence: high):

```json
"mcpServers": {
  "olostep-mcp-server": {
    "command": "npx",
    "args": ["-y","olostep-mcp"],
    "env": {
      "OLOSTEP_API_KEY": ""
    }
  }
}
```

**Requires environment variables:** `OLOSTEP_API_KEY` — the values above are empty placeholders; fill in real credentials before running (see the repository for what each one is for).

## Documentation

## What olostep/olostep-mcp-server does

Olostep exposes web search and data extraction capabilities through MCP. It can retrieve one webpage, return structured search results, discover URLs on a site, crawl a site from a starting URL, scrape a known list of URLs, or answer factual questions using web research. AI answers include sources and citations and can be shaped into a flat JSON object.

Single-page extraction supports HTML, Markdown, JSON, and plain text output. Optional parsers can specialize extraction for supported use cases. Requests can also specify a country for localized content and a delay before scraping JavaScript-heavy pages.

## How it works

The server can be consumed through the hosted endpoint at `https://mcp.olostep.com/mcp`, authenticated with an Olostep API key in a Bearer token. It can also run locally through the `olostep-mcp` npm package or the published Docker image.

Whole-site crawling is asynchronous. Call `create_crawl` with a starting URL; it follows links, discovers pages, and scrapes them in the background. Use the returned `crawl_id` with `get_crawl_results` to check progress and retrieve the pages. Poll again while the job is still running, roughly every 10 seconds.

Known URL lists use a separate workflow. `batch_scrape_urls` starts a batch and returns a `batch_id`; `get_batch_results` retrieves its status and content. Batch jobs can take approximately five to eight minutes, and the batch tool is intended for explicit URLs, including URLs from unrelated domains, rather than whole-site crawling.

## Setup and configuration

An Olostep account and API key are required. Set the key as `OLOSTEP_API_KEY` when running the local package or Docker image. The optional `ORBIT_KEY` supports routing requests through Orbit.

Run the package without a global installation with:

```bash
env OLOSTEP_API_KEY=your-api-key npx -y olostep-mcp
```

The Docker option uses the `olostep/mcp-server` image and passes the same environment variable. Client configurations are documented for Claude Desktop, Cursor, Claude Code, Windsurf, Cline, VS Code, and Gemini CLI. These clients can use either the hosted endpoint or a local stdio process, depending on the configuration. Gemini CLI stores the API key in its sensitive extension settings.

## Tools and capabilities

- `scrape_website` extracts one URL with selectable output format, JavaScript wait time, country routing, and an optional parser.
- `search_web` returns structured, parser-based web search results without an AI-generated answer.
- `answers` researches a factual question and returns citations, with optional JSON-shaped output.
- `get_webpage_content` retrieves webpage content as Markdown.
- `create_map` lists URLs discovered on a website without scraping their content.
- `get_website_urls` finds relevant URLs matching a website search query, also without scraping.
- `create_crawl` and `get_crawl_results` handle autonomous, multi-page site crawling.
- `batch_scrape_urls` and `get_batch_results` process a specific list of URLs, with support for batches of up to 10,000 URLs.

## Limitations and notes

URL discovery tools return links rather than page content. Use `create_crawl` when the objective is to scrape an entire site, and reserve `batch_scrape_urls` for an already-known URL list. `answers` is intended for bounded factual questions; it is not suited to reliably enumerating a live list from a page, such as the newest posts and their dates. Crawl and batch jobs return asynchronously, so their companion result tools are required to obtain completed content.

_Full upstream README: https://allmcps.com/mcp/olostep-olostep-mcp-server/readme_