# Mingye-Lu/AgenticCrawler [Health: Active]

**Category:** 🔎 Search & Data Extraction  
**Repository:** https://github.com/Mingye-Lu/AgenticCrawler  
**GitHub Stars:** 15  
**Views:** 2  
**Installs:** 0  
**Upvotes:** 0  
**Directory Page:** https://allmcps.com/mcp/mingye-lu-agenticcrawler

## Description
Autonomous LLM-powered web crawler with 17 browser tools and goal-driven agent. Single binary, stealth browsing, 25 LLM providers.

## Claude Desktop Quick Installation
No generated config for this server: it installs through its own setup steps.

The project's own setup steps (from https://github.com/Mingye-Lu/AgenticCrawler), copied verbatim. Run them in order; they are not an mcpServers config.

1. Install:

```sh
curl -fsSL https://raw.githubusercontent.com/Mingye-Lu/AgenticCrawler/main/install.sh | bash
```

2. Install:

```sh
irm https://raw.githubusercontent.com/Mingye-Lu/AgenticCrawler/main/install.ps1 | iex
```

3. Add to your MCP client:

```sh
acrawl mcp install                     # add acrawl to Claude Code, Cursor, VS Code and other MCP clients
```

## Documentation

## What Mingye-Lu/AgenticCrawler does

Mingye-Lu/AgenticCrawler is a local browser automation agent, command-line application, and MCP server packaged as a Rust binary. An agent receives a natural-language goal, chooses actions, observes the resulting page state, and returns data or findings. Typical tasks include collecting fields across paginated sites, submitting forms, following multi-step flows, and investigating why a page or request behaves incorrectly.

The project supports both ordinary page extraction and developer-oriented browser inspection. In addition to navigation and interaction, its tools can expose network requests, console messages, WebSocket frames, performance timings, cookies, browser storage, code coverage, and axe-core accessibility audits.

## How it works

A request can enter through the interactive terminal UI, a one-shot `acrawl prompt` command, or an MCP client. The runtime sends the goal and available tool results to a selected LLM provider, then executes the model's actions in the browser. Static pages are fetched over HTTP where possible; JavaScript-dependent pages can be handled by the bundled browser runtime.

The agent can divide larger jobs across separate tabs and uses a URL-claiming registry to avoid duplicate work between sub-agents. After an exploratory run, a crawl can be saved as a JSON script containing loops, conditions, and parallel branches. `acrawl run` replays that script without making further LLM calls, which suits repeatable jobs in cron or CI.

The MCP mode lets supported coding agents drive the same local browser through acrawl's MCP entry point. The optional Bridge extension connects the runtime to an already logged-in Chrome, Edge, or Brave session.

## Setup and configuration

Prebuilt installation commands are available for Linux and macOS on x64 and ARM64, plus Windows on x64:

```bash
curl -fsSL https://raw.githubusercontent.com/Mingye-Lu/AgenticCrawler/main/install.sh | bash
```

```powershell
irm https://raw.githubusercontent.com/Mingye-Lu/AgenticCrawler/main/install.ps1 | iex
```

The installer downloads the latest release, verifies its SHA256 checksum, and sets up CloakBrowser. Browser features require Node.js 20 or newer. Source builds are also supported through the project's installation guide.

Choose an LLM provider with the authentication command, then start the interactive interface or submit a single goal:

```bash
acrawl auth anthropic
acrawl
acrawl prompt "list the titles of the top 10 stories on news.ycombinator.com"
```

The provider list includes Anthropic, OpenAI, Google Gemini, DeepSeek, Bedrock, Azure OpenAI, Vertex AI, GitHub Copilot, Groq, Mistral, OpenRouter, Ollama, and other supported providers. An OpenAI-compatible endpoint such as Ollama can also be used. Run `acrawl mcp install` to configure the server for an MCP client.

## Tools and capabilities

- Navigate websites and interact with controls using clicks, typing, selection, hovering, and key presses.
- Extract structured information from static and JavaScript-heavy sites.
- Inspect requests, responses, WebSocket activity, console errors, cookies, storage, performance data, and code coverage.
- Run axe-core accessibility audits.
- Emulate phones and connect to a user's existing browser session through the Bridge extension.
- Delegate work across tabs and coordinate URL ownership between sub-agents.
- Save crawls as deterministic JSON scripts and replay them without model calls.
- Expose the browser and inspection functions through MCP for coding agents.

## Limitations and notes

The application is local and uses the operator's own LLM credentials. Node.js is needed for bundled-browser features even though the main application is distributed as a single Rust binary. The MCP workflow depends on an MCP-compatible client, while use of an existing logged-in browser requires the acrawl Bridge extension. MIT licensing applies to the project.

_Full upstream README: https://allmcps.com/mcp/mingye-lu-agenticcrawler/readme_