AgenticCrawler

What Mingye-Lu/AgenticCrawler does

Mingye-Lu/AgenticCrawler is a local browser automation agent, command-line application, and MCP server packaged as a Rust binary. An agent receives a natural-language goal, chooses actions, observes the resulting page state, and returns data or findings. Typical tasks include collecting fields across paginated sites, submitting forms, following multi-step flows, and investigating why a page or request behaves incorrectly.

The project supports both ordinary page extraction and developer-oriented browser inspection. In addition to navigation and interaction, its tools can expose network requests, console messages, WebSocket frames, performance timings, cookies, browser storage, code coverage, and axe-core accessibility audits.

How it works

A request can enter through the interactive terminal UI, a one-shot acrawl prompt command, or an MCP client. The runtime sends the goal and available tool results to a selected LLM provider, then executes the model's actions in the browser. Static pages are fetched over HTTP where possible; JavaScript-dependent pages can be handled by the bundled browser runtime.

The agent can divide larger jobs across separate tabs and uses a URL-claiming registry to avoid duplicate work between sub-agents. After an exploratory run, a crawl can be saved as a JSON script containing loops, conditions, and parallel branches. acrawl run replays that script without making further LLM calls, which suits repeatable jobs in cron or CI.

The MCP mode lets supported coding agents drive the same local browser through acrawl's MCP entry point. The optional Bridge extension connects the runtime to an already logged-in Chrome, Edge, or Brave session.

Setup and configuration

Prebuilt installation commands are available for Linux and macOS on x64 and ARM64, plus Windows on x64:

curl -fsSL https://raw.githubusercontent.com/Mingye-Lu/AgenticCrawler/main/install.sh | bash

irm https://raw.githubusercontent.com/Mingye-Lu/AgenticCrawler/main/install.ps1 | iex

The installer downloads the latest release, verifies its SHA256 checksum, and sets up CloakBrowser. Browser features require Node.js 20 or newer. Source builds are also supported through the project's installation guide.

Choose an LLM provider with the authentication command, then start the interactive interface or submit a single goal:

acrawl auth anthropic
acrawl
acrawl prompt "list the titles of the top 10 stories on news.ycombinator.com"

The provider list includes Anthropic, OpenAI, Google Gemini, DeepSeek, Bedrock, Azure OpenAI, Vertex AI, GitHub Copilot, Groq, Mistral, OpenRouter, Ollama, and other supported providers. An OpenAI-compatible endpoint such as Ollama can also be used. Run acrawl mcp install to configure the server for an MCP client.

Tools and capabilities

• Navigate websites and interact with controls using clicks, typing, selection, hovering, and key presses.
• Extract structured information from static and JavaScript-heavy sites.
• Inspect requests, responses, WebSocket activity, console errors, cookies, storage, performance data, and code coverage.
• Run axe-core accessibility audits.
• Emulate phones and connect to a user's existing browser session through the Bridge extension.
• Delegate work across tabs and coordinate URL ownership between sub-agents.
• Save crawls as deterministic JSON scripts and replay them without model calls.
• Expose the browser and inspection functions through MCP for coding agents.

Limitations and notes

The application is local and uses the operator's own LLM credentials. Node.js is needed for bundled-browser features even though the main application is distributed as a single Rust binary. The MCP workflow depends on an MCP-compatible client, while use of an existing logged-in browser requires the acrawl Bridge extension. MIT licensing applies to the project.