Lookacrawler

What lucasmartins-ai/lookacrawler does

lucasmartins-ai/lookacrawler is a local web extraction and crawling tool in the Search & Data Extraction category. It removes common page noise such as scripts, styles, tracking elements, navigation, footers, redundant forms, and highly repetitive containers before returning Markdown. Link and image output can be configured to preserve inline links, convert links to references, strip URLs, ignore images, retain image alt text, or emit full Markdown images.

The server also supports structured extraction. It can return Open Graph metadata, canonical URLs, publication dates, and values selected with CSS selectors. Contact extraction covers email addresses, phone numbers, WhatsApp links, and social profiles. Other bundled workflows cover Google Maps places, B2B lead enrichment, website audits, competitor review, growth-plan generation, copy linting, social-profile lookup, and buying-intent searches.

How it works

lucasmartins-ai/lookacrawler uses a hybrid crawler. Fast mode makes native HTTP requests and can escalate when a page depends on JavaScript. Deep mode launches Google Chrome through Playwright for JavaScript-heavy pages and single-page applications. Browser actions can run before extraction, including clicks, scrolling, text entry, waits, cookie-banner dismissal, and accordion expansion.

Site mapping checks robots.txt, sitemaps, and links from the root page. Recursive crawling uses breadth-first traversal with maximum depth, page limits, URL filters, and token accounting. A local SQLite cache stores extracted Markdown and uses a 24-hour time-to-live to avoid repeated network requests. MCP resources expose crawler metrics and cache statistics, while prompts provide crawl-and-summarize and compare-pages templates.

Setup and configuration

The runtime requirement is Bun 1.1 or newer. Clone the repository, enter its directory, and run bun install. The MCP process can then be registered with a client using a stdio command that runs index.ts through Bun:

{
  "mcpServers": {
    "lookacrawler": {
      "command": "bun",
      "args": ["run", "/absolute/path/to/lookacrawler/index.ts"]
    }
  }
}

CLI users can run commands such as extract, map, crawl, batch, structured, contacts, places, and enrich with their respective URL, filtering, output, and concurrency options. No required environment variables are documented.

Tools and capabilities

The MCP tool set includes:

• extract_web_content, batch_extract_web_content, and extract_structured_data for page extraction.
• map_website and crawl_website for discovery and recursive crawling.
• extract_contacts, search_places, discover_local_leads, and deep_enrich_lead for contact and lead workflows.
• audit_company_website, analyze_competitors, generate_growth_plan, and gtm_pipeline_run for the bundled GTM workflows.
• lint_anti_slop, lookup_social_profile, and search_social_intent for copy and public social-data tasks.

The GTM pipeline drafts outreach data to JSONL or CSV and does not send messages. Google Maps workflows use the bundled scraper engine; its build step is required for the related CLI workflow.

Limitations and notes

The project is local-first and does not document required API keys. Deep extraction depends on a locally available Google Chrome installation and Playwright. CAPTCHA and other bot-challenge pages are detected and reported rather than repeatedly retried. Results depend on the accessible page content, browser actions, URL filters, crawl limits, and extraction selectors. The repository is licensed under MIT.