Content
" - Default: None website_markdown (Optional[str]): Markdown content to process locally. - Mutually exclusive with website_url and website_html - Maximum size: 2MB - Useful for extracting from markdown documents or converted content - Works well with documentation, README files, or converted web content - Example: "# Title ## Section Content here..." - Default: None output_schema (Optional[Union[str, Dict]]): JSON schema defining expected output structure. - Can be provided as a dictionary or JSON string - Helps ensure consistent, structured output format - Optional but recommended for complex extractions - Examples: * As dict: {'type': 'object', 'properties': {'title': {'type': 'string'}, 'price': {'type': 'number'}}} * As JSON string: '{"type": "object", "properties": {"name": {"type": "string"}}}' * For arrays: {'type': 'array', 'items': {'type': 'object', 'properties': {...}}} - Default: None (AI will infer structure from prompt) number_of_scrolls (Optional[int]): Number of infinite scrolls to perform before scraping. - Range: 0-50 scrolls - Default: 0 (no scrolling) - Useful for dynamically loaded content (lazy loading, infinite scroll) - Each scroll waits for content to load before continuing - Examples: * 0: Static content, no scrolling needed * 3: Social media feeds, product listings * 10: Long articles, extensive product catalogs - Note: Increases processing time proportionally total_pages (Optional[int]): Number of pages to process for pagination. - Range: 1-100 pages - Default: 1 (single page only) - Automatically follows pagination links when available - Useful for multi-page listings, search results, catalogs - Examples: * 1: Single page extraction * 5: First 5 pages of search results * 20: Comprehensive catalog scraping - Note: Each page counts toward credit usage (10 credits Γ pages) render_heavy_js (Optional[bool]): Enable heavy JavaScript rendering for dynamic sites. - Default: false - Set to true for Single Page Applications (SPAs), React apps, Vue.js sites - Increases processing time but captures client-side rendered content - Use when content is loaded dynamically via JavaScript - Examples of when to use: * React/Angular/Vue applications * Sites with dynamic content loading * AJAX-heavy interfaces * Content that appears after page load - Note: Significantly increases processing time (30-60 seconds vs 5-15 seconds) stealth (Optional[bool]): Enable stealth mode to avoid bot detection. - Default: false - Helps bypass basic anti-scraping measures - Uses techniques to appear more like a human browser - Useful for sites with bot detection systems - Examples of when to use: * Sites that block automated requests * E-commerce sites with protection * Sites that require "human-like" behavior - Note: May increase processing time and is not 100% guaranteed Returns: Dictionary containing: - extracted_data: The structured data matching your prompt and optional schema - metadata: Information about the extraction process - credits_used: Number of credits consumed (10 per page processed) - processing_time: Time taken for the extraction - pages_processed: Number of pages that were analyzed - status: Success/error status of the operation Raises: ValueError: If no input source provided or multiple sources provided HTTPError: If website_url cannot be accessed TimeoutError: If processing exceeds timeout limits ValidationError: If output_schema is malformed JSON - **searchscraper** β Perform AI-powered web searches with structured data extraction. This tool searches the web based on your query and uses AI to extract structured information from the search results. Ideal for research, competitive analysis, and gathering information from multiple sources. Each website searched costs 10 credits (default 3 websites = 30 credits). Read-only operation but results may vary over time (non-idempotent). - **smartcrawler_initiate** β Initiate an asynchronous multi-page web crawling operation with AI extraction or markdown conversion. This tool starts an intelligent crawler that discovers and processes multiple pages from a starting URL. Choose between AI Extraction Mode (10 credits/page) for structured data or Markdown Mode (2 credits/page) for content conversion. The operation is asynchronous - use smartcrawler_fetch_results to retrieve results. Creates a new crawl request (non-idempotent, non-read-only). SmartCrawler supports two modes: - AI Extraction Mode: Extracts structured data based on your prompt from every crawled page - Markdown Conversion Mode: Converts each page to clean markdown format - **smartcrawler_fetch_results** β Retrieve the results of an asynchronous SmartCrawler operation. This tool fetches the results from a previously initiated crawling operation using the request_id. The crawl request processes asynchronously in the background. Keep polling this endpoint until the status field indicates 'completed'. While processing, you'll receive status updates. Read-only operation that safely retrieves results without side effects. - **scrape** β Fetch raw page content from any URL with optional JavaScript rendering. This tool performs basic web scraping to retrieve the raw HTML content of a webpage. Optionally enable JavaScript rendering for Single Page Applications (SPAs) and sites with heavy client-side rendering. Lower cost than AI extraction (1 credit/page). Read-only operation with no side effects. - **sitemap** β Extract and discover the complete sitemap structure of any website. This tool automatically discovers all accessible URLs and pages within a website, providing a comprehensive map of the site's structure. Useful for understanding site architecture before crawling or for discovering all available content. Very cost-effective at 1 credit per request. Read-only operation with no side effects. - **agentic_scrapper** β Execute complex multi-step web scraping workflows with AI-powered automation. This tool runs an intelligent agent that can navigate websites, interact with forms and buttons, follow multi-step workflows, and extract structured data. Ideal for complex scraping scenarios requiring user interaction simulation, form submissions, or multi-page navigation flows. Supports custom output schemas and step-by-step instructions. Variable credit cost based on complexity. Can perform actions on the website (non-read-only, non-idempotent). The agent accepts flexible input formats for steps (list or JSON string) and output_schema (dict or JSON string) to accommodate different client implementations. ## Claude Desktop Quick Installation Install path detected from listing signals. Uses `uvx` (confidence: high): ```json "mcpServers": { "scrapegraphai-scrapegraph-mcp": { "command": "uvx", "args": ["scrapegraph-mcp"], "env": { "SGAI_API_KEY": "" } } } ``` **Requires environment variables:** `SGAI_API_KEY` β the values above are empty placeholders; fill in real credentials before running (see the repository for what each one is for). ## Documentation ## What ScrapeGraphAI-scrapegraph-mcp does ScrapeGraphAI-scrapegraph-mcp exposes ScrapeGraph AI API v2 through the Model Context Protocol. It is intended for agents that need to retrieve and transform web content without implementing direct HTTP calls to the ScrapeGraph service themselves. The server supports single-page scraping, prompt-based extraction, web search, asynchronous multi-page crawling, JSON Schema generation, and account-related queries. Scraping can return markdown, HTML, screenshots, branding information, links, images, or summaries. Extraction takes a website URL and a user prompt, with an optional output schema for structured results. ## How it works Requests are forwarded to the ScrapeGraph API v2 base URL, which defaults to `https://v2-api.scrapegraphai.com/api`. Authentication uses the `SGAI-APIKEY` header. An API key can be supplied through the environment or through the MCP configuration mechanisms described by the project. The `search` tool accepts a result count from 1 to 20 and can use country, time-range, and schema options. For larger jobs, `crawl_start` begins an asynchronous crawl and returns an identifier. Call `crawl_get_status` with that identifier and continue polling until the response reports `status: completed`. The `schema` tool can create or extend a JSON Schema from a prompt. `history` provides paginated request history with an optional service filter, while `monitor_activity` returns paginated activity records containing fields such as status, change information, elapsed time, and diffs. ## Setup and configuration ScrapeGraphAI-scrapegraph-mcp requires a ScrapeGraph API key, available from the ScrapeGraph Dashboard. The repository supports Python 3.13 or later and installation with either pip or uv. After installing the package, run `scrapegraph-mcp` or `python -m scrapegraph_mcp.server`; local communication uses standard input and output. The primary environment variables are: - `SGAI_API_KEY`: API key used to authenticate with ScrapeGraph. - `SGAI_API_URL`: optional replacement for the default API base URL. - `SGAI_TIMEOUT`: optional request timeout in seconds; the documented default is 120. The older names `SCRAPEGRAPH_API_BASE_URL` and `SGAI_TIMEOUT_S` remain supported as aliases. Smithery installation is documented for Claude Desktop, and the repository also provides local and remote configuration examples for Claude Desktop and Cursor. The hosted endpoint documented for new integrations is `https://sgai-mcp-main.onrender.com`; the older `mcp.scrapegraphai.com/mcp` endpoint is marked for deprecation. ## Tools and capabilities - `scrape`: retrieves a page in a selected output format. - `extract`: extracts information from a URL using a user prompt and optional schema. - `search`: searches for web results with filtering and schema options. - `crawl_start`: starts a crawl with a selected extraction mode. - `crawl_get_status`: checks the status of a crawl by ID. - `schema`: generates or augments a JSON Schema. - `credits`: retrieves account credit information. - `history`: lists paginated service history. - `monitor_activity`: lists paginated activity ticks for a monitor. ## Limitations and notes The server targets ScrapeGraph API v2, so requests depend on that service, its API key, and available account credits. Crawl operations are asynchronous and require status polling. The repository documents the removal of older names and features, including `markdownify`, `sitemap`, and `agentic_scrapper`; markdown output should be requested through `scrape` instead. ScrapeGraphAI-scrapegraph-mcp does not replace the underlying API's access, rate, or service requirements. _Full upstream README: https://allmcps.com/mcp/scrapegraphai-scrapegraph-mcp/readme_