LLM Documentation Scraper (doc-scraper)

A configurable, concurrent, and resumable web crawler written in Go. Specifically designed to scrape technical documentation websites, extract core content, convert it cleanly to Markdown format suitable for ingestion by Large Language Models (LLMs), and save the results locally.

Overview

This project provides a powerful command-line tool to crawl documentation sites based on settings defined in a config.yaml file. It navigates the site structure, extracts content from specified HTML sections using CSS selectors, and converts it into clean Markdown files.

Why Use This Tool?

• Built for LLM Training & RAG Systems - Creates clean, consistent Markdown optimized for ingestion
• Preserves Documentation Structure - Maintains the original site hierarchy for context preservation
• Production-Ready Features - Offers resumable crawls, rate limiting, and graceful error handling
• High Performance - Uses Go's concurrency model for efficient parallel processing

Goal: Preparing Documentation for LLMs

The main objective of this tool is to automate the often tedious process of gathering and cleaning web-based documentation for use with Large Language Models. By converting structured web content into clean Markdown, it aims to provide a dataset that is:

• Text-Focused: Prioritizes the textual content extracted via CSS selectors
• Structured: Maintains the directory hierarchy of the original documentation site, preserving context
• Cleaned: Converts HTML to Markdown, removing web-specific markup and clutter
• Locally Accessible: Provides the content as local files for easier processing and pipeline integration

Key Features

Getting Started

Prerequisites

• Go: Version 1.26 or later
• Git: For cloning the repository
• Disk Space: Sufficient for storing crawled content and state database

Installation

Option 1: Direct Installation (Recommended)

Install the latest version directly from GitHub:

go install github.com/Sriram-PR/doc-scraper/v2/cmd/doc-scraper@latest

This installs the doc-scraper binary to your GOPATH/bin directory (usually ~/go/bin or %USERPROFILE%\go\bin). Make sure this directory is in your PATH.

Option 2: Clone and Build

1. Clone the repository:

   bash

   Copy

   git clone https://github.com/Sriram-PR/doc-scraper.git
   cd doc-scraper
   

2. Install Dependencies:

   bash

   Copy

   go mod tidy
   

3. Build the Binary:

   bash

   Copy

   make build
   # or: go build -o doc-scraper ./cmd/doc-scraper
   

   This creates an executable named doc-scraper in the project root.

Quick Start

Create a minimal config.yaml in the project root:

output_base_dir: "./crawled_docs"
state_dir: "./crawler_state"
enable_jsonl_output: true
sites:
  rust_cli_book:
    start_urls:
      - "https://rust-cli.github.io/book/index.html"
    allowed_domain: "rust-cli.github.io"
    allowed_path_prefix: "/book/"
    content_selector: "#content, main"
    max_depth: 2          # seed plus one level; set 0 for the whole book

Run the crawl:

./doc-scraper crawl -site rust_cli_book -loglevel info

The Markdown, plus pages.jsonl, llms.txt, and llms-full.txt, lands under ./crawled_docs/rust_cli_book/ (output is organized by site key). A small book like this finishes in a few seconds; large sites can take minutes, so start with a low max_depth to gauge size before removing the bound.

Configuration (config.yaml)

A config.yaml file is required to run the crawler. Create this file in the project root or specify its path using the -config flag.

Key Settings for LLM Use

When configuring for LLM documentation processing, pay special attention to these settings:

• sites.<your_site_key>.content_selector: Define precisely to capture only relevant text
• sites.<your_site_key>.allowed_domain / allowed_path_prefix: Define scope accurately
• skip_images: Images are not downloaded by default (text-first). Set to false globally or per-site to download and localize images for offline consumption
• Adjust concurrency/delay settings based on the target site and your resources

Example Configuration

# Global settings (applied if not overridden by site)
default_delay_per_host: 500ms
num_workers: 8
num_image_workers: 8
max_requests: 48
max_requests_per_host: 4
output_base_dir: "./crawled_docs"
state_dir: "./crawler_state"
max_retries: 4
initial_retry_delay: 1s
max_retry_delay: 30s
global_crawl_timeout: 0s
skip_images: true # Default. Set to false to download and localize images
max_image_size_bytes: 10485760 # 10 MiB (applies only when images are downloaded)
enable_jsonl_output: true
jsonl_output_filename: "pages.jsonl"

# HTTP Client Settings
http_client_settings:
  timeout: 45s
  max_idle_conns_per_host: 6

# Site-specific configurations
sites:
  # Key used with -site flag
  pytorch_docs:
    start_urls:
      - "https://pytorch.org/docs/stable/"
    allowed_domain: "pytorch.org"
    allowed_path_prefix: "/docs/stable/"
    content_selector: "article.pytorch-article .body"
    max_depth: 0 # 0 for unlimited depth
    skip_images: false # Opt in to downloading images for this site
    disallowed_path_patterns:
      - "/docs/stable/.*/_modules/.*"
      - "/docs/stable/.*\.html#.*"

  tensorflow_docs:
    start_urls:
      - "https://www.tensorflow.org/guide"
      - "https://www.tensorflow.org/tutorials"
    allowed_domain: "www.tensorflow.org"
    allowed_path_prefix: "/"
    content_selector: ".devsite-article-body"
    max_depth: 0
    delay_per_host: 1s  # Site-specific override
    # Disable JSONL output for this site, overriding global
    enable_jsonl_output: false
    disallowed_path_patterns:
      - "/install/.*"
      - "/js/.*"

Full Configuration Options