ContextWeaver

🧵 A codebase context engine woven for AI agents

Semantic Code Retrieval for AI Agents — Hybrid Search • Graph Expansion • Token-Aware Packing

English · 简体中文

ContextWeaver is a semantic retrieval engine purpose-built for AI coding assistants. It combines hybrid search (vector + lexical), intelligent context expansion, and token-aware packing to deliver precise, relevant, and context-complete code snippets to LLMs.

✨ Core Features

🔍 Hybrid Retrieval Engine

• Vector Retrieval: deep semantic understanding via similarity
• Lexical Retrieval (FTS): exact matching for function names, class names, and other technical terms
• RRF Fusion (Reciprocal Rank Fusion): intelligently merges multiple recall channels

🧠 AST Semantic Chunking

• Tree-sitter parsing: supports TypeScript, JavaScript, Python, Go, Java, Rust, C, C++, C#, and more
• Dual-Text strategy: displayCode for presentation, vectorText for embedding
• Gap-Aware merging: handles code gaps intelligently while preserving semantic integrity
• Breadcrumb injection: vector text carries hierarchical paths to boost recall
• UTF-16 character-domain normalization: offsets are unified via SourceAdapter.toCharOffset before writing metadata, preventing multi-byte character slicing errors (v1.4.0+)

📊 Three-Stage Context Expansion

• E1 Neighbor expansion: adjacent chunks within the same file, preserving block completeness
• E2 Breadcrumb completion: sibling methods under the same class/function for structural understanding
• E3 Import resolution: cross-file dependency tracking (configurable toggle)

🎯 Smart TopK Cutoff

• Anchor & Floor: dynamic threshold plus an absolute floor as dual safeguards
• Delta Guard: prevents misjudgment in Top1-outlier scenarios
• Safe Harbor: the first N results only check the floor, guaranteeing baseline recall

🔌 Native MCP Support

• MCP Server mode: launch a Model Context Protocol server with one command
• Multi-tool granularity (v1.5.0+): beyond core semantic retrieval, adds dedicated tools for structure browsing, symbol references, symbol definitions, and statistics
• Intent/term separation: an LLM-friendly API design
• Auto-indexing: the first query triggers indexing automatically; incremental updates are transparent

⚡ Query Cache & File Watching (v1.5.0+)

• Query cache (QueryCache): in-process per-project LRU cache (50 entries by default); a hit skips the entire vector recall / rerank / expansion pipeline
• Automatic cache invalidation: the cache key is composed of normalized query + projectId + index version + search-config fingerprint, so it invalidates automatically after an index update or config change — stale results are never returned
• Watch mode: contextweaver watch watches the filesystem and triggers incremental indexing automatically, with debouncing (500ms by default) and scan de-duplication (no concurrent scans)

📈 Statistics & Observability (v1.5.0+)

• Three metric groups: indexing process, search quality/behavior, health/consistency
• Dual exits: contextweaver stats CLI (with --json) plus the MCP stats tool
• Consistency diagnostics: automatically detects abnormal migration state, pending_marks backlog, missing vector rows, and more — with suggested fixes

🛡️ Crash-Safe Data Architecture (v1.4.0+)

• Single source of truth for content: LanceDB stores only vectors and locating metadata; content is read back from files.content, reducing index size by 30–50%
• Cross-store transactional compensation: three-stage write LanceDB → FTS+outbox → SQLite mark, with automatic rollback or replay on any failure
• Migration state machine: pending/done/aborted persisted, auto-rebuilt on crash recovery
• Cross-process mutual exclusion: an advisory lock prevents the MCP server and CLI from triggering LanceDB migration concurrently
• chunk_id de-duplication: pre-delete before write to avoid duplicate rows on retry

📦 Quick Start

Requirements

• Node.js >= 20
• pnpm (recommended) or npm

Installation

# Global install
npm install -g @chiway/contextweaver

# Or with pnpm
pnpm add -g @chiway/contextweaver

Initialize Configuration

# Create the config file (~/.contextweaver/.env)
contextweaver init
# Or the short alias
cw init

Edit ~/.contextweaver/.env and fill in your API keys:

# Embedding API config (required)
EMBEDDINGS_API_KEY=your-api-key-here
EMBEDDINGS_BASE_URL=https://api.siliconflow.cn/v1/embeddings
EMBEDDINGS_MODEL=BAAI/bge-m3
EMBEDDINGS_MAX_CONCURRENCY=10
EMBEDDINGS_DIMENSIONS=1024

# Reranker config (required)
RERANK_API_KEY=your-api-key-here
RERANK_BASE_URL=https://api.siliconflow.cn/v1/rerank
RERANK_MODEL=BAAI/bge-reranker-v2-m3
RERANK_TOP_N=20

# Search parameters (optional, override built-in defaults)
CW_SEARCH_WVEC=0.6
CW_SEARCH_WLEX=0.4
CW_SEARCH_RERANK_TOP_N=10
CW_SEARCH_MAX_TOTAL_CHARS=48000
CW_SEARCH_VECTOR_TOP_K=80
CW_SEARCH_SMART_MAX_K=8
CW_SEARCH_IMPORT_FILES_PER_SEED=3

# Ignore patterns (optional, comma-separated)
# IGNORE_PATTERNS=.venv,node_modules

Index a Codebase

# Run from the codebase root
contextweaver index

# Specify a path
contextweaver index /path/to/your/project

# Force a full re-index
contextweaver index --force

Watch Mode (v1.5.0+)

# Watch for file changes and auto-index incrementally (Ctrl+C to stop)
contextweaver watch

# Specify a path and debounce window (ms)
contextweaver watch /path/to/project --debounce 800

watch runs one full incremental scan on startup, then listens to filesystem events; changes trigger a de-duplicated scan within the debounce window, and paths excluded by ignore rules never trigger a scan.

Local Search

# Semantic search
cw search --information-request "How is the user authentication flow implemented?"

# With exact terms
cw search --information-request "Database connection logic" --technical-terms "DatabasePool,Connection"

Structure Browsing & Symbol Lookup (v1.5.0+)

The following commands are CLI mirrors of MCP tools, with zero Embedding API cost:

# List indexed files (supports glob / language / count filters)
contextweaver list-files --glob "src/**/*.ts" --language typescript --max-results 100

# Look up a symbol definition
contextweaver definition SearchService --hint-path src/search

# Look up symbol references
contextweaver references handleStats --exclude-definition

Statistics (v1.5.0+)

# Human-readable stats report
contextweaver stats

# JSON output (for scripting)
contextweaver stats --json

# Specify a project path
contextweaver stats --path /path/to/project

Start the MCP Server

# Launch the MCP server (for use by Claude and other AI assistants)
contextweaver mcp

Index Management (v1.4.0+)

# Show LanceDB migration state
contextweaver migrate

# Clear the aborted state: wipe LanceDB and trigger a full rebuild
# Triggered when: the Indexer refuses to write after sampling validation fails;
# run this, then index again.
contextweaver migrate --reset

# Specify a project path
contextweaver migrate --path /path/to/project

🔧 MCP Integration

Claude Desktop Configuration

Add the following to your Claude Desktop config file:

{
  "mcpServers": {
    "contextweaver": {
      "command": "contextweaver",
      "args": ["mcp"]
    }
  }
}

MCP Tools Overview (v1.5.0+)

ContextWeaver exposes 5 MCP tools, following a layered design of "semantic retrieval first, structure browsing second":

codebase-retrieval Parameters