Grounding

Local-first document corpus pipeline for grounded AI agents.

Grounding converts PDF, EPUB, DOCX, and Markdown documents into a structured, searchable corpus with per-agent embedding indexes. Drop documents into staging, get chunked Markdown with provenance hashing, FAISS vector indexes, and agent-filtered search -- all running locally, no cloud APIs required.

What It Does

Documents (PDF/EPUB/DOCX/MD)
    |
    v
[ Parse ] ── Unstructured / Marker
    |
    v
[ Chunk ] ── LangChain text splitters + YAML front matter
    |
    v
[ Hash ]  ── SHA-1 + SHA-256 + BLAKE3 provenance
    |
    v
[ Index ] ── FAISS embeddings, filtered per agent
    |
    v
[ Query ] ── Local RAG via Ollama with agentic tool calling

Key Features

• Deterministic pipeline -- same inputs produce byte-identical outputs
• Content provenance -- SHA-1, SHA-256, and BLAKE3 hashing on every document and chunk
• Agent-based corpus partitioning -- YAML-defined agents filter the corpus by collection tags, each with their own FAISS embedding index
• Persona system -- agents have configurable communication styles, expertise areas, and greeting messages
• Staging watcher -- drop files into a folder, auto-ingest with embedding updates
• Multi-machine ready -- optional Syncthing-based architecture for dedicated ingestion servers
• Fully local -- no cloud APIs, no telemetry, your documents stay on your machine
• Agentic RAG -- local LLMs autonomously decide when to search the corpus via tool calling

Why Grounding? (vs. LlamaIndex / Haystack / txtai)

Grounding is an opinionated pipeline, not a framework:

• Per-agent FAISS partitioning as a first-class primitive. Each agent YAML defines a corpus slice and gets its own index -- no single monolithic index where unrelated domains compete for top-k. Other frameworks treat this as a filter you bolt on at query time; here it's the core data model.
• Determinism and provenance for citable output. Every chunk carries SHA-1 / SHA-256 / BLAKE3 hashes, page ranges, and section headings; same inputs produce byte-identical chunks. Built for agents that must cite sources.
• Local-first by construction. No cloud APIs, no telemetry. Runs against Ollama or any OpenAI-compatible local server. Cloud isn't a default path -- it's not a path at all.
• Opinionated fixed pipeline (parse → chunk → hash → index → query). Less surface area than a framework, less to configure, less to break.

Use LlamaIndex / Haystack if you want a framework to assemble custom retrieval flows. Use grounding-ai if you want a citation-grade local RAG pipeline with per-agent separation, working today.

Quick Start

Install from PyPI

python3 -m venv venv   # Python 3.10-3.13 supported
source venv/bin/activate
pip install grounding-ai

Then grab the example configs and agents from the repo:

curl -O https://raw.githubusercontent.com/andyliszewski/grounding-ai/main/config.example.yaml
curl -O https://raw.githubusercontent.com/andyliszewski/grounding-ai/main/.mcp.example.json
mkdir -p agents && cd agents && \
  curl -O https://raw.githubusercontent.com/andyliszewski/grounding-ai/main/agents/examples/scientist.yaml && \
  cd ..
cp config.example.yaml config.yaml
cp .mcp.example.json .mcp.json

Install from source (for development)

git clone https://github.com/andyliszewski/grounding-ai.git
cd grounding-ai
python3 -m venv venv
source venv/bin/activate
pip install -e .
cp config.example.yaml config.yaml
cp .mcp.example.json .mcp.json
cp agents/examples/*.yaml agents/

First run (end-to-end)

# 1. Ingest some documents
grounding ./my-documents ./corpus --collections science

# 2. Generate embeddings for the scientist agent
grounding embeddings --agent scientist --corpus ./corpus

# 3. Query with a local LLM (requires Ollama running)
python scripts/local_rag.py --agent scientist -A

A typical session looks like this:

$ python scripts/local_rag.py --agent scientist -A
🔬 Scientist agent ready (3,142 chunks indexed across 8 collections)

> What does Kuhn mean by a paradigm shift?

[searching corpus... 5 chunks retrieved]

A paradigm shift, in Kuhn's framing, is a discontinuous change in the
fundamental assumptions of a scientific community [Source: Kuhn, The
Structure of Scientific Revolutions, corpus]. It happens when accumulated
anomalies can no longer be explained within the existing paradigm and a
new framework displaces the old one — not through gradual refinement but
through a gestalt-like reorientation.

[Derived] The process is social as much as epistemic: Kuhn emphasizes that
competing paradigms are often incommensurable, meaning proponents of each
literally see the world differently.

Agent System

Agents are YAML files that define a persona and a corpus filter:

name: scientist
description: Scientific research agent

persona:
  icon: "🔬"
  style: |
    You communicate like a rigorous scientist: analytical,
    evidence-based, and methodical.
  expertise:
    - Scientific method and experimental design
    - Biology and biochemistry
    - Physics fundamentals
  greeting: |
    I'm your scientific advisor. What would you like to investigate?

corpus_filter:
  collections:
    - science
    - biology
    - chemistry
    - physics

Each agent gets its own FAISS embedding index containing only documents matching its collections. See agents/examples/ for starter templates.

Creating Your Own Agents

1. Define the agent. Create a YAML file in agents/:

   yaml

   Copy

   # agents/my-agent.yaml
   name: my-agent
   description: What this agent knows about
   
   persona:
     icon: "🎯"
     style: |
       How you want the agent to communicate.
     expertise:
       - Domain area 1
       - Domain area 2
     greeting: |
       Message shown when the agent activates.
   
   corpus_filter:
     collections:
       - collection-tag-1
       - collection-tag-2
   

2. Ingest documents with matching collection tags. Collections are kebab-case labels you assign when ingesting:

   bash

   Copy

   grounding ./physics-textbooks ./corpus --collections physics
   grounding ./biology-papers ./corpus --collections biology,science
   

   A document can belong to multiple collections (comma-separated). An agent sees all documents whose collection tags overlap with its corpus_filter.collections list.

3. Generate embeddings for the agent:

   bash

   Copy

   grounding embeddings --agent my-agent --corpus ./corpus
   

   This builds a FAISS index at embeddings/my-agent/ containing only chunks from documents matching the agent's collection filter.

4. Query the agent:

   bash

   Copy

   python scripts/local_rag.py --agent my-agent -A
   

Where Do My Agent Files Live?

By default, agents/*.yaml is gitignored (only agents/examples/ is tracked). This means YAMLs you create in agents/ won't show up in git status and won't be committed to your fork. There are three common workflows depending on how you want to manage your agents:

Workflow A: Local-only (simplest, no version control)

Just create YAMLs in agents/ and use them. Nothing extra to manage.

cp agents/examples/scientist.yaml agents/my-physicist.yaml
# Edit, then use immediately
grounding embeddings --agent my-physicist --corpus ./corpus

Good for: Trying things out, single machine, agents you don't need to back up.

Workflow B: Separate private agents repo (recommended for serious use)

Create your own private repo for agent definitions and point AGENTS_DIR at it. This is how the maintainer runs grounding -- agents are version-controlled and sync between machines via git.

# Create a private repo with this structure:
#   my-agents/
#   ├── agents/
#   │   ├── physicist.yaml
#   │   └── biologist.yaml
#   └── commands/         # Optional: Claude Code slash commands

# Clone it alongside grounding-ai
git clone git@github.com:youruser/my-agents.git ~/my-agents

# Point grounding at it
grounding embeddings --agent physicist --corpus ./corpus --agents-dir ~/my-agents/agents

# Or set the environment variable for the staging watcher
export AGENTS_DIR=~/my-agents/agents

Good for: Multi-machine setups, version-controlled agent definitions, keeping personal agents private while contributing back to grounding-ai.

Workflow C: Fork grounding-ai

Fork the repo and remove agents/*.yaml from .gitignore. Your agents become part of your fork.

# After forking
sed -i '' '/agents\/\*\.yaml/d' .gitignore   # remove the gitignore line
git add agents/ .gitignore
git commit -m "track personal agent definitions"

Good for: Single-repo workflow, public agent libraries, contributing agent templates back upstream.

Organizing Collections

Collections are free-form tags -- there's no predefined list. Choose whatever makes sense for your domain:

staging/
├── physics/           # Collection: physics
├── biology/           # Collection: biology
├── game-theory/       # Collection: game-theory
└── machine-learning/  # Collection: machine-learning

One agent can span many collections (a "scientist" agent might include physics, biology, and chemistry). Multiple agents can share the same collections. The agent YAML is the only thing that defines which slices of the corpus each agent can search.

Project Structure