FYI CLI

A multi-jurisdiction Freedom-of-Information (FOI) client for Alaveteli-based platforms — track, submit, and archive official information requests across New Zealand, Australia, the UK, and any other Alaveteli deployment (foi-repository, WhatDoTheyKnow, FragDenStaat, and more), from a single privacy-focused CLI.

The project's core is a Rust workspace (fyi-core, fyi-cli, fyi-mcp); a legacy Python package (fyi_system) remains available as a reference implementation but is no longer extended.

Where fyi-cli / fyi-mcp is listed

Draft / not yet submitted: Scoop, WinGet, Homebrew, Chocolatey, GHCR multi-arch workflow, PulseMCP, mcp.so, Docker MCP Catalog, mcp-get, OpenTools, AUR, nixpkgs, Snap, Flatpak, asdf/mise, Debian/PPA, Fedora/COPR.

cargo-binstall is already assets-ready and wired in-repo, but still depends on matching GitHub Release assets before it can be treated as a live distribution path. Codex and Anthropic packets are assets-ready but still require external submission.

Full status matrix (including planned channels): docs/registry-distribution-matrix.md.

🚀 Quick Start

# Install (Rust workspace, recommended)
cargo install --path crates/fyi-cli
# or: cargo install fyi-cli   (once published to crates.io)

# Initialize
fyi-cli init-db

# Create your first request
fyi-cli register-request ministry-of-justice "OIA Request" "Request body..." --status draft

# Track and manage requests
fyi-cli list-requests

Full guide: QUICKSTART.md

✨ Features

• 🌐 Multi-Jurisdiction: Works with any Alaveteli deployment — foi-repository (NZ), WhatDoTheyKnow (UK), FragDenStaat (Germany), righttoknow.org.au (Australia), and more, with jurisdiction-aware metadata and localization on the way
• 🔒 Privacy-First: All data stored locally, optional encryption, Tor support
• 📊 Track Requests: Monitor FOI/OIA requests from creation to completion
• 🤖 Automated Monitoring: Watch instances for updates automatically
• 📈 Reports & Analytics: Generate dashboards, attention reports, handover docs
• 🔐 Secure Storage: Encrypted credentials, OS keyring integration
• 💻 CLI + MCP: Command-line tool and an MCP server for AI assistants
• 📦 Export Options: JSON, CSV, HTML export capabilities
• 🗄️ Faithful Archiving: WARC/WACZ capture of requests, correspondence, and attachments

📦 Installation

Full guide (Rust-first, all channels): INSTALL.md · version 0.1.2 · homepage github.com/edithatogo/fyi-cli

git clone https://github.com/edithatogo/fyi-cli.git
cd fyi-cli
cargo install --path crates/fyi-cli --locked
fyi-cli --help

Live listings and draft status: Where fyi-cli / fyi-mcp is listed · full matrix: docs/registry-distribution-matrix.md · MCP catalog checklist: packaging/mcp-catalogs/SUBMISSION_CHECKLIST.md.

📖 Documentation

📚 Citation and Zenodo

Release metadata is versioned in CITATION.cff, .zenodo.json, and artifacts/release/zenodo-mirror-manifest.json. DOI fields remain unset until the live Zenodo record is verified for the tagged release.

🛡️ Security

Reporting a Vulnerability

Please do NOT report security vulnerabilities through public GitHub issues.

Use GitHub's private vulnerability reporting: https://github.com/edithatogo/fyi-cli/security/advisories/new

Security Policy: SECURITY.md

Security Features

• ✅ AES-256-GCM encryption for sensitive data
• ✅ PBKDF2-HMAC-SHA256 key derivation
• ✅ OS keyring integration for credential storage
• ✅ Tamper-evident audit logging
• ✅ Secure session management
• ✅ Input validation and sanitization
• ✅ Security headers (CSP, HSTS, X-Frame-Options)
• ✅ Automated security scanning (CodeQL, pip-audit, bandit)

🧪 Testing

# Rust release checks used by this repository
cargo +stable-x86_64-pc-windows-gnu fmt --all -- --check
cargo +stable-x86_64-pc-windows-gnu clippy --workspace --all-targets --all-features -- -D warnings
cargo +stable-x86_64-pc-windows-gnu test --workspace --all-features

# Python legacy/support checks
.\.venv\Scripts\python.exe -m pytest tests/test_release_readiness.py

# Opt-in live smoke test
FYI_LIVE_SMOKE=1 .\.venv\Scripts\python.exe -m pytest -m smoke tests/test_discovery_smoke.py

Test Coverage: Rust workspace checks are the release gate. Python support tests remain available for legacy docs and archive tooling.

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

Quick Start for Contributors

# Fork and clone
git clone https://github.com/YOUR_USERNAME/fyi-cli.git
cd fyi-cli

# Build & test the Rust workspace
cargo build --workspace --all-features
cargo test --workspace --all-features

# Make your changes, then submit a PR
git commit -m "feat: Add awesome feature"
git push origin feature/awesome-feature

See CONTRIBUTING.md for the full development setup, including the legacy Python fyi_system package.

Code of Conduct

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

📊 Project Status

GitHub Projects

The project board for this repository is fyi-cli Conductor Roadmap. It mirrors the local .conductor/tracks.md registry and tracks completed Conductor work, release-readiness items, and external MCP registry follow-ups.

The umbrella Rare Insights on Open Policy from Aotearoa project is synchronized at the item level. GitHub Projects does not support nested projects, so synchronization is handled by scripts/sync_github_projects.py and the manual/scheduled project-sync workflow.

# Preview synchronization without changing either project
.\.venv\Scripts\python.exe scripts\sync_github_projects.py --dry-run

# Apply synchronization from fyi-cli project 6 into RIOPA project 4
.\.venv\Scripts\python.exe scripts\sync_github_projects.py

The workflow requires a repository secret named PROJECT_SYNC_TOKEN with GitHub Projects access for user-level ProjectsV2 writes. The sync is conservative: it adds missing fyi-cli issue/PR items to RIOPA, copies shared status values, sets the RIOPA mirror source to other unless a fyi-cli option exists, and never deletes umbrella-board items.

Project Status

This project is actively evolving from an NZ-only tool into a multi-jurisdiction FOI/Alaveteli platform built on the Rust workspace. Current work is tracked as Conductor tracks under .conductor/tracks.md, mirrored to GitHub epic issues (#37-#46) and their sub-issues. See the project board below for live status, and CHANGELOG.md for released versions.

🔗 Compatible Platforms

FYI CLI works with any Alaveteli-based platform. The embedded jurisdiction catalog currently includes:

See .conductor/tracks/jurisdiction-* for the rollout plan covering additional instances and non-English localization.

📋 CLI Commands

Archive Discovery

The archive commands are read-only and do not require an API key.

# Import the official FYI authority list into the local SQLite database
fyi import-authorities

# Walk the public search feed for a date window and save discovered requests
fyi discover --date-from 2024-01-01 --date-to 2024-02-01 \
  --checkpoint data/_state/discovery-2024-01.json \
  --output data/_state/discovered-2024-01.jsonl

# Probe a numeric request ID range for gaps
fyi discover --backfill-ids --id-from 1 --id-to 5000 \
  --output data/_state/backfill-1-5000.jsonl

# Compare feed discovery against the ID backfill
fyi discover-reconcile \
  --feed data/_state/discovered-2024-01.jsonl \
  --backfill data/_state/backfill-1-5000.jsonl \
  --output data/_state/discovery-reconciliation.json

Read-only authority discovery

discover-bodies keeps --base-url as the instance/capture URL. Use the separate --catalog-url option only when the authority CSV is hosted at an explicit alternate URL:

fyi discover-bodies --base-url https://www.righttoknow.org.au \
  --catalog-url https://catalog.example/au-authorities.csv \
  --rate-limit-name archive-discovery-au-rtk --format jsonl --output bodies.jsonl

With --format jsonl, each output line contains only the stable downstream contract fields url_name, name, and tags[]. The default JSON format also includes retrieval provenance for audit records. Both modes are read-only, robots-aware, and use the configured shared limiter.

The JSON output includes the effective catalog URL, whether the default or an override was used, retrieval time, HTTP status, row count, and a SHA-256 of the raw CSV payload. The override is read-only and still uses the catalog host's robots policy, contactable User-Agent, retry backoff, and shared rate limiter. Treat an override URL as trusted configuration: it can redirect discovery to a different authority set and should be reviewed and recorded alongside archive provenance.

Discovery uses a contactable User-Agent, checks robots.txt, and backs off on transient 429/5xx responses. Keep live runs polite: use small date windows, resume with checkpoints, and coordinate archive work with the ethics guidance in the sibling fyi-archive repo at docs/ethics-and-compliance.md.

The Rust fyi-core network path is the primary resource-aware implementation: it parses RateLimit-*/Retry-After, adapts pacing, enforces request/bytes/time guardrails, persists load memory, caches safe GETs, and can emit secret-redacted JSONL traces. Python remains a deliberately compatible discovery/capture path and follows the same identity and retry etiquette.

For concurrent workers, point discover and backfill at the same SQLite database with --db. That enables the shared cross-worker limiter, which reserves one aggregate request slot across processes and records both normal reservations and transient-failure backoff events. Inspect the current state with fyi rate-limit-status --db fyi_system.db.

CLI and MCP surfaces

The full command and server cross-reference is maintained in docs/cli-entrypoints-audit.md. It covers the Python CLI entrypoints (fyi, fyi-cli, fyi-system), the Rust CLI binary (fyi-cli), and the Rust MCP server (fyi-mcp) with its published registry pages.

Opt-in live smoke test:

FYI_LIVE_SMOKE=1 pytest -m smoke tests/test_discovery_smoke.py

The AU RightToKnow smoke is bounded to one discovery page and at most five request captures. It writes only to temporary test directories and remains disabled unless FYI_LIVE_SMOKE=1 is explicitly set.

In the current live verification, RightToKnow returned HTTP 403 for the bounded JSON search request, so the smoke records an explicit external availability skip and does not attempt capture. This does not change default offline CI behavior or retry the denied request aggressively.

Faithful Archive Capture

fyi capture stores the public request JSON, rendered HTML, and attachments as WARC records, deduplicates attachment bytes by SHA-256, and maintains a derived request view for downstream dataset tooling.

fyi capture 12345 \
  --data-dir data \
  --dist-dir dist \
  --max-bytes 500000000 \
  --max-runtime-minutes 30

Capture layout:

data/
  warc/<runid>-<request>.warc.gz
  attachments/<sha-prefix>/<sha256>
  raw/requests/<authority>/<request_id>/
    request.json
    page.html
    attachments.json
    snapshot_meta.json
dist/
  site_snapshots/<YYYYMMDD>.wacz

Each daily WACZ is appendable: subsequent captures add another WARC segment under archive/ and merge the resource metadata in datapackage.json. Replay tooling that supports WACZ/WARC can open the package from dist/site_snapshots/; for low-level inspection, unzip it and read the WARC segments with warcio.

# Database
fyi init-db                    # Initialize database
fyi config show                # Show configuration

# Requests
fyi register-request ...       # Create new request
fyi list-requests              # List all requests
fyi request-detail <id>        # View request details
fyi set-status <id> <status>   # Update status

# Submission
fyi build-prefilled-url <id>   # Generate submission URL

# Monitoring
fyi ingest-feed <url>          # Ingest RSS/Atom feed
fyi scheduler <url>            # Run continuous monitoring
fyi discover                   # Discover public FYI requests
fyi discover-reconcile         # Compare discovery JSONL outputs

# Reports
fyi dashboard --output ...     # Generate dashboard
fyi attention-report           # Generate attention report
fyi handover --output ...      # Generate handover document

# Export
fyi export-requests            # Export all requests
fyi export-bundle <id>         # Export request bundle

# Security
fyi privacy-audit              # Privacy compliance check
fyi health-check               # System health verification

Full CLI reference: See fyi --help or USER_GUIDE.md

🏗️ Architecture

┌───────────────────────────────────────────────────────────────┐
│                         fyi-cli workspace                      │
├───────────────────┬───────────────────────┬───────────────────┤
│   fyi-cli (bin)   │   fyi-mcp (MCP server) │  legacy fyi_system │
│  CLI commands,     │  JSON-RPC over stdio,   │  (Python, reference │
│  TUI dashboard      │  tools for AI assistants│  implementation)   │
├───────────────────┴───────────────────────┴───────────────────┤
│                          fyi-core (lib)                         │
│   Alaveteli API client · Tor client · encryption/keyring ·       │
│   sync engine · jurisdiction/instance registry (in progress)     │
├─────────────────────────────────────────────────────────────────┤
│                        SQLite database                          │
│      (requests, correspondence, authorities, sync_metadata)     │
├─────────────────────────────────────────────────────────────────┤
│      Alaveteli instances: foi-repository · WhatDoTheyKnow ·      │
│      FragDenStaat · righttoknow.org.au · any self-hosted deploy  │
└─────────────────────────────────────────────────────────────────┘

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Built on the Alaveteli platform by mySociety
• Inspired by the transparency work of foi-repository
• Uses cryptography for encryption

📞 Support

• Documentation: https://edithatogo.github.io/fyi-cli/
• Issues: https://github.com/edithatogo/fyi-cli/issues
• Discussions: https://github.com/edithatogo/fyi-cli/discussions

Made with ❤️ for transparency and privacy