Research Repo Doctor

Get your research artifact ready for Artifact Evaluation before the deadline: scan the repo, scaffold the easy fixes, verify the run path, and generate the appendix.

Web demo for public repos: https://research-repo-doctor-bckncrcwwmg6jrbsrd6btj.streamlit.app/ If the hosted demo is waking up or reinstalling dependencies, the local zero-install command below is the reliable fallback:

uvx rrdoctor scan .

Maintainers can check anonymous demo access with python scripts/check_live_demo.py https://research-repo-doctor-bckncrcwwmg6jrbsrd6btj.streamlit.app/.

rrdoctor is a local CLI and GitHub Action for research artifact preparation. It audits whether a repo is reviewable, citable, and close to runnable; scaffolds safe mechanical fixes; maps findings to an AE-style readiness level; and turns the rest into a checklist any coding agent or human can finish.

Make it a pull-request gate

The GitHub Action is the main adoption path: run the CLI while preparing the artifact, then keep the same deterministic preflight on every pull request.

name: Reproducibility preflight

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  rrdoctor:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: Tom409114/research-repo-doctor@v0.2.24
        with:
          profile: standard
          fail-on: error

The full GitHub Action guide covers sticky PR comments, new-finding baselines, agent plans, appendices, and complete AE prep packets.

AE deadline loop

uvx rrdoctor prepare . --profile acm --out-dir rrdoctor-prep

# Or run the pieces explicitly:
uvx rrdoctor scan . --profile acm
uvx rrdoctor fix . --write
uvx rrdoctor appendix . --profile acm --output ARTIFACT_APPENDIX.md
uvx rrdoctor verify . --profile acm
uvx rrdoctor verify . --profile acm --run --timeout 600 --fail-on error  # trusted repos only
# Or pin the official quickstart command as the dynamic gate:
uvx rrdoctor verify . --profile acm --command "python train.py config/default.py" --run --timeout 600 --fail-on error

For trusted repositories, rrdoctor verify --run can go beyond static checks. For supported Python repositories it creates a temporary isolated environment, installs declared dependencies, and executes the declared entrypoint there under a timeout. Other ecosystems retain an explicit resolver preflight. With the default gate (--fail-on error), failed or blocked dynamic L2/L3 steps return a nonzero exit code. Use --command when the artifact has a specific smoke-test or quickstart command that reviewers should run. Dynamic mode may run dependency build/install hooks as well as the entrypoint, so never use it on untrusted code. rrdoctor prepare writes the report, agent plan, artifact appendix, and verification ladder into one local evidence directory.

Artifact Evaluation chairs and lab maintainers can use the AE chair guide for optional pre-submission wording and CI examples.

For public calibration evidence, see the 80-repository data brief and the nanoGPT first-run regression case study.

What it catches

• "Your --seed flag does nothing." RRD052 spots code that declares a seed option but never calls random.seed, np.random.seed, torch.manual_seed, tf.random.set_seed, or random_state=seed.
• "This worked on my laptop." Local-only data paths, missing data provenance, and undocumented retrieval steps.
• "The environment silently changed." Unpinned dependencies, missing runtime versions, undeclared imports, and absent dependency manifests.
• "The notebook lies." Stale outputs, out-of-order execution, checkpoint artifacts, and secret-like notebook output.
• "Reviewers cannot tell how to cite or rerun this." Missing license, citation, CI, tests, changelog, results provenance, or experiment entrypoint.

Install

Run once, without installing:

uvx rrdoctor scan .

Alternatives:

pipx run rrdoctor scan .
pip install rrdoctor
rrdoctor scan .

Developer install from source:

git clone https://github.com/Tom409114/research-repo-doctor.git
cd research-repo-doctor
python -m pip install -e ".[dev]"
rrdoctor scan .

Fix the easy gaps

Let rrdoctor create the safe scaffolding for you. It is deterministic, idempotent, and never overwrites existing files.

rrdoctor fix . --write

It can scaffold missing governance docs, citation metadata, data/results provenance notes, a reproducible-seed helper, changelog entries, and common research .gitignore entries. The hard parts become a reviewable plan:

rrdoctor plan . --output plan.md

Use with your coding agent

Paste this into Claude Code, Cursor, GitHub Copilot, or any other coding agent:

Use rrdoctor as the deterministic, offline, no-API-key grader for this research repo.

Run:
rrdoctor scan . --format json --output baseline.json
rrdoctor plan . --output plan.md

Work through plan.md without weakening rrdoctor checks.

Definition of done:
rrdoctor scan . --baseline baseline.json --fail-on-new error

The final command is the objective gate: it verifies the agent's work against the starting baseline and fails only on newly introduced errors.

Copyable agent templates are available for Agent Skills / Claude Code-style workflows and Cursor project rules under integrations/.

Keywords: research software, reproducibility, artifact evaluation, repository audit, auto-fix, coding agents, AGENTS.md, GitHub Action, notebooks, data availability, citation metadata.

Help calibrate the rules

The fastest way to improve rrdoctor is real scan feedback from real research repositories. After one scan, the low-friction 10-minute trial report asks only what was useful, wrong or missing, and whether the tool fits a real deadline. For a focused rule report, please open a false-positive, false-negative, scan-case, or new-rule issue. Include the rule ID, command, rrdoctor version, and a sanitized minimal repo shape. See feedback and calibration for the short checklist.

Why this matters

Research code often lands on GitHub under deadline pressure. A reviewer or future lab member finds a promising repository and then loses hours because the environment is underspecified, data paths are local, notebooks contain stale outputs, dependencies are unpinned, or the citation is unclear.

Research Repo Doctor turns those recurring release blockers into deterministic checks with concrete remediation - and, where it is safe to do so, scaffolds the mechanical starting points. It is built to sit in the ordinary maintenance path: run locally while preparing a release, then run automatically on pull requests through GitHub Actions.

The audit runs without an AI API key, network access, or hosted service. That same determinism makes it an honest grader: it can verify fixes made by a person or a coding agent.

audit -> fix -> plan -> (your coding agent / you) -> verify -> PR
  |       |       |                                  |
  |       |       rrdoctor plan                      rrdoctor scan --baseline
  |       rrdoctor fix --write                       --fail-on-new error
  rrdoctor scan

What's new in 0.2.24

• Better systems-artifact coverage: existing checks now recognize Cargo, CMake, Meson, container, and Nix evidence plus documented artifact scripts and Cargo commands across entrypoint, test, CI, and release workflows.
• Quieter static scans: target-source SyntaxWarning noise is suppressed, and escaped Python exception text no longer looks like a local Windows path.
• A rechecked first-run trust case: public PyPI rrdoctor==0.2.23 still scores nanoGPT as Functional at 76/100 with 0 errors, while the original noisy RRD050 and RRD063 findings remain absent.
• An explicit JOSS boundary: the draft-paper notes now separate the more-than-six-month public-history gate from research-use, authorship, conflict-of-interest, and AI-disclosure evidence that maintainers must supply.
• A configuration file that actually governs the scan: CLI and MCP calls load the target repository's .rrdoctor.yml from any working directory; profile, report, threshold, failure, include/exclude, and rule settings now work beneath explicit CLI overrides.

What's new in 0.2.23