InferBench

Every "best local LLM engine" article benchmarks someone else's machine. InferBench benchmarks yours.

Local-inference engines all publish their own benchmarks, on their own hardware, in their own README. None of them tell you which one is actually fastest on the machine sitting in front of you. InferBench runs a fixed, varied prompt set against whichever supported engines are installed on your own hardware and reports real, measured tokens/second -- not a number copied from someone else's blog post.

Install, first run, and a real omlx benchmark against a cached model:

npx inferbench-cli run --engines llama.cpp --model "bartowski/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M"

Table of contents

• Install
• Features
• Quickstart
• CLI command reference
• Library API reference
• How the measurement works
• Comparison
• Why this exists
• Documentation
• FAQ
• Contributing
• Security
• License

Install

InferBench ships two independent, equally first-class packages -- pick whichever fits your toolchain, or install both. Neither is deprecated in favor of the other; both run the same measurement architecture against the same two supported engines.

# npm -- JavaScript/TypeScript CLI
npm install -g inferbench-cli
# or, no install:
npx inferbench-cli run --engines llama.cpp --model "<repo>:<quant>"

# PyPI -- Python CLI + library (genuine port, not a wrapper around the Node binary)
pip install inferbench-cli

Both packages are published and installable today. npm install -g inferbench-cli and pip install inferbench-cli both work -- see npmjs.com/package/inferbench-cli and pypi.org/project/inferbench-cli, or python/README.md and docs/getting-started.md for the Python-specific walkthrough, and CHANGELOG.md for each distribution's version history.

Requires Node.js >=18 for the npm package, Python >=3.9 for the PyPI package. At least one supported engine must already be installed either way (InferBench does not install engines for you):

• llama.cpp: brew install llama.cpp (macOS) or build from ggml-org/llama.cpp
• omlx: brew tap jundot/omlx https://github.com/jundot/omlx && brew install omlx (Apple Silicon only)

Features

• Cross-engine, same measurement code. InferBench starts each engine's own OpenAI-compatible HTTP server (llama-server, omlx serve) and sends every engine the identical prompt set through the identical timing code, instead of comparing numbers each engine's own benchmark tool produced differently.
• Full-response-body timing, not headers. An earlier version of this code measured elapsed time right after the HTTP response object resolved, which only captures headers arriving, and once reported a physically impossible 64,646 tok/s before the bug was caught. Both distributions now time the complete response body, with a regression test guarding the fix in each language's harness.
• 8-prompt fixed sweep with warm-up. One throwaway completion absorbs first-request latency, then 8 varied prompts are timed individually and reported as avg/min/max tok/s (n=8 in the results table).
• Two independently maintained distributions, matching output. npm's inferbench-cli (TypeScript) and PyPI's inferbench-cli (a genuine Python port, not a wrapper around the Node binary) expose the same CLI flags and the same JSON report field names.
• Machine-readable reports. --json / --out <file> writes a full BenchmarkReport as camelCase JSON on both distributions, so CI or an agent can parse it without special-casing which language produced it.
• Local cloud-cost context (Python library). compare_to_cloud() looks up a static, dated cloud API price alongside your measured local throughput -- it discloses plainly that it's a snapshot, not a live quote, and returns None for a model it doesn't recognize rather than guessing a number.
• Path-safe --out. A relative --out value that resolves outside the current working directory is rejected, so an agent-supplied output path can't escape the intended directory.

Quickstart

# llama.cpp -- pass a Hugging Face repo spec; llama.cpp downloads and
# caches it automatically, no manual step required
inferbench run --engines llama.cpp --model "bartowski/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M"

# omlx -- pass the model-directory subdirectory name under ~/.omlx/models/;
# omlx has no CLI download flow, so the model must already be present there
# (download it once via `omlx`'s own admin dashboard, or huggingface_hub's
# snapshot_download into that directory)
inferbench run --engines omlx --model "qwen2.5-1.5b-instruct-4bit"

# Both installed engines, machine-readable output, saved to a file
inferbench run --model "<spec>" --json --out report.json

Real output from a live run against an actual llama-server process:

$ inferbench run --engines llama.cpp --model "bartowski/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M"
Hardware: Apple M4 (darwin/arm64), 16GB

llama.cpp: starting server...
llama.cpp: warming up...
llama.cpp: [1/8] benchmarking...
...
llama.cpp: [8/8] benchmarking...

Results:
  llama.cpp: avg 75.54 tok/s (range 69.54-79.78, n=8)

Recommendation: llama.cpp -- highest measured throughput on this run (75.54 tok/s avg) -- specific to this hardware and model, not a universal ranking

[!WARNING] --model means something different per engine (a downloadable HF spec for llama.cpp, a pre-downloaded local directory name for omlx), because the two engines have genuinely different model-acquisition capabilities -- omlx's serve command has no flag to pull an arbitrary model from Hugging Face directly. Running both engines against the same model in one command therefore needs the model already available in both engines' own expected forms.

CLI command reference

inferbench run [options]

Options:
  --model <spec>    Model spec (engine-specific, see Quickstart above)   [required]
  --engines <list>  Comma-separated engines to test (default: all installed --
                    omlx, llama.cpp)
  --max-tokens <n>  Max completion tokens per prompt (default: 200)
  --json            Output machine-readable JSON instead of a human table
  --out <file>      Also write the full JSON report to this file
  --verbose         Show raw engine server stdout/stderr

Exit code 0 on a successful run with at least one engine tested; 1 on a usage error or when no supported engine is installed. The Python CLI has one small, documented divergence: a missing required --model flag exits 2 (the standard argparse convention for a parse-time error) instead of 1.

Library API reference

The Python package (pip install inferbench-cli) exposes a documented library surface, meant for use in scripts or notebooks instead of the CLI. The npm package's package.json main field points at the CLI script itself (dist/cli.js, which runs the argument parser as a side effect on import) and does not declare a separate library entry point, so today only the Python distribution is a supported library import.

from inferbench import (
    benchmark_engine, detect_hardware, all_engines, resolve_engines,
    recommend, compare_to_cloud, report_to_dict, write_json_report,
)

Read the full README on GitHub →