Inferrail

Know what your AI work costs.

Inferrail turns supported OpenAI chat-completion traffic into local, attributable economic receipts. Give related requests a customer-defined work_id, declare an outcome when your application knows one, and inspect the known inference economics associated with that work without storing prompts, responses, or tool payloads in Inferrail's own records.

For the supported chat-completions surface, Inferrail records known cost when measured usage and a verified price are available. Otherwise it reports unknown, never a fabricated $0.

30-second demo

Current main / upcoming Work Economics release. Work Economics was added after the current PyPI release. To try the current product before the next release, install from main:

pip install "inferrail @ git+https://github.com/domondi1/inferrail.git@main"
inferrail demo

The demo needs no API key, no network call, and no provider billing. It runs canned requests through Inferrail's real engine with made-up prices labeled DEMO, then shows receipts, attribution, work-level economics, and explicit unknown evidence.

Stable PyPI release. pip install inferrail currently installs 0.1.2. It includes the gateway, receipts, reports, and TaskTransaction, but not the new work commands. It remains the stable released install until the next package publication.

What just happened?

AI request
  -> InferenceReceipt
  -> caller-supplied attribution
  -> related requests share work_id
  -> customer-declared outcome
  -> Work Economics

• Receipt: one inference request produced payload-free economic evidence.
• Attribution: the caller can attach identifiers such as customer, workflow, or project.
• Work: several requests can share a work_id that your application defines.
• Outcome: your application can append a declaration of what happened to that work.
• Work Economics: Inferrail joins that declaration with matching receipts and reports known attributed inference economics for the work.

You decide what a unit of work means: a contract review, support resolution, coding task, research run, or document-processing job. Inferrail associates economic evidence with the identifier your application supplies; it does not interpret the business meaning of that identifier or its outcome.

Request economics vs. Work Economics

Request economics: what known inference economics belong to one request?

Work Economics: what known inference economics belonged to the customer-defined unit of work those requests were performing?

This is not a full cost of work, COGS, margin, or business-value calculation.

Track a unit of work

The following uses real provider requests and requires OPENAI_API_KEY:

export OPENAI_API_KEY=<your-openai-api-key>

inferrail try "Review this contract clause" \
  -a work_id=contract_review_42

inferrail try "Identify remaining risks" \
  -a work_id=contract_review_42

inferrail work outcome contract_review_42 --status completed
inferrail work contract_review_42
inferrail work --all

For a gateway client, the equivalent generic attribution header is:

X-Inferrail-Attribute-Work-Id: contract_review_42

The deterministic offline demo includes this synthetic example:

work-contract-1
  2 inference receipts
  customer-declared outcome: resolved
  known attributed inference cost: $0.000483

resolved is only the demo application's own outcome meaning. Inferrail does not treat any outcome status as universally successful.

If Inferrail cannot verify the price for an observed inference event, its cost remains unknown rather than being treated as zero. No receipt evidence is also not the same thing as known zero cost.

First real request and reports

inferrail try is the shortest route to one real receipt. It uses your existing OPENAI_API_KEY; if it is not set, Inferrail prints what is required. It prints the response, receipt, measured tokens, known cost or unknown, the local receipt path, and the next report command.

inferrail try "Reply with one word: ready" --customer acme
inferrail report
inferrail report --by customer
inferrail report --by workflow
inferrail report --by provider

What a receipt contains

One payload-free JSON receipt per supported request:

{
  "receipt_id": "ir_1e6c916bac8940ca8a85",
  "provider": "openai",
  "model": "gpt-4o-mini",
  "prompt_tokens": 842,
  "completion_tokens": 191,
  "estimated_cost_usd": "0.000241",
  "attributes": { "customer": "acme", "workflow": "contract-review" }
}

(Trimmed — the full record also carries pricing provenance, status, route, timestamp, latency, and retry count. See Privacy boundary below for the complete shape.)

TaskTransaction: receipt-only task grouping

One task is rarely one call. Tag every request belonging to one unit of work with the same attribution value, then ask Inferrail what the task cost:

export OPENAI_API_KEY=<your-openai-api-key>
inferrail try "Reply with one word: ready" -a task_id=bug_9281
inferrail try "Summarize: the retry patch is deployed" -a task_id=bug_9281
inferrail transaction bug_9281

Task:        bug_9281
Transaction: tx_72fcfcca9ede9d2facc3
Status:      success

EVENT TYPE  EVENT ID                 STATUS   COST
inference   ir_f6fb6403d5324ea0acf9  success  $0.000003
inference   ir_756cc072a27f42f4a2ea  success  $0.000007

Known total cost: $0.00001

This TaskTransaction example uses real provider requests and a task_id. The offline demo instead correlates requests with work_id and shows Work Economics. Over HTTP, an X-Inferrail-Attribute-Task-Id: bug_9281 header does the same thing; inferrail.track_task(task_id=...) (see Attribute spend below) attaches it automatically to every nested call in an agent run, no header-threading required. See docs/adr/0008.

Use it as a gateway

For a long-running application, start the separate gateway process. The gateway process must have access to the provider credential through the configured environment variable; a key held only inside application memory is not automatically transferred to the gateway.

inferrail serve --quickstart

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Inferrail-Attribute-Customer: acme" \
  -d '{
    "model": "default",
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'

The response is standard OpenAI choices/usage plus a non-standard inferrail block (route, provider, latency, retries) any OpenAI client already ignores. X-Inferrail-Attribute-* headers are optional attribution — never forwarded upstream. See examples/basic_chat_request.py for a minimal Python client, or point a supported OpenAI-compatible chat client at http://127.0.0.1:8000/v1. An OpenAI SDK client that does not set base_url can use its existing OPENAI_BASE_URL environment mechanism instead.

The default receipt is one JSONL line per supported request in ./inferrail-receipts.jsonl, relative to the gateway's working directory. Treat that file as machine/audit evidence; use inferrail report for the human aggregate, inferrail transaction <task-id> for receipt-only task grouping, and inferrail work <work-id> for work-attributed inference economics plus a customer-declared outcome.

# LangChain
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="not-needed",  # or your INFERRAIL_GATEWAY_TOKEN if auth is enabled
    model="default",
)

# LlamaIndex
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="default",
    api_base="http://127.0.0.1:8000/v1",
    api_key="not-needed",
    is_chat_model=True,
    context_window=8192,
)

# CrewAI
from crewai import LLM

llm = LLM(
    model="openai/default",  # "openai/" prefix required by CrewAI
    base_url="http://127.0.0.1:8000/v1",
    api_key="not-needed",
)

"model" normally selects a named route from inferrail.yaml (e.g. "default"), which maps to a provider + underlying model. If default_provider is set in your config, a model that matches no route is instead forwarded to that provider unchanged — so "model": "gpt-5.6-sol" works with no route pre-registered for it. Named routes always take priority. This passthrough is on by default for the zero-config quickstart path, off by default otherwise. Full design: docs/adr/0007.

Attribute spend

Three ways to attach business context to a request, all landing in the same attributes: dict[str, str] on its receipt:

• HTTP header (gateway): X-Inferrail-Attribute-<Name>: <value>, e.g. X-Inferrail-Attribute-Task-Id: bug_9281.
• CLI flag (inferrail try): --customer/--workflow shorthand, or generic -a <name>=<value> for anything else, including task_id.
• Ambient, for nested agent calls: inferrail.track_task attaches X-Inferrail-Attribute-Task-Id to every outgoing request for the duration of a with block or decorated function — no threading a task_id parameter through nested function signatures by hand.

import inferrail
from openai import OpenAI