Lm README
The full upstream README, mirrored here for reference. Install config, tool schemas, adoption signals, and an original overview live on the Lm listing page.
Houtini LM (@houtini/lm) - Offload Work from Claude Code to a Local LLM, OpenAI GPT-5/6, a Router or a Cheaper Cloud Model
Houtini LM is an MCP server that lets Claude (or any MCP client) hand bounded work to another model - a local LLM on your GPU, OpenAI's latest GPT models, a LiteLLM router, OpenRouter or a cheap cloud API - while you carry on working in the AI platform you already like. It cuts your token bill, and it gives you a second model to review your code whenever you want one.
Quick Navigation
What's new | Why use it | Install | How it handles different models | What to hand over | Tools | Reading the footer | Configuration | Endpoints | The manual
I built this because I kept leaving Claude Code running overnight on big refactors and the token bill was painful. A huge chunk of that spend went on bounded tasks any decent model handles fine - generating boilerplate, code review, commit messages, format conversion, the sort of work that doesn't need Claude's reasoning or its tool access.
So Claude stays the architect, doing the planning, the multi-file changes and the judgement calls, and houtini-lm passes the drafting to whatever model you've got running. That could be Qwen on a GPU box under your desk, GPT-5 or GPT-6 straight from OpenAI, a model behind a LiteLLM router, one of OpenRouter's 300+ models or DeepSeek at pennies per million tokens. Claude QAs everything that comes back.
I wrote a full walkthrough of why I built this and how I use it day to day if you'd like the longer story.
What's new in 3.3
houtini-lm now speaks to far more than a local GPU. Point it at OpenAI directly and GPT-5, GPT-6 and the o-series reasoning models work without any router in between. They reject parameters every open model accepts (max_tokens, temperature), so houtini-lm sends them max_completion_tokens only, learns a model's output cap from its own error when the endpoint doesn't report it, and leaves the image, speech and moderation models out of the list. Point it at a LiteLLM router and it reads which real model sits behind each alias, along with that model's true context window and output cap, so a mixed fleet of local and hosted models is sized and profiled correctly. Thinking is now your call (auto, off or on), there's a Docker guide built from a working deployment, and the manual has a page per job. The changelog has the detail.
Why use houtini-lm?
The bottom line before we go deep: you keep your favourite AI platform, and you stop paying frontier prices for work that doesn't need a frontier model.
The obvious win is cost. When Claude delegates a review with code_task_files, the source files are read by the houtini-lm process and sent straight to the other model, so they never enter Claude's context window at all. Claude sends a short tool call and reads back a short answer. I benchmarked this on real TypeScript source files:
| Task | Claude reads it directly | Delegated | Saved |
|---|---|---|---|
| Code review (1,352 lines) | 14,466 tokens | 769 tokens | 95% |
| Architecture review (2,022 lines) | 20,014 tokens | 983 tokens | 95% |
| External repo review (581 lines) | 5,344 tokens | 741 tokens | 86% |
| Code explanation (833 lines) | 8,678 tokens | 744 tokens | 91% |
That averages out at 93.3% saved across the session. To be fair, small tasks like a one-line question or a commit message don't save much, because the tool call overhead (around 250 tokens) is about the same size as the answer. Anything that involves reading files, which is most of a real coding session, pays for itself straight away. You can run the same benchmark against your own setup with LM_STUDIO_URL=http://your-server:1234 node scripts/benchmark.mjs.
The less obvious win is a second opinion. A different model reads your code with different blind spots, and it costs you next to nothing to ask. As it happens, the 3.3.0 release of this repo is a decent example: I had Claude point houtini-lm at gpt-6-astra (through my LiteLLM router) and ask it to review the release's own 14,000-token diff. It came back with three real bugs - a router probe that cached a temporary 429 as a permanent "not a router", an output cap that got overwritten, and a race in the model list cache - and all three were fixed before the release shipped. Two of the three new test files were drafted the same way, then reviewed by Claude before commit.
Code review is where this pays off hardest, because reviews are exactly the bounded, read-a-lot-write-a-little work that a cheaper model does well. From there the list keeps growing: test stubs, docstrings, commit messages, changelog drafts, format conversion, mock data, type definitions, embeddings for a RAG pipeline, a quick sanity check on a regex, brainstorming three approaches before Claude commits to one, and so on.
The trade-off is wall-clock time. Local inference is typically 3-30x slower than a frontier model, so delegation wins on bounded, self-contained tasks rather than everything. A local model keeps your code private and costs nothing per token, a cloud model is cheap, and neither touches your Claude quota or its rate limits.
How it works
Claude's the architect, the other model's the drafter, and Claude checks everything that comes back.
Install
You'll need Node 22.5 or newer and an OpenAI-compatible endpoint: LM Studio, Ollama, vLLM, SGLang, a LiteLLM router, or a cloud API key for OpenAI, DeepSeek, Groq and the like. In Claude Code, with LM Studio running on the same machine, it's one command:
That's it. LM Studio listens on localhost:1234 by default, which is where houtini-lm looks first, so Claude can start delegating straight away. Anywhere else, set the URL (and a key, if the endpoint needs one):
OpenAI works the same way. Pin the model you want, because OpenAI lists dozens and they all score the same in routing:
Installing houtini-lm walks through every route: a GPU on another machine, OpenAI and other cloud APIs, OpenRouter, a LiteLLM router, Claude Desktop and other MCP clients, plus how to check it worked and how to update. If you'd rather run it in a container, Running houtini-lm in Docker covers both a plain docker run -i and serving it over HTTP behind Docker's MCP Gateway. New to local models altogether? Start with Getting started, which covers which models fit on 16, 32, 64, 96 or 128 GB of VRAM.
To check everything's wired up, ask Claude to run houtini-lm's discover tool. It tells you the version, which endpoint it found, which model is active and how big its context window is.
How houtini-lm handles different models
No two open source LLMs are the same. They differ in context window, output cap, prompt template, whether they think before they answer and how much of that they report, so a lot of houtini-lm's code is about working out what it's talking to and adjusting for it. How houtini-lm handles different models has the full detail, and here's the short version.
At startup houtini-lm asks your server for every model it has, loaded and downloaded, and looks each one up on HuggingFace for its architecture, licence and chat template, caching the lot in SQLite so later startups are instant. For the families I know well (Qwen, Nemotron, Granite, LLaMA, GLM, GPT-OSS, DeepSeek, Gemma, Kimi and OpenAI's hosted GPT models) there's a curated profile, and each family gets its own temperature, output constraints and think-block handling, while OpenAI's hosted reasoning models (GPT-5/6, o-series) are sent only the parameters they accept. Run list_models and you get the whole picture:
Output budgets come from the model each call is actually sent to. Leave max_tokens unset and the call gets 25% of that model's context window, never more than its declared output cap or the room left beside your prompt, so a hosted model doesn't get sent a request it'll reject. There's a floor too, because MCP clients habitually pass tiny caps like 256 that strangle reasoning models mid-thought, so anything under 4,096 is ignored unless you set HOUTINI_LM_MIN_TOKENS=0.
Thinking is your decision, through HOUTINI_LM_THINKING. The default, auto, switches thinking off for models detected as supporting the toggle (Qwen3, Gemma 4, Nemotron, DeepSeek R1, GLM-4, gpt-oss), which suits Claude doing the reasoning and the other model doing the drafting. off forces that on every call, which you need when a backend serves a thinking model under a name detection can't recognise. on forces thinking on, which is worth it for bug-hunting or checking an argument, at the cost of time and tokens. Whichever you choose, houtini-lm inflates the output budget for thinking models and strips any <think> blocks from the answer, so the reasoning doesn't leave you with an empty reply.
With several models available, houtini-lm scores each against the task type and picks the best, and it suggests a better model rather than swapping one in, since loading a model takes minutes. On a big catalogue every unknown model scores the same and the first listed wins, so pin one with HOUTINI_LM_MODEL, or pass model on an individual call.
Point it at a LiteLLM router and it reads /model/info as well, which tells it the real model behind each alias (my local alias is qwen3.6-27b) and, for hosted models, the true context window and output cap. Each alias is then profiled and sized as the model it actually is, the TTS, image and video models a router lists by the dozen are filtered out, and rate-limit errors are retried with backoff. Calls are queued one at a time by default, because a single GPU can only serve one request anyway; OpenRouter skips the queue, and for OpenAI, a router or a batching backend like vLLM you can turn it off with HOUTINI_LM_SERIALISE=0.
What to hand over
The best candidates are bounded and well defined, with a clear input and a clear output:
| Task | Why it works on another model |
|---|---|
| Code review | Paste the full source (or pass the paths), ask for bugs |
| A second opinion on a plan | Doesn't commit to anything, costs next to nothing |
| Generate test stubs | Source in, tests out |
| Explain a function | Summarisation doesn't need tool access |
| Draft commit messages | Diff in, message out |
| Convert formats | JSON to YAML, snake_case to camelCase |
| Generate mock data | Schema in, data out |
| Write type definitions | Source in, types out |
| Structured JSON output | Grammar-constrained, valid by construction |
| Text embeddings | Semantic search, RAG pipelines |
Anything that needs reasoning across the codebase, tool access or multi-step orchestration stays on Claude: architectural decisions, reading and writing files, running tests and interpreting the results, multi-file refactoring plans and anything that has to call other tools. The tool descriptions are written to nudge Claude into planning delegation at the start of a big task, rather than only using it when it happens to remember.
The tools
There are eight of them. The full parameter reference is in The tools, in depth.
| Tool | What it's for |
|---|---|
chat | The workhorse. Send a task, get an answer. |
custom_prompt | System, context and instruction kept separate, which consistently beats stuffing everything into one message on local models. I tested this properly one weekend with the same batch of review tasks run both ways, and the three-part version won every round. |
code_task | Code analysis with a code-tuned system prompt and per-family temperature. |
code_task_files | Like code_task, but houtini-lm reads the files from disk itself, so the source never passes through Claude's context. Unreadable files are reported inline rather than sinking the call, and a pre-flight estimator refuses inputs that measured data says would time out. |
embed | Text embeddings via /v1/embeddings (Nomic Embed is a solid choice). |
discover | Health check: endpoint, active model, context window, output cap, and measured speed once there's a real call to measure. |
list_models | Everything on the server, loaded and downloaded, with profiles. |
stats | Session and lifetime offload totals and per-model performance, without the model catalogue. |
The inference tools (chat, custom_prompt, code_task, code_task_files) all take an optional model to pin the call, max_tokens, and sampling controls (temperature, seed, stop, top_p, top_k, repeat_penalty, frequency_penalty, presence_penalty). chat and custom_prompt also take a json_schema, which forces the answer to conform to a JSON Schema; on LM Studio that's grammar-based sampling, so there's no hoping the model remembers to close its brackets:
If you're driving houtini-lm from your own scripts rather than from Claude, set HOUTINI_LM_STRUCTURED=1 and each inference result also carries a structuredContent block (the answer, model, tokens, timing, quality flags) to read as JSON. Leave it off for Claude Code, which shows the model only that block when it's present (the tools page has the story).
Reading the footer
Every response ends with a footer computed from the SSE stream itself:
The first-call line appears once per model per session, and it's a benchmark from a real task rather than a synthetic warm-up. The 💰 line updates every call, and it counts the tokens the other model handled (its prompt and completion, reasoning included), which is work Claude didn't do rather than a measure of Claude tokens saved; the benchmark above is the honest measure of that. When a model reports its reasoning tokens, the token count splits into reasoning and visible, so you can see when a thinking model is burning budget on hidden reasoning.
When something went wrong, a quality line says so: TRUNCATED for a partial result (a stalled connection gives you what arrived rather than a timeout error), hit-max-tokens when the budget ran out, think-blocks-stripped when reasoning was removed and tokens-estimated when the server didn't report usage. Clean output gets no quality line at all.
Per-model speed and token counts persist in ~/.houtini-lm/model-cache.db, so discover shows your measured tok/s and time to first token from the first call of a new session, and stats gives you the lifetime totals. That data is specific to your workstation on purpose, because delegation decisions should reflect your hardware rather than somebody else's benchmark. In practice, Claude delegates more the longer a session runs; after about 5,000 offloaded tokens it starts hunting for more work to push over.
Getting good results
Qwen, Llama, Nemotron and GLM score brilliantly on coding benchmarks now, and the gap between a good and a bad result is almost always the prompt rather than the model. I've spent a fair bit of time on this, and the short version goes like this. Send complete code, because local models make up details when the input is truncated, so send the whole function rather than a snippet with ... in the middle. Be explicit about the output format ("return a JSON array"), since smaller models need it. Give the model a specific persona ("expert Rust developer who cares about memory safety" does noticeably better than "helpful assistant"), state what not to do as well as what to do, and for code generation include the imports, types and signatures around the function body.
Keep long jobs in chunks, too. Most MCP clients time a tool call out at around 60 seconds, and although houtini-lm sends a progress notification on every streamed chunk to keep the clock reset, not every client or gateway passes those through. Calls of roughly 500-900 output tokens finish comfortably inside the limit. The craft of delegation goes much deeper, including the verbatim-echo pattern I use for fixes.
Check your install
scripts/shakedown.mjs runs seven of the eight tools end to end (everything except stats) and prints a table of real TTFT, tok/s, token counts and reasoning split for each call. It takes under a minute on a decent rig:
If you'd rather have a quality review than latency numbers, paste SHAKEDOWN.md into a Claude session with houtini-lm attached and Claude will drive the same steps and write you a report on the output as well as the speed.
Configuration
Most setups need only the first two or three of these. The full list, including the file-access and queuing controls, is in Configuration.
| Variable | Default | What it does |
|---|---|---|
HOUTINI_LM_ENDPOINT_URL | http://localhost:1234 | Base URL of the OpenAI-compatible API, without /v1. |
HOUTINI_LM_API_KEY | (none) | Bearer token for authenticated endpoints. |
HOUTINI_LM_MODEL | (auto-detect) | The model calls use unless they name one. Pin it on routers and big catalogues. |
HOUTINI_LM_THINKING | auto | auto, off or on - see Thinking: auto, off or on. |
HOUTINI_LM_SERIALISE | 1 | Set to 0 for cloud APIs like OpenAI, routers in front of cloud models, and backends that batch natively (vLLM, SGLang). |
HOUTINI_LM_MIN_TOKENS | 4096 | Floor for caller-supplied max_tokens. Set to 0 to honour any value. |
Compatible endpoints
Anything that speaks the OpenAI /v1/chat/completions API will work:
| What | URL | Notes |
|---|---|---|
| LM Studio | http://localhost:1234 | Default, zero config, rich metadata via its v0 API. Setup guide |
| Ollama | http://localhost:11434 | Thinking models (qwen3, deepseek-r1) handled via Ollama's delta.reasoning channel. Setup guide |
| vLLM | http://localhost:8000 | Native OpenAI API. Setup guide |
| SGLang | http://localhost:30000 | Good for repeated-context work. See Getting started |
| LiteLLM router | http://localhost:4000 | Auto-detected: aliases resolved, real limits read, non-chat models filtered, 429 backoff |
| llama.cpp | http://localhost:8080 | Server mode |
| OpenAI | https://api.openai.com | GPT-5, GPT-6 and the o-series sent only the parameters they accept; image, speech and moderation models left out of the list. Pin a model |
| OpenRouter | https://openrouter.ai/api | 300+ models, auto-detected, parallel requests allowed |
| DeepSeek | https://api.deepseek.com | Very cheap per token |
| Groq | https://api.groq.com/openai | Fast |
| Cerebras | https://api.cerebras.ai | Very fast |
| Any OpenAI-compatible API | Any URL | Set the URL and API key |
The manual
This README is the overview, and the depth lives in these pages:
| Page | What's in it |
|---|---|
| Installing houtini-lm | Every install route: local, remote GPU, cloud, OpenRouter, LiteLLM, Claude Desktop, other clients, and updating |
| Running houtini-lm in Docker | docker run -i, or served over HTTP behind Docker's MCP Gateway, with the traps we measured |
| How houtini-lm handles different models | Discovery, profiles, thinking (auto, off or on), output budgets, routing, LiteLLM routers, models that reject parameters |
| Configuration | Every environment variable, per-call settings, and where state lives |
| The tools, in depth | All eight tools: the parameters, reading the footer, the max_tokens floor |
| The craft of delegation | What to hand off and how to brief it, the verbatim-echo pattern, micro-chunking, reasoning-model budgets |
| Troubleshooting | Symptom > cause > fix for empty responses, timeouts, context-length 400s, queuing and routers |
| Getting started | Local models from zero: LM Studio or Docker, what small models are good at, which fit your VRAM |
| LM Studio, Ollama and vLLM setup | Backend guides, each with the traps that cause silent failures |
| Shakedown test | The end-to-end check, as a script or as a prompt for Claude |
| Developer guide | Architecture, contributing, release process |
Development
DEVELOPER.md covers the architecture, the reasoning-model pipeline, backend detection, the SQLite performance cache and how to add new tools or backends. If you find a model family houtini-lm handles badly, open an issue with the discover output and I'll take a look.
Good luck, and let me know how you get on!
Licence
Apache-2.0