Forging Local Models

The interesting question isn't whether a small model can match a frontier one in the abstract. It's whether a model you own, on hardware you own, can do one bounded job better than a generalist API can — and keep doing it when the cloud, the contract, or the rate limit changes. Everything below is what we've actually learned forging those models on a desk, not what the benchmarks promise.

"Local" is a thesis, not a constraint. A model served on-prem keeps its weights, its corpus, and its latency inside the building, under the operator's control. For work that touches proprietary documents — and most real work does — that's not a privacy nicety, it's the whole point. The trade we make in exchange is that we have to be deliberate: a 600 GB frontier model isn't coming to the laptop, so we have to compress the right capability into the smallest thing that still holds it. That compression is the craft. It comes in two motions — distillation (capturing judgment) and fine-tuning (binding it to a base) — and a lot of unglamorous decisions around them.

01Pick a base that can reason — then adapt it

The first and most expensive mistake is choosing the base model by size alone. Small is a deployment win and a capability risk, and the risk is invisible until it bites.

A concrete case from our bench: we put a 4-bit, ~4B-parameter vision-language model on a real finish plan. It read three of four finish callouts correctly — exact product names, right locations — which is genuinely impressive for a model that fits in a few gigabytes. Then it silently dropped the fourth: a sealer that was named only in the general notes, not on the keyed plan. It didn't flag uncertainty. It didn't cross-reference the sheets. It just returned a confident, incomplete answer. On a takeoff, a confident omission is worse than a visible failure, because nobody goes looking for it.

The lesson isn't "4B is bad." It's that reasoning capacity is load-bearing for any task where the answer requires reconciling more than one source, and you cannot fine-tune reasoning capacity into a model that never had room for it. So the order matters: choose a base that can actually reason over the task — for us that's been a 7B-class vision-language model for the seeing, and a reasoning-capable text model in the 30B class for the reconciling — and then adapt. Distillation and LoRA sharpen a capability; they don't conjure one.

02Distillation is knowledge capture, not just logit matching

The textbook framing of distillation is a student network learning to match a teacher's outputs. That's real, and the current research is getting sharper about it — the move toward on-policy distillation, where the teacher grades what the student actually generated rather than feeding it flawless prefixes, is the right correction, because errors compound over long outputs and a student trained only on perfect inputs has never seen its own mistakes (Song & Zheng, 2026). For transferring a frontier model's judgment into something that has to hold up over a whole document, that distinction is the difference between a demo and a tool.

But in applied work the most valuable teacher often isn't another model — it's a person, and the most valuable knowledge is the part that isn't in the pixels.

Predict the visible; look up the invisible.

Here's what that means in practice. In our domain, a plan shows you geometry — areas, rooms, finish tags. What it does not show is scope: which of those finishes a given trade actually installs versus which belong to someone else, which substitutions an owner tends to make, which conditions are always in-contract and which are variable. No amount of looking at the drawing recovers that; it lives in an expert's head. So we distill it deliberately — sit with the domain expert, and turn their judgment into a structured knowledge graph: an explicit, versioned record of scope rules the model can look things up in. The model predicts what's visible and defers to the graph for what isn't. The graph is teacher-distilled knowledge in a form you can audit, correct, and grow — which is exactly what you want sitting next to a probabilistic model.

This reframes the whole exercise. The moat was never the raw documents; everyone in the trade has documents. The moat is the captured judgment about what they mean — and a structured artifact you can inspect beats a few thousand opaque weights for anything you'll be held accountable for.

03One core, many adapters: LoRA per vertical

Full-parameter fine-tuning a multi-billion-parameter base, per domain, is neither necessary nor wise on a single workstation. LoRA — low-rank adaptation — trains a small set of additional weights that ride on top of a frozen base, and it's the right granularity for applied work: you keep one strong, quantized core in memory and swap lightweight adapters for each vertical. A fitness-center build and a retail build want different priors; they don't want different 7B models.

The numbers are friendlier than people expect. A recent identity-LoRA we trained — an unrelated image model, but the mechanics are the same — converged on roughly twenty reference examples, about 2,200 steps, a single trigger token to anchor the new concept, and inference settled at a LoRA scale near 1.0 with guidance around 3.5. The point is the scale of effort: small, curated datasets and a few thousand steps, not a research cluster. The discipline is in the dataset, not the GPU bill — a tightly-scoped, correctly-labeled set of a few dozen to a few hundred examples will out-train ten times as much noisy data.

On Apple Silicon the framework that makes this practical is MLX. It runs and fine-tunes quantized models against unified memory natively, converts Hugging Face weights in seconds, and — per Apple's own measurements — leans on the M-series neural accelerators hard enough to make on-device inference genuinely usable, fitting a 30B-class model into a fraction of the memory a workstation now ships with.

02 ½The 128 GB envelope

Unified memory changed the arithmetic. A 128 GB machine is not a toy: at 4-bit quantization, 30B–70B-class models fit with room to spare, and a 7B vision model plus a reasoning text model plus their adapters coexist comfortably. The constraint people imagine — "you can't run real models locally" — is two years stale.

Quantization is the lever that opens the envelope, and it's nearly free when you respect it: 4-bit weights cost a little accuracy and buy a lot of headroom, and mixed-precision modes let you spend bits where they matter. The operational shape we settled on is boring in the best way — a local runtime exposing an OpenAI-compatible endpoint on localhost, so the application code doesn't know or care that the model is running three inches away instead of in a datacenter. Sovereign on the inside, ordinary on the outside.

03The gotchas that actually cost you time

None of the following is in the model card. All of it cost us a session to learn.

Reasoning eats your token budget

Reasoning-capable models think before they answer, and that thinking counts against max_tokens. Set a tight limit and the model spends its entire budget reasoning and returns an empty completion — not an error, just nothing. The fix is to budget generously (we run ≥ 1,500–3,000 tokens for reasoning models) and, where the API supports it, route the chain-of-thought to a separate reasoning_content field so it doesn't crowd the answer. When you fine-tune a reasoning model, remember its "forward pass" includes the thinking; size everything accordingly.

Sometimes the best distillation is the right parser

The reflex on a document problem is "point a vision model at it." Often that's the wrong tool. Many of the PDFs we work with are vectored — they carry a true text layer with thousands of selectable characters per sheet. For those, format-aware extraction (a library like pdfplumber, or pdftotext -layout) is more accurate than any vision model, deterministic, and effectively free. We reserve the vision model for the genuinely graphical regions — keyed plans, hatch legends — where there's no text to recover.

When one adapter can't hold a constraint, chain a specialist

There's a temptation to encode every requirement into a single fine-tune. It doesn't hold when the constraints are fine-grained and competing. We hit this trying to control specific visual features through prompts alone on a tuned image model — each correction nudged the whole output and broke something else, classic whack-a-mole. The fix wasn't a bigger LoRA; it was a two-stage pipeline: let the tuned model do the thing it's good at, then hand its output to a specialized post-processor (here, an image-edit model given explicit references) that changes only what it's told and preserves the rest. The general lesson generalizes well beyond images: when one model can't satisfy all the constraints at once, stage specialists instead of overloading a generalist.

04Evaluate against outcomes, or you're just vibing

The part that turns a tuned model from a parlor trick into an asset is the feedback loop, and it has to close on reality, not on a held-out split scraped from the same source. For us the ground truth is the as-built outcome — what actually got installed, captured at closeout — fed back to correct both the model and the knowledge graph. That loop is slow, it's unglamorous, and it is the entire moat: anyone can fine-tune on public data; almost no one has a disciplined record of where their model was wrong and what the world actually did. Build the eval harness before you build the model, and make it consume real outcomes.

---

The throughline is that the model is rarely the hard part. The hard parts are choosing a base that can reason, capturing the judgment that isn't in the data, respecting the boring operational details — token budgets, file formats, memory envelopes — and grinding an honest feedback loop against what actually happened. Do those, and a model you own on a desk you own will quietly out-perform a rented generalist on the one job you built it for. That's the whole bet, and so far it keeps paying out.

We ship weights, not decks. If you want one of these put into your own operation — served locally, corpus in your building — that's the work we do.