local-horse/BUILDS.md

local-horse — the two priced builds (as of 2026-07-17)

⚠️ Prices dated 2026-07-17, ±15% on used parts and worse on RAM/GPU (see NOTES N-3). Re-price before buying. ⚠️ Reviewed 2026-07-17 (external pass, NOTES N-5): decode-rate and fit numbers corrected below; any purchase is now gated on the benchmark plan at the bottom of this file. Budget target: ~$20k. Constraint: no Mac. Both shapes serve an OpenAI-compatible endpoint on the tailnet (headless Debian, Dockge like everything else). The inference stack is part of the spec, not a detail (N-5 #4): a reproducible build names model file + quant + engine + version + expert placement + context length + prefix-cache config — "llama.cpp / vLLM" hand-waving is not a design.

---

Build 1 — The Workhorse (one big card + cheap capacity)

Quiet, <1kW under load, standard 120V outlet, racks next to the Mill. The pick (NOTES N-4).

Part	Spec	Est. price (2026-07-17)
GPU	RTX Pro 6000 Blackwell Max-Q 96GB (300W blower, 1.8TB/s)	$9,500–13,000
CPU	EPYC 7763 used (64c Milan, 8-ch DDR4; full 8 CCDs = full bandwidth)	~$1,200
Board	Supermicro H12SSL-i	~$650
RAM	8× 64GB DDR4-3200 RDIMM refurb (512GB, ~205GB/s)	~$2,000–2,800
Storage	4TB NVMe	~$450
Chassis/PSU	4U + 1300W + cooler + fans	~$900
Total		~$15k–19k

What it runs (MoE trick: hot path/attention in VRAM, cold experts in system RAM):

Model	Fit	Speed (est.)	Capability, plainly
GPT-OSS-120B (5B active)	Entirely in VRAM @ 4-bit	100+ tok/s	Reliable daily hand — summaries, scripts, small fixes. Tier claims need re-testing on the exact quant (N-5 #5).
Qwen3-Coder 480B-A35B @ 4-bit (~270GB)	Hybrid VRAM+RAM	UNKNOWN — plan at ~5 tok/s. Naive PCIe offload ≈2–3; CPU-executed experts could hit low teens but is unproven on Milan/AVX2 (N-5 #1)	The prize if the hybrid path works. Benchmark before believing.
GLM 5.2 quantized	Hybrid (footprint unverified)	same unknown as above	Current open SWE-Bench champ (full precision — quant untested).
Kimi K2.6 (1T) @ Q3	Barely (608GB total, tight)	low single digits	Party trick, not a daily driver.
K3 (2.8T)	No	—	Cloud only. Its distills should land in this envelope.

KV-cache headroom (N-5 #3): the 96GB card is NOT all weight capacity. Qwen3-480B (62 layers, 8 KV heads, 128-dim) needs ≈24GiB of KV at 100K context in BF16 (≈12 FP8), plus CUDA workspaces and buffers — realistic weight budget on-card is ~60–75GB, which pushes more experts to RAM and worsens the offload math above. Size every fit calculation at your target context, not at zero.

Build 2 — The Feral Cats (8× used RTX 3090, 192GB VRAM)

Everything model-resident → prefill screams. Also screams literally; needs a 240V circuit (~3kW load), ~150W combined idle 24/7, eight points of used-silicon failure.

Part	Spec	Est. price (2026-07-17)
GPUs	8× used RTX 3090 24GB (~$850 ea; GDDR6X dodged the DRAM spike)	~$6,800
CPU	EPYC 7402 used (Rome — just needs the PCIe lanes)	~$300
Board	ASRock Rack ROMED8-2T — 7× PCIe x16: the 8th GPU needs a bifurcated slot (N-5 #4)	~$700
RAM	256GB DDR4 refurb	~$1,000
Risers/frame	Bifurcation risers + open frame or 4U conversion	~$500
Power	2× 1600W PSU + sync board, 240V	~$650
Storage/misc	NVMe, fans, zip ties, regret	~$550
Total		~$10.5k–12k

Engineering debt this table hides (N-5 #6) — required before this is a plan, not a sketch: lane map (which slots run x16 vs bifurcated), Above-4G decoding / MMIO validation that 8 large-BAR devices actually boot, riser signal integrity at PCIe 4.0, 16–24 PCIe power connectors with transient headroom, an airflow design, a power-cap strategy, and a spare-GPU replacement procedure. Also a correction: 3090s DO support pairwise NVLink (no 8-way fabric) — four bridged pairs may help some topologies if the framework understands the nonuniform layout.

Model	Fit	Speed (est.)	Capability, plainly
Qwen3-235B-A22B @ 4-bit (~130GB)	Fully in VRAM — the rig's sweet spot	fast decode AND prefill	Strong all-rounder one tier below Coder-480B. Best agentic feel per dollar here.
GPT-OSS-120B	Fully in VRAM, headroom	blazing	Same as workhorse, faster.
70B-class dense (Llama/Qwen)	Trivially	blazing	Helper tier.
Qwen3-Coder 480B / GLM 5.2	Doesn't fit at useful quant	offload defeats the rig's point	The one thing the workhorse does that this can't.
K2.6 / K3	No	—	—

The trade in one sentence

Workhorse might reach the best open coding models (unproven — N-5 #1) and behaves like an appliance; the cats run one tier down fast for half the money, paid for in a 240V run, fan roar, jank, and the engineering debt above. For agentic loops (cache-dominated — NOTES N-1) what matters most is keeping the KV cache resident and un-evicted across tool calls; all-VRAM helps there, but model ceiling still matters: a smarter slow model wastes fewer turns than a fast one going in circles.

Why one big card and not 8 (the physics, kept for re-derivation)

1. Power/noise: 8×350W = 2.8kW load — over a 15A/120V circuit's ceiling; plus ~150W idle, forever.
2. Interconnect tax: split models cross PCIe (~32GB/s) every token vs 1.8TB/s on-card; 3090s have only pairwise NVLink (no 8-way fabric; P2P disabled on 4090+). Multi-GPU wins at batch throughput (many users), not single-user latency — and the counter has one user.
3. MoE changed the buy: ~35B active params need one fast GPU + cheap capacity, not 8 GPUs of compute — if the expert-offload path performs (N-5 #1: currently unproven on this CPU generation).

Five strategies, not two (N-5 #8)

Strategy	Role	Est. cost
One DGX Spark (128GB coherent, 273GB/s, ARM64)	Low-ops control case; slow but simple	~$4k
Two Sparks (256GB aggregate)	Distributed-inference experiment	~$8k
One RTX Pro 6000, strong 80–120B model	Fast, simple, proven parts — the defensible baseline	~$11–15k
Workhorse hybrid (this doc's Build 1)	Experimental 480B CPU/GPU execution — benchmark-gated	~$15–19k
Local 80–120B + cloud escalation for hard turns	Likely economic optimum; pairs with the baseline row	baseline + subscription

The workhorse is honestly a poor man's DGX Station (748GB coherent memory, outside budget) without the coherent interconnect — whether the workaround performs is the entire open question.

Benchmark gate — do this BEFORE buying anything (N-5, replaces the N-4 lean)

1. Rent or borrow equivalent hardware (cloud RTX Pro 6000 instances exist; a Spark is ~$4k as a control).
2. Run the exact quantized artifacts — Qwen3-Coder 80B-A3B first (≈96% of 480B quality; if it holds on our repos, one GPU suffices and the hybrid question is moot), then 120B, then 480B hybrid.
3. Measure: uncached prefill, cached TTFT, decode rate, cache-hit rate across real tool-call loops, max stable context, wall power, and successful agent turns per hour on our actual repos — not SWE-bench, not tok/s alone. Include: tool-call validity, patch correctness, instruction retention at long context, recovery after failed commands, hallucinated-completion rate.
4. Treat 5 tok/s as the conservative workhorse case until a receipt exists.
5. Only then pick a row from the strategy table.

TCO beyond the parts bill (N-5 #9)

Subscription makes current marginal Claude cost $0 — this box is bought for privacy, control, and unlimited background tokens, not savings. At ~$0.15/kWh: 700W continuous ≈ $920/yr; 3kW continuous ≈ $3,940/yr; realistic duty cycle (idle 100–150W, bursty load) lands a few hundred $/yr for the workhorse. Add: cooling, a 240V circuit if cats, dead used silicon, and admin time.

Sources (as read 2026-07-17)

• Open-model landscape: digitalapplied.com (models↔hardware matching) · llmconfigurator.com (local coding report) · modal.com (SWE-bench open models) · mindstudio.ai (agentic open-source 2026)
• Hardware/pricing: videocardz.com (RTX Pro 6000 list $13,250) · thundercompute.com (build pricing) · corewavelabs.com + pcserverandparts.com (DRAM crisis) · tomshardware.com RAM price index · memory.net
• K3: venturebeat.com · cnbc.com · marktechpost.com · simonwillison.net · platform.kimi.ai quickstart
• External review (2026-07-17): GPT-5.6-thinking pass via jhestyr's friend — full accept/pushback record in NOTES N-5; its cites included vLLM offload docs, KTransformers AVX2 issues, Qwen3-Coder config, NVIDIA DGX Spark/Station specs, and mykolaaleksandrov.dev on Claude Code breaking llama.cpp prefix cache