admin
ad15ab5200
Review source: GPT-5.6 pass shared by jhestyr 2026-07-17. Accept/pushback record in NOTES N-5. [ezra]
132 lines
8.7 KiB
Markdown
132 lines
8.7 KiB
Markdown
# local-horse — the two priced builds (as of 2026-07-17)
|
||
|
||
⚠️ **Prices dated 2026-07-17, ±15% on used parts and worse on RAM/GPU (see NOTES N-3). Re-price before buying.**
|
||
⚠️ **Reviewed 2026-07-17 (external pass, NOTES N-5): decode-rate and fit numbers corrected below; any
|
||
purchase is now gated on the benchmark plan at the bottom of this file.**
|
||
Budget target: ~$20k. Constraint: no Mac. Both shapes serve an OpenAI-compatible endpoint on the tailnet
|
||
(headless Debian, Dockge like everything else). The inference stack is **part of the spec, not a detail**
|
||
(N-5 #4): a reproducible build names *model file + quant + engine + version + expert placement + context
|
||
length + prefix-cache config* — "llama.cpp / vLLM" hand-waving is not a design.
|
||
|
||
---
|
||
|
||
## Build 1 — The Workhorse (one big card + cheap capacity)
|
||
|
||
Quiet, <1kW under load, standard 120V outlet, racks next to the Mill. The pick (NOTES N-4).
|
||
|
||
| Part | Spec | Est. price (2026-07-17) |
|
||
|---|---|---|
|
||
| GPU | RTX Pro 6000 Blackwell **Max-Q** 96GB (300W blower, 1.8TB/s) | $9,500–13,000 |
|
||
| CPU | EPYC 7763 used (64c Milan, 8-ch DDR4; full 8 CCDs = full bandwidth) | ~$1,200 |
|
||
| Board | Supermicro H12SSL-i | ~$650 |
|
||
| RAM | 8× 64GB DDR4-3200 RDIMM refurb (512GB, ~205GB/s) | ~$2,000–2,800 |
|
||
| Storage | 4TB NVMe | ~$450 |
|
||
| Chassis/PSU | 4U + 1300W + cooler + fans | ~$900 |
|
||
| **Total** | | **~$15k–19k** |
|
||
|
||
What it runs (MoE trick: hot path/attention in VRAM, cold experts in system RAM):
|
||
|
||
| Model | Fit | Speed (est.) | Capability, plainly |
|
||
|---|---|---|---|
|
||
| GPT-OSS-120B (5B active) | Entirely in VRAM @ 4-bit | 100+ tok/s | Reliable daily hand — summaries, scripts, small fixes. Tier claims need re-testing on the exact quant (N-5 #5). |
|
||
| **Qwen3-Coder 480B-A35B** @ 4-bit (~270GB) | Hybrid VRAM+RAM | **UNKNOWN — plan at ~5 tok/s.** Naive PCIe offload ≈2–3; CPU-executed experts could hit low teens but is unproven on Milan/AVX2 (N-5 #1) | The prize *if* the hybrid path works. Benchmark before believing. |
|
||
| GLM 5.2 quantized | Hybrid (footprint unverified) | same unknown as above | Current open SWE-Bench champ (full precision — quant untested). |
|
||
| Kimi K2.6 (1T) @ Q3 | Barely (608GB total, tight) | low single digits | Party trick, not a daily driver. |
|
||
| K3 (2.8T) | No | — | Cloud only. Its **distills** should land in this envelope. |
|
||
|
||
**KV-cache headroom (N-5 #3):** the 96GB card is NOT all weight capacity. Qwen3-480B (62 layers,
|
||
8 KV heads, 128-dim) needs ≈24GiB of KV at 100K context in BF16 (≈12 FP8), plus CUDA workspaces and
|
||
buffers — realistic weight budget on-card is **~60–75GB**, which pushes *more* experts to RAM and
|
||
worsens the offload math above. Size every fit calculation at your target context, not at zero.
|
||
|
||
## Build 2 — The Feral Cats (8× used RTX 3090, 192GB VRAM)
|
||
|
||
Everything model-resident → prefill screams. Also screams literally; needs a **240V circuit** (~3kW load),
|
||
~150W combined idle 24/7, eight points of used-silicon failure.
|
||
|
||
| Part | Spec | Est. price (2026-07-17) |
|
||
|---|---|---|
|
||
| GPUs | 8× used RTX 3090 24GB (~$850 ea; GDDR6X dodged the DRAM spike) | ~$6,800 |
|
||
| CPU | EPYC 7402 used (Rome — just needs the PCIe lanes) | ~$300 |
|
||
| Board | ASRock Rack ROMED8-2T — **7× PCIe x16: the 8th GPU needs a bifurcated slot** (N-5 #4) | ~$700 |
|
||
| RAM | 256GB DDR4 refurb | ~$1,000 |
|
||
| Risers/frame | Bifurcation risers + open frame or 4U conversion | ~$500 |
|
||
| Power | 2× 1600W PSU + sync board, 240V | ~$650 |
|
||
| Storage/misc | NVMe, fans, zip ties, regret | ~$550 |
|
||
| **Total** | | **~$10.5k–12k** |
|
||
|
||
**Engineering debt this table hides (N-5 #6) — required before this is a plan, not a sketch:** lane
|
||
map (which slots run x16 vs bifurcated), Above-4G decoding / MMIO validation that 8 large-BAR devices
|
||
actually boot, riser signal integrity at PCIe 4.0, 16–24 PCIe power connectors with transient headroom,
|
||
an airflow design, a power-cap strategy, and a spare-GPU replacement procedure. Also a correction: 3090s
|
||
DO support **pairwise** NVLink (no 8-way fabric) — four bridged pairs may help some topologies if the
|
||
framework understands the nonuniform layout.
|
||
|
||
| Model | Fit | Speed (est.) | Capability, plainly |
|
||
|---|---|---|---|
|
||
| **Qwen3-235B-A22B** @ 4-bit (~130GB) | Fully in VRAM — the rig's sweet spot | fast decode AND prefill | Strong all-rounder one tier below Coder-480B. Best agentic feel per dollar here. |
|
||
| GPT-OSS-120B | Fully in VRAM, headroom | blazing | Same as workhorse, faster. |
|
||
| 70B-class dense (Llama/Qwen) | Trivially | blazing | Helper tier. |
|
||
| Qwen3-Coder 480B / GLM 5.2 | **Doesn't fit** at useful quant | offload defeats the rig's point | The one thing the workhorse does that this can't. |
|
||
| K2.6 / K3 | No | — | — |
|
||
|
||
## The trade in one sentence
|
||
|
||
Workhorse *might* reach the best open coding models (unproven — N-5 #1) and behaves like an appliance;
|
||
the cats run one tier down fast for half the money, paid for in a 240V run, fan roar, jank, and the
|
||
engineering debt above. For agentic loops (cache-dominated — NOTES N-1) what matters most is **keeping
|
||
the KV cache resident and un-evicted across tool calls**; all-VRAM helps there, but model ceiling still
|
||
matters: a smarter slow model wastes fewer turns than a fast one going in circles.
|
||
|
||
## Why one big card and not 8 (the physics, kept for re-derivation)
|
||
|
||
1. **Power/noise:** 8×350W = 2.8kW load — over a 15A/120V circuit's ceiling; plus ~150W idle, forever.
|
||
2. **Interconnect tax:** split models cross PCIe (~32GB/s) every token vs 1.8TB/s on-card; 3090s have
|
||
only pairwise NVLink (no 8-way fabric; P2P disabled on 4090+). Multi-GPU wins at batch *throughput*
|
||
(many users), not single-user latency — and the counter has one user.
|
||
3. **MoE changed the buy:** ~35B active params need one fast GPU + cheap capacity, not 8 GPUs of compute —
|
||
*if* the expert-offload path performs (N-5 #1: currently unproven on this CPU generation).
|
||
|
||
## Five strategies, not two (N-5 #8)
|
||
|
||
| Strategy | Role | Est. cost |
|
||
|---|---|---|
|
||
| One DGX Spark (128GB coherent, 273GB/s, ARM64) | Low-ops control case; slow but simple | ~$4k |
|
||
| Two Sparks (256GB aggregate) | Distributed-inference experiment | ~$8k |
|
||
| **One RTX Pro 6000, strong 80–120B model** | Fast, simple, proven parts — **the defensible baseline** | ~$11–15k |
|
||
| Workhorse hybrid (this doc's Build 1) | Experimental 480B CPU/GPU execution — benchmark-gated | ~$15–19k |
|
||
| **Local 80–120B + cloud escalation for hard turns** | Likely economic optimum; pairs with the baseline row | baseline + subscription |
|
||
|
||
The workhorse is honestly a *poor man's DGX Station* (748GB coherent memory, outside budget) without
|
||
the coherent interconnect — whether the workaround performs is the entire open question.
|
||
|
||
## Benchmark gate — do this BEFORE buying anything (N-5, replaces the N-4 lean)
|
||
|
||
1. Rent or borrow equivalent hardware (cloud RTX Pro 6000 instances exist; a Spark is ~$4k as a control).
|
||
2. Run the **exact quantized artifacts** — Qwen3-Coder 80B-A3B first (≈96% of 480B quality; if it holds
|
||
on our repos, one GPU suffices and the hybrid question is moot), then 120B, then 480B hybrid.
|
||
3. Measure: uncached prefill, cached TTFT, decode rate, **cache-hit rate across real tool-call loops**,
|
||
max stable context, wall power, and **successful agent turns per hour** on our actual repos — not
|
||
SWE-bench, not tok/s alone. Include: tool-call validity, patch correctness, instruction retention at
|
||
long context, recovery after failed commands, hallucinated-completion rate.
|
||
4. Treat **5 tok/s as the conservative workhorse case** until a receipt exists.
|
||
5. Only then pick a row from the strategy table.
|
||
|
||
## TCO beyond the parts bill (N-5 #9)
|
||
|
||
Subscription makes current marginal Claude cost **$0** — this box is bought for privacy, control, and
|
||
unlimited background tokens, not savings. At ~$0.15/kWh: 700W continuous ≈ $920/yr; 3kW continuous ≈
|
||
$3,940/yr; realistic duty cycle (idle 100–150W, bursty load) lands a few hundred $/yr for the workhorse.
|
||
Add: cooling, a 240V circuit if cats, dead used silicon, and admin time.
|
||
|
||
## Sources (as read 2026-07-17)
|
||
|
||
- Open-model landscape: digitalapplied.com (models↔hardware matching) · llmconfigurator.com (local coding
|
||
report) · modal.com (SWE-bench open models) · mindstudio.ai (agentic open-source 2026)
|
||
- Hardware/pricing: videocardz.com (RTX Pro 6000 list $13,250) · thundercompute.com (build pricing) ·
|
||
corewavelabs.com + pcserverandparts.com (DRAM crisis) · tomshardware.com RAM price index · memory.net
|
||
- K3: venturebeat.com · cnbc.com · marktechpost.com · simonwillison.net · platform.kimi.ai quickstart
|
||
- External review (2026-07-17): GPT-5.6-thinking pass via jhestyr's friend — full accept/pushback record
|
||
in NOTES N-5; its cites included vLLM offload docs, KTransformers AVX2 issues, Qwen3-Coder config,
|
||
NVIDIA DGX Spark/Station specs, and mykolaaleksandrov.dev on Claude Code breaking llama.cpp prefix cache
|