admin
c46ff918a2
Rumination 2026-07-17, not a purchase order. Front-door README + NOTES N-1..N-4 + BUILDS.md. [ezra]
79 lines
4.5 KiB
Markdown
79 lines
4.5 KiB
Markdown
# local-horse — the two priced builds (as of 2026-07-17)
|
||
|
||
⚠️ **Prices dated 2026-07-17, ±15% on used parts and worse on RAM/GPU (see NOTES N-3). Re-price before buying.**
|
||
Budget target: ~$20k. Constraint: no Mac. Both shapes serve an OpenAI-compatible endpoint on the tailnet
|
||
(llama.cpp / vLLM, headless Debian, Dockge like everything else).
|
||
|
||
---
|
||
|
||
## Build 1 — The Workhorse (one big card + cheap capacity)
|
||
|
||
Quiet, <1kW under load, standard 120V outlet, racks next to the Mill. The pick (NOTES N-4).
|
||
|
||
| Part | Spec | Est. price (2026-07-17) |
|
||
|---|---|---|
|
||
| GPU | RTX Pro 6000 Blackwell **Max-Q** 96GB (300W blower, 1.8TB/s) | $9,500–13,000 |
|
||
| CPU | EPYC 7763 used (64c Milan, 8-ch DDR4; full 8 CCDs = full bandwidth) | ~$1,200 |
|
||
| Board | Supermicro H12SSL-i | ~$650 |
|
||
| RAM | 8× 64GB DDR4-3200 RDIMM refurb (512GB, ~205GB/s) | ~$2,000–2,800 |
|
||
| Storage | 4TB NVMe | ~$450 |
|
||
| Chassis/PSU | 4U + 1300W + cooler + fans | ~$900 |
|
||
| **Total** | | **~$15k–19k** |
|
||
|
||
What it runs (MoE trick: hot path/attention in VRAM, cold experts in system RAM):
|
||
|
||
| Model | Fit | Speed (est.) | Capability, plainly |
|
||
|---|---|---|---|
|
||
| GPT-OSS-120B (5B active) | Entirely in VRAM @ 4-bit | 100+ tok/s | Reliable daily hand — summaries, scripts, small fixes. ~Sonnet-3.5 tier. |
|
||
| **Qwen3-Coder 480B-A35B** @ 4-bit (~270GB) | Hybrid VRAM+RAM | ~15–25 tok/s | **The prize.** Real agentic coding on a known repo. Sonnet-4-era. |
|
||
| GLM 5.2 quantized | Hybrid (footprint unverified) | similar | Current open SWE-Bench champ; same tier or a notch up. |
|
||
| Kimi K2.6 (1T) @ Q3 | Barely (608GB total, tight) | single digits | Party trick, not a daily driver. |
|
||
| K3 (2.8T) | No | — | Cloud only. Its **distills** should land in this envelope. |
|
||
|
||
## Build 2 — The Feral Cats (8× used RTX 3090, 192GB VRAM)
|
||
|
||
Everything model-resident → prefill screams. Also screams literally; needs a **240V circuit** (~3kW load),
|
||
~150W combined idle 24/7, eight points of used-silicon failure.
|
||
|
||
| Part | Spec | Est. price (2026-07-17) |
|
||
|---|---|---|
|
||
| GPUs | 8× used RTX 3090 24GB (~$850 ea; GDDR6X dodged the DRAM spike) | ~$6,800 |
|
||
| CPU | EPYC 7402 used (Rome — just needs the PCIe lanes) | ~$300 |
|
||
| Board | ASRock Rack ROMED8-2T (7× PCIe x16) | ~$700 |
|
||
| RAM | 256GB DDR4 refurb | ~$1,000 |
|
||
| Risers/frame | Bifurcation risers + open frame or 4U conversion | ~$500 |
|
||
| Power | 2× 1600W PSU + sync board, 240V | ~$650 |
|
||
| Storage/misc | NVMe, fans, zip ties, regret | ~$550 |
|
||
| **Total** | | **~$10.5k–12k** |
|
||
|
||
| Model | Fit | Speed (est.) | Capability, plainly |
|
||
|---|---|---|---|
|
||
| **Qwen3-235B-A22B** @ 4-bit (~130GB) | Fully in VRAM — the rig's sweet spot | fast decode AND prefill | Strong all-rounder one tier below Coder-480B. Best agentic feel per dollar here. |
|
||
| GPT-OSS-120B | Fully in VRAM, headroom | blazing | Same as workhorse, faster. |
|
||
| 70B-class dense (Llama/Qwen) | Trivially | blazing | Helper tier. |
|
||
| Qwen3-Coder 480B / GLM 5.2 | **Doesn't fit** at useful quant | offload defeats the rig's point | The one thing the workhorse does that this can't. |
|
||
| K2.6 / K3 | No | — | — |
|
||
|
||
## The trade in one sentence
|
||
|
||
Workhorse reaches the **best** open coding models slowly-but-surely and behaves like an appliance; the
|
||
cats run one tier down blisteringly fast for half the money, paid for in a 240V run, fan roar, and jank.
|
||
For agentic loops (99% context re-reads — NOTES N-1) all-VRAM prefill genuinely matters, but model
|
||
ceiling matters more: a smarter slow model wastes fewer turns than a fast one going in circles.
|
||
|
||
## Why one big card and not 8 (the physics, kept for re-derivation)
|
||
|
||
1. **Power/noise:** 8×350W = 2.8kW load — over a 15A/120V circuit's ceiling; plus ~150W idle, forever.
|
||
2. **Interconnect tax:** split models cross PCIe (~32GB/s) every token vs 1.8TB/s on-card; consumer cards
|
||
have no P2P/NVLink (disabled on 4090+). Multi-GPU wins at batch *throughput* (many users), not
|
||
single-user latency — and the counter has one user.
|
||
3. **MoE changed the buy:** ~35B active params need one fast GPU + cheap capacity, not 8 GPUs of compute.
|
||
|
||
## Sources (as read 2026-07-17)
|
||
|
||
- Open-model landscape: digitalapplied.com (models↔hardware matching) · llmconfigurator.com (local coding
|
||
report) · modal.com (SWE-bench open models) · mindstudio.ai (agentic open-source 2026)
|
||
- Hardware/pricing: videocardz.com (RTX Pro 6000 list $13,250) · thundercompute.com (build pricing) ·
|
||
corewavelabs.com + pcserverandparts.com (DRAM crisis) · tomshardware.com RAM price index · memory.net
|
||
- K3: venturebeat.com · cnbc.com · marktechpost.com · simonwillison.net · platform.kimi.ai quickstart
|