diff --git a/BUILDS.md b/BUILDS.md index 43ef85e..e6d5861 100644 --- a/BUILDS.md +++ b/BUILDS.md @@ -1,8 +1,12 @@ # local-horse — the two priced builds (as of 2026-07-17) ⚠️ **Prices dated 2026-07-17, ±15% on used parts and worse on RAM/GPU (see NOTES N-3). Re-price before buying.** +⚠️ **Reviewed 2026-07-17 (external pass, NOTES N-5): decode-rate and fit numbers corrected below; any +purchase is now gated on the benchmark plan at the bottom of this file.** Budget target: ~$20k. Constraint: no Mac. Both shapes serve an OpenAI-compatible endpoint on the tailnet -(llama.cpp / vLLM, headless Debian, Dockge like everything else). +(headless Debian, Dockge like everything else). The inference stack is **part of the spec, not a detail** +(N-5 #4): a reproducible build names *model file + quant + engine + version + expert placement + context +length + prefix-cache config* — "llama.cpp / vLLM" hand-waving is not a design. --- @@ -24,12 +28,17 @@ What it runs (MoE trick: hot path/attention in VRAM, cold experts in system RAM) | Model | Fit | Speed (est.) | Capability, plainly | |---|---|---|---| -| GPT-OSS-120B (5B active) | Entirely in VRAM @ 4-bit | 100+ tok/s | Reliable daily hand — summaries, scripts, small fixes. ~Sonnet-3.5 tier. | -| **Qwen3-Coder 480B-A35B** @ 4-bit (~270GB) | Hybrid VRAM+RAM | ~15–25 tok/s | **The prize.** Real agentic coding on a known repo. Sonnet-4-era. | -| GLM 5.2 quantized | Hybrid (footprint unverified) | similar | Current open SWE-Bench champ; same tier or a notch up. | -| Kimi K2.6 (1T) @ Q3 | Barely (608GB total, tight) | single digits | Party trick, not a daily driver. | +| GPT-OSS-120B (5B active) | Entirely in VRAM @ 4-bit | 100+ tok/s | Reliable daily hand — summaries, scripts, small fixes. Tier claims need re-testing on the exact quant (N-5 #5). | +| **Qwen3-Coder 480B-A35B** @ 4-bit (~270GB) | Hybrid VRAM+RAM | **UNKNOWN — plan at ~5 tok/s.** Naive PCIe offload ≈2–3; CPU-executed experts could hit low teens but is unproven on Milan/AVX2 (N-5 #1) | The prize *if* the hybrid path works. Benchmark before believing. | +| GLM 5.2 quantized | Hybrid (footprint unverified) | same unknown as above | Current open SWE-Bench champ (full precision — quant untested). | +| Kimi K2.6 (1T) @ Q3 | Barely (608GB total, tight) | low single digits | Party trick, not a daily driver. | | K3 (2.8T) | No | — | Cloud only. Its **distills** should land in this envelope. | +**KV-cache headroom (N-5 #3):** the 96GB card is NOT all weight capacity. Qwen3-480B (62 layers, +8 KV heads, 128-dim) needs ≈24GiB of KV at 100K context in BF16 (≈12 FP8), plus CUDA workspaces and +buffers — realistic weight budget on-card is **~60–75GB**, which pushes *more* experts to RAM and +worsens the offload math above. Size every fit calculation at your target context, not at zero. + ## Build 2 — The Feral Cats (8× used RTX 3090, 192GB VRAM) Everything model-resident → prefill screams. Also screams literally; needs a **240V circuit** (~3kW load), @@ -39,13 +48,20 @@ Everything model-resident → prefill screams. Also screams literally; needs a * |---|---|---| | GPUs | 8× used RTX 3090 24GB (~$850 ea; GDDR6X dodged the DRAM spike) | ~$6,800 | | CPU | EPYC 7402 used (Rome — just needs the PCIe lanes) | ~$300 | -| Board | ASRock Rack ROMED8-2T (7× PCIe x16) | ~$700 | +| Board | ASRock Rack ROMED8-2T — **7× PCIe x16: the 8th GPU needs a bifurcated slot** (N-5 #4) | ~$700 | | RAM | 256GB DDR4 refurb | ~$1,000 | | Risers/frame | Bifurcation risers + open frame or 4U conversion | ~$500 | | Power | 2× 1600W PSU + sync board, 240V | ~$650 | | Storage/misc | NVMe, fans, zip ties, regret | ~$550 | | **Total** | | **~$10.5k–12k** | +**Engineering debt this table hides (N-5 #6) — required before this is a plan, not a sketch:** lane +map (which slots run x16 vs bifurcated), Above-4G decoding / MMIO validation that 8 large-BAR devices +actually boot, riser signal integrity at PCIe 4.0, 16–24 PCIe power connectors with transient headroom, +an airflow design, a power-cap strategy, and a spare-GPU replacement procedure. Also a correction: 3090s +DO support **pairwise** NVLink (no 8-way fabric) — four bridged pairs may help some topologies if the +framework understands the nonuniform layout. + | Model | Fit | Speed (est.) | Capability, plainly | |---|---|---|---| | **Qwen3-235B-A22B** @ 4-bit (~130GB) | Fully in VRAM — the rig's sweet spot | fast decode AND prefill | Strong all-rounder one tier below Coder-480B. Best agentic feel per dollar here. | @@ -56,18 +72,52 @@ Everything model-resident → prefill screams. Also screams literally; needs a * ## The trade in one sentence -Workhorse reaches the **best** open coding models slowly-but-surely and behaves like an appliance; the -cats run one tier down blisteringly fast for half the money, paid for in a 240V run, fan roar, and jank. -For agentic loops (99% context re-reads — NOTES N-1) all-VRAM prefill genuinely matters, but model -ceiling matters more: a smarter slow model wastes fewer turns than a fast one going in circles. +Workhorse *might* reach the best open coding models (unproven — N-5 #1) and behaves like an appliance; +the cats run one tier down fast for half the money, paid for in a 240V run, fan roar, jank, and the +engineering debt above. For agentic loops (cache-dominated — NOTES N-1) what matters most is **keeping +the KV cache resident and un-evicted across tool calls**; all-VRAM helps there, but model ceiling still +matters: a smarter slow model wastes fewer turns than a fast one going in circles. ## Why one big card and not 8 (the physics, kept for re-derivation) 1. **Power/noise:** 8×350W = 2.8kW load — over a 15A/120V circuit's ceiling; plus ~150W idle, forever. -2. **Interconnect tax:** split models cross PCIe (~32GB/s) every token vs 1.8TB/s on-card; consumer cards - have no P2P/NVLink (disabled on 4090+). Multi-GPU wins at batch *throughput* (many users), not - single-user latency — and the counter has one user. -3. **MoE changed the buy:** ~35B active params need one fast GPU + cheap capacity, not 8 GPUs of compute. +2. **Interconnect tax:** split models cross PCIe (~32GB/s) every token vs 1.8TB/s on-card; 3090s have + only pairwise NVLink (no 8-way fabric; P2P disabled on 4090+). Multi-GPU wins at batch *throughput* + (many users), not single-user latency — and the counter has one user. +3. **MoE changed the buy:** ~35B active params need one fast GPU + cheap capacity, not 8 GPUs of compute — + *if* the expert-offload path performs (N-5 #1: currently unproven on this CPU generation). + +## Five strategies, not two (N-5 #8) + +| Strategy | Role | Est. cost | +|---|---|---| +| One DGX Spark (128GB coherent, 273GB/s, ARM64) | Low-ops control case; slow but simple | ~$4k | +| Two Sparks (256GB aggregate) | Distributed-inference experiment | ~$8k | +| **One RTX Pro 6000, strong 80–120B model** | Fast, simple, proven parts — **the defensible baseline** | ~$11–15k | +| Workhorse hybrid (this doc's Build 1) | Experimental 480B CPU/GPU execution — benchmark-gated | ~$15–19k | +| **Local 80–120B + cloud escalation for hard turns** | Likely economic optimum; pairs with the baseline row | baseline + subscription | + +The workhorse is honestly a *poor man's DGX Station* (748GB coherent memory, outside budget) without +the coherent interconnect — whether the workaround performs is the entire open question. + +## Benchmark gate — do this BEFORE buying anything (N-5, replaces the N-4 lean) + +1. Rent or borrow equivalent hardware (cloud RTX Pro 6000 instances exist; a Spark is ~$4k as a control). +2. Run the **exact quantized artifacts** — Qwen3-Coder 80B-A3B first (≈96% of 480B quality; if it holds + on our repos, one GPU suffices and the hybrid question is moot), then 120B, then 480B hybrid. +3. Measure: uncached prefill, cached TTFT, decode rate, **cache-hit rate across real tool-call loops**, + max stable context, wall power, and **successful agent turns per hour** on our actual repos — not + SWE-bench, not tok/s alone. Include: tool-call validity, patch correctness, instruction retention at + long context, recovery after failed commands, hallucinated-completion rate. +4. Treat **5 tok/s as the conservative workhorse case** until a receipt exists. +5. Only then pick a row from the strategy table. + +## TCO beyond the parts bill (N-5 #9) + +Subscription makes current marginal Claude cost **$0** — this box is bought for privacy, control, and +unlimited background tokens, not savings. At ~$0.15/kWh: 700W continuous ≈ $920/yr; 3kW continuous ≈ +$3,940/yr; realistic duty cycle (idle 100–150W, bursty load) lands a few hundred $/yr for the workhorse. +Add: cooling, a 240V circuit if cats, dead used silicon, and admin time. ## Sources (as read 2026-07-17) @@ -76,3 +126,6 @@ ceiling matters more: a smarter slow model wastes fewer turns than a fast one go - Hardware/pricing: videocardz.com (RTX Pro 6000 list $13,250) · thundercompute.com (build pricing) · corewavelabs.com + pcserverandparts.com (DRAM crisis) · tomshardware.com RAM price index · memory.net - K3: venturebeat.com · cnbc.com · marktechpost.com · simonwillison.net · platform.kimi.ai quickstart +- External review (2026-07-17): GPT-5.6-thinking pass via jhestyr's friend — full accept/pushback record + in NOTES N-5; its cites included vLLM offload docs, KTransformers AVX2 issues, Qwen3-Coder config, + NVIDIA DGX Spark/Station specs, and mykolaaleksandrov.dev on Claude Code breaking llama.cpp prefix cache diff --git a/NOTES.md b/NOTES.md index ff67284..0871e33 100644 --- a/NOTES.md +++ b/NOTES.md @@ -16,8 +16,11 @@ cached-in · $3/M in · $15/M out). Repricing the adventure game's real build to - **1,147M cache-read** + 18M uncached-in + 3.5M out → **≈ $451 at K3 rates** (±30%, session-level granularity) vs ≈ $1,307 at Claude list prices. Actual marginal cost: $0 (subscription). -- The load-bearing fact: agentic coding is **~99% context re-reads, ~1% output**. Cached-input price - (cloud) / prefill speed + KV-cache persistence (local) dominate everything. +- The load-bearing fact: agentic coding is **~99% cache reads, ~1% output** — cache-*dominated*. + (Corrected per N-5: cache reads are **hits**, not reprocessing. Local first-order requirements = + KV-cache persistence + hit rate + capacity; raw prefill bandwidth matters on the misses only. + Operational gotcha: harnesses that mutate the prompt *prefix* silently break llama.cpp/vLLM prefix + caching — a slower box that keeps a 100K-token cache warm beats a faster box that keeps evicting it.) - K3 itself is never local: ~1.4TB weights at 4-bit. "Local like that" = best open model that fits a box. ## N-2 (2026-07-17) — The open-model landscape, mid-2026 @@ -46,7 +49,7 @@ mid-2026; SK Hynix sold out through 2026, fabs pivoted to HBM). Consequences: - RTX Pro 6000 Blackwell 96GB: Nvidia list jumped 55% in 16 months to $13,250; street/Max-Q hunts lower. - Relief not expected before **2027**. Waiting is a legitimate strategy; re-price at buy time. -## N-4 (2026-07-17) — Decision lean (not a decision) +## N-4 (2026-07-17) — Decision lean (not a decision) — ⚠️ SUPERSEDED BY N-5 Ezra's recommendation if/when the itch turns real: **the workhorse** (Milan + 512GB DDR4 refurb + one RTX Pro 6000 Max-Q, ~$15–19k) over the 8×3090 rig — a smarter model that's slower wastes fewer turns @@ -55,3 +58,38 @@ second card slot) without replacing bones. Integration shape: headless Debian, l OpenAI-compatible endpoint on the tailnet; per [[mill-is-home-for-workloads]] it's a Mill annex, not a new species. Sensible trigger to revisit: **after 2026-07-27** (K3 weights + first distills) or when DDR5 unclenches. + +**Superseded 2026-07-17 (same day) by N-5:** the workhorse's decode-rate assumption didn't survive +external review. Lean is now benchmark-gated; see N-5. + +## N-5 (2026-07-17) — External review (GPT-5.6 via jhestyr's friend) — what survived, what didn't + +jhestyr shared BUILDS.md out (repo made public for it) and brought back a GPT-5.6-thinking review. +Graded on merits; most of it holds. **Accepted corrections** (BUILDS.md updated to match): + +1. **Workhorse decode rate was wrong.** My "15–25 tok/s" for Qwen3-Coder-480B silently assumed + CPU-executed experts. Naive vLLM/llama.cpp offload ships ~11GB of active expert weights over PCIe + (~25GB/s) *per token* → **2–3 tok/s ceiling**. KTransformers-style CPU-expert execution could reach + low teens against 205GB/s DDR4 — but its fast paths want AVX-512/AMX, and **Milan is AVX2** (native + AVX2 support still maturing as of Feb 2026). Honest number: **unknown; plan at ~5 tok/s** until + benchmarked on this exact combo. The workhorse is a hypothesis, not a validated build. +2. **Cache semantics were backward** (fixed in N-1): cache reads = hits, not re-prefill. Persistence/ + hit-rate/capacity are first-order; prefill bandwidth is the miss path. Corollary gotcha: Claude + Code-style harnesses can mutate the prompt prefix and silently kill prefix-cache reuse. +3. **KV-cache headroom ignored:** Qwen3-480B at 100K context ≈ 24GiB BF16 (12 FP8) of KV — the 96GB + card really offers ~60–75GB for weights, pushing MORE experts to RAM. Compounds #1. +4. **Feral cats under-engineered:** ROMED8-2T has **7** x16 slots — my table put 8 GPUs on it with no + bifurcation plan, lane map, Above-4G/MMIO validation, or power engineering. Also: 3090s DO have + pairwise NVLink (no 8-way fabric) — my "no NVLink" line was too flat. +5. **Capability tiers not decision-grade:** "Sonnet-4-era" conflates full-precision benchmarks with an + aggressive quant on an exotic execution path. The metric that matters: **successful agent turns per + hour** on OUR repos. And test the **80B-A3B first** (≈96% quality) — if it holds, one GPU suffices. +6. **Economics:** subscription makes marginal Claude cost $0 — the box is justified by privacy/control/ + unlimited background tokens, not savings. Add TCO: ~$0.15/kWh → 700W continuous ≈ $920/yr. + (My pushback: our duty cycle isn't continuous; idle ~100–150W. Real number is a few hundred $/yr.) + +**Pushback recorded (minor):** the 2–3 tok/s "upper bound" assumes uniformly distributed expert +selection with zero locality; hot-expert pinning skews that upward in practice — but direction stands +and the burden of proof is ours. **New lean (replaces N-4):** *benchmark before buying* — the friend's +strongest frame: **one RTX Pro 6000 running a strong 80–120B local model, with cloud escalation for +hard turns, is more defensible than either full build.** Benchmark gate lives in BUILDS.md. diff --git a/README.md b/README.md index 3be22d9..7a18bc1 100644 --- a/README.md +++ b/README.md @@ -15,9 +15,15 @@ re-derive it: two build shapes, the models each runs, honest capability tiers, a that will invalidate the numbers over time. The trigger was real data: repricing the lassiter-creek-adventure-game's build (10 sessions, Jul 5–17, -from local transcripts) showed **1.15B cache-read tokens vs 3.5M output** — agentic coding is ~99% -re-reading context. That shape drives every hardware conclusion here (prefill/memory-bandwidth is the -bottleneck, not generation speed). +from local transcripts) showed **1.15B cache-read tokens vs 3.5M output** — agentic coding is +cache-dominated. The hardware conclusion that follows (corrected by external review, NOTES N-5): the +first-order requirements are **KV-cache persistence, hit rate, and capacity** — cache reads are *hits*, +not reprocessing; raw prefill bandwidth matters on the misses, not on the 99%. + +**Status 2026-07-17 (late):** externally reviewed (jhestyr's friend via GPT-5.6 — NOTES N-5). Two +original numbers did not survive: the workhorse decode-rate estimate and the feral-cats slot count. +**Any purchase is now gated on the benchmark plan in BUILDS.md** — the workhorse is a hypothesis, not +a validated build. Standing constraints from jhestyr: **no Mac** ("not interested in the mac approach"). Not that fancy a man. Any coffee is coffee as long as it's not burnt.