local-horse/NOTES.md

local-horse — findings log

Running, numbered (N-1, N-2, …). Each entry = something we learned and want to keep: a signature, a gotcha, a decision, a constraint — the stuff we'd otherwise re-derive. NOT the session blow-by-blow (that lives in the weave thread); this is the distilled, durable record a fresh hand reads to catch up fast.

Newest at the bottom. When a truth changes, update every place it lives — a stale note is worse than none.

---

N-1 (2026-07-17) — Origin: the K3 repricing that started this

Kimi K3 dropped 2026-07-16 (Moonshot AI, 2.8T-param open MoE, weights due 2026-07-27; API $0.30/M cached-in · $3/M in · $15/M out). Repricing the adventure game's real build tokens (10 sessions, 2026-07-05..17, summed from ~/.claude/projects/-home-obx/*.jsonl, deduped by message id):

• 1,147M cache-read + 18M uncached-in + 3.5M out → ≈ $451 at K3 rates (±30%, session-level granularity) vs ≈ $1,307 at Claude list prices. Actual marginal cost: $0 (subscription).
• The load-bearing fact: agentic coding is ~99% cache reads, ~1% output — cache-dominated. (Corrected per N-5: cache reads are hits, not reprocessing. Local first-order requirements = KV-cache persistence + hit rate + capacity; raw prefill bandwidth matters on the misses only. Operational gotcha: harnesses that mutate the prompt prefix silently break llama.cpp/vLLM prefix caching — a slower box that keeps a 100K-token cache warm beats a faster box that keeps evicting it.)
• K3 itself is never local: ~1.4TB weights at 4-bit. "Local like that" = best open model that fits a box.

N-2 (2026-07-17) — The open-model landscape, mid-2026

• GLM 5.2 (Zhipu, MIT, released 2026-06-13): first open model to beat GPT-5.5 on SWE-Bench Pro (62.1; Terminal-Bench 2.1: 81.0). Current open coding champ. Exact memory footprint unverified — check before sizing hardware.
• Qwen3-Coder 480B-A35B: best downloadable coder with known footprint (~270GB @ 4-bit; 35B active). The 80B-A3B variant ≈ 96% of the quality at single-workstation size.
• Kimi K2.6: strongest for agentic stability (consistent tool calling, recoverable failures). ~1T params — barely/slowly fits the workhorse at aggressive quant.
• The lag law (jhestyr's intuition, confirmed): open/distilled models run ≈ 12 months behind frontier, and the lag is stable. A $20k box today ≈ mid-2025 frontier (Sonnet-4-era agentic coding). K3 weights (07-27) will spawn distills sized for the workhorse envelope.
• Capability ceiling, honestly: scoped features / refactors / tests / review on a known repo = credible. Long-horizon overnight autonomy, judgment, Opus/Fable-tier storekeeping = not at any local price.

N-3 (2026-07-17) — GOTCHA: the 2026 DRAM crisis broke the classic build

Server DDR5 RDIMM prices up >400% since mid-2025 (64GB: ~$255 Q3'25 → >$900 Q1'26 → $1,200–2,300 mid-2026; SK Hynix sold out through 2026, fabs pivoted to HBM). Consequences:

• The classic "EPYC + 768GB DDR5 + one big GPU" build went from ~$16k to $28–42k. Dead at $20k.
• Workaround: drop to used Milan + refurb DDR4-3200 (~half the bandwidth, ~1/6 the price) — see BUILDS.md.
• Used 3090s got relatively cheaper (GDDR6X didn't spike) — the multi-card rig gained ground.
• RTX Pro 6000 Blackwell 96GB: Nvidia list jumped 55% in 16 months to $13,250; street/Max-Q hunts lower.
• Relief not expected before 2027. Waiting is a legitimate strategy; re-price at buy time.

N-4 (2026-07-17) — Decision lean (not a decision) — ⚠️ SUPERSEDED BY N-5

Ezra's recommendation if/when the itch turns real: the workhorse (Milan + 512GB DDR4 refurb + one RTX Pro 6000 Max-Q, ~$15–19k) over the 8×3090 rig — a smarter model that's slower wastes fewer turns than a fast one that circles. Quiet, <1kW, 120V, racks by the Mill, and the skeleton upgrades (RAM, second card slot) without replacing bones. Integration shape: headless Debian, llama.cpp/vLLM serving an OpenAI-compatible endpoint on the tailnet; per mill-is-home-for-workloads it's a Mill annex, not a new species. Sensible trigger to revisit: after 2026-07-27 (K3 weights + first distills) or when DDR5 unclenches.

Superseded 2026-07-17 (same day) by N-5: the workhorse's decode-rate assumption didn't survive external review. Lean is now benchmark-gated; see N-5.

N-5 (2026-07-17) — External review (GPT-5.6 via jhestyr's friend) — what survived, what didn't

jhestyr shared BUILDS.md out (repo made public for it) and brought back a GPT-5.6-thinking review. Graded on merits; most of it holds. Accepted corrections (BUILDS.md updated to match):

1. Workhorse decode rate was wrong. My "15–25 tok/s" for Qwen3-Coder-480B silently assumed CPU-executed experts. Naive vLLM/llama.cpp offload ships ~11GB of active expert weights over PCIe (~25GB/s) per token → 2–3 tok/s ceiling. KTransformers-style CPU-expert execution could reach low teens against 205GB/s DDR4 — but its fast paths want AVX-512/AMX, and Milan is AVX2 (native AVX2 support still maturing as of Feb 2026). Honest number: unknown; plan at ~5 tok/s until benchmarked on this exact combo. The workhorse is a hypothesis, not a validated build.
2. Cache semantics were backward (fixed in N-1): cache reads = hits, not re-prefill. Persistence/ hit-rate/capacity are first-order; prefill bandwidth is the miss path. Corollary gotcha: Claude Code-style harnesses can mutate the prompt prefix and silently kill prefix-cache reuse.
3. KV-cache headroom ignored: Qwen3-480B at 100K context ≈ 24GiB BF16 (12 FP8) of KV — the 96GB card really offers ~60–75GB for weights, pushing MORE experts to RAM. Compounds #1.
4. Feral cats under-engineered: ROMED8-2T has 7 x16 slots — my table put 8 GPUs on it with no bifurcation plan, lane map, Above-4G/MMIO validation, or power engineering. Also: 3090s DO have pairwise NVLink (no 8-way fabric) — my "no NVLink" line was too flat.
5. Capability tiers not decision-grade: "Sonnet-4-era" conflates full-precision benchmarks with an aggressive quant on an exotic execution path. The metric that matters: successful agent turns per hour on OUR repos. And test the 80B-A3B first (≈96% quality) — if it holds, one GPU suffices.
6. Economics: subscription makes marginal Claude cost $0 — the box is justified by privacy/control/ unlimited background tokens, not savings. Add TCO: ~$0.15/kWh → 700W continuous ≈ $920/yr. (My pushback: our duty cycle isn't continuous; idle ~100–150W. Real number is a few hundred $/yr.)

Pushback recorded (minor): the 2–3 tok/s "upper bound" assumes uniformly distributed expert selection with zero locality; hot-expert pinning skews that upward in practice — but direction stands and the burden of proof is ours. New lean (replaces N-4): benchmark before buying — the friend's strongest frame: one RTX Pro 6000 running a strong 80–120B local model, with cloud escalation for hard turns, is more defensible than either full build. Benchmark gate lives in BUILDS.md.