local-horse/NOTES.md

# local-horse — findings log

Running, numbered (`N-1`, `N-2`, …). Each entry = something we **learned and want to keep**: a signature,
a gotcha, a decision, a constraint — the stuff we'd otherwise re-derive. NOT the session blow-by-blow (that
lives in the weave thread); this is the distilled, durable record a fresh hand reads to catch up fast.

Newest at the bottom. When a truth changes, **update every place it lives** — a stale note is worse than none.

---

## N-1 (2026-07-17) — Origin: the K3 repricing that started this

Kimi K3 dropped 2026-07-16 (Moonshot AI, 2.8T-param open MoE, weights due **2026-07-27**; API $0.30/M
cached-in · $3/M in · $15/M out). Repricing the adventure game's real build tokens (10 sessions,
2026-07-05..17, summed from `~/.claude/projects/-home-obx/*.jsonl`, deduped by message id):

- **1,147M cache-read** + 18M uncached-in + 3.5M out → **≈ $451 at K3 rates** (±30%, session-level
  granularity) vs ≈ $1,307 at Claude list prices. Actual marginal cost: $0 (subscription).
- The load-bearing fact: agentic coding is **~99% cache reads, ~1% output** — cache-*dominated*.
  (Corrected per N-5: cache reads are **hits**, not reprocessing. Local first-order requirements =
  KV-cache persistence + hit rate + capacity; raw prefill bandwidth matters on the misses only.
  Operational gotcha: harnesses that mutate the prompt *prefix* silently break llama.cpp/vLLM prefix
  caching — a slower box that keeps a 100K-token cache warm beats a faster box that keeps evicting it.)
- K3 itself is never local: ~1.4TB weights at 4-bit. "Local like that" = best open model that fits a box.

## N-2 (2026-07-17) — The open-model landscape, mid-2026

- **GLM 5.2** (Zhipu, MIT, released 2026-06-13): first open model to beat GPT-5.5 on SWE-Bench Pro
  (62.1; Terminal-Bench 2.1: 81.0). Current open coding champ. **Exact memory footprint unverified — check
  before sizing hardware.**
- **Qwen3-Coder 480B-A35B**: best downloadable coder with known footprint (~270GB @ 4-bit; 35B active).
  The 80B-A3B variant ≈ 96% of the quality at single-workstation size.
- **Kimi K2.6**: strongest for agentic *stability* (consistent tool calling, recoverable failures). ~1T
  params — barely/slowly fits the workhorse at aggressive quant.
- **The lag law** (jhestyr's intuition, confirmed): open/distilled models run ≈ **12 months behind
  frontier**, and the lag is stable. A $20k box today ≈ mid-2025 frontier (Sonnet-4-era agentic coding).
  K3 weights (07-27) will spawn distills sized for the workhorse envelope.
- Capability ceiling, honestly: scoped features / refactors / tests / review on a known repo = credible.
  Long-horizon overnight autonomy, judgment, Opus/Fable-tier storekeeping = not at any local price.

## N-3 (2026-07-17) — GOTCHA: the 2026 DRAM crisis broke the classic build

Server DDR5 RDIMM prices up **>400% since mid-2025** (64GB: ~$255 Q3'25 → >$900 Q1'26 → $1,200–2,300
mid-2026; SK Hynix sold out through 2026, fabs pivoted to HBM). Consequences:

- The classic "EPYC + 768GB DDR5 + one big GPU" build went from ~$16k to **$28–42k**. Dead at $20k.
- Workaround: drop to **used Milan + refurb DDR4-3200** (~half the bandwidth, ~1/6 the price) — see BUILDS.md.
- **Used 3090s got *relatively* cheaper** (GDDR6X didn't spike) — the multi-card rig gained ground.
- RTX Pro 6000 Blackwell 96GB: Nvidia list jumped 55% in 16 months to $13,250; street/Max-Q hunts lower.
- Relief not expected before **2027**. Waiting is a legitimate strategy; re-price at buy time.

## N-4 (2026-07-17) — Decision lean (not a decision) — ⚠️ SUPERSEDED BY N-5

Ezra's recommendation if/when the itch turns real: **the workhorse** (Milan + 512GB DDR4 refurb + one
RTX Pro 6000 Max-Q, ~$15–19k) over the 8×3090 rig — a smarter model that's slower wastes fewer turns
than a fast one that circles. Quiet, <1kW, 120V, racks by the Mill, and the skeleton upgrades (RAM,
second card slot) without replacing bones. Integration shape: headless Debian, llama.cpp/vLLM serving an
OpenAI-compatible endpoint on the tailnet; per [[mill-is-home-for-workloads]] it's a Mill annex, not a
new species. Sensible trigger to revisit: **after 2026-07-27** (K3 weights + first distills) or when
DDR5 unclenches.

**Superseded 2026-07-17 (same day) by N-5:** the workhorse's decode-rate assumption didn't survive
external review. Lean is now benchmark-gated; see N-5.

## N-5 (2026-07-17) — External review (GPT-5.6 via jhestyr's friend) — what survived, what didn't

jhestyr shared BUILDS.md out (repo made public for it) and brought back a GPT-5.6-thinking review.
Graded on merits; most of it holds. **Accepted corrections** (BUILDS.md updated to match):

1. **Workhorse decode rate was wrong.** My "15–25 tok/s" for Qwen3-Coder-480B silently assumed
   CPU-executed experts. Naive vLLM/llama.cpp offload ships ~11GB of active expert weights over PCIe
   (~25GB/s) *per token* → **2–3 tok/s ceiling**. KTransformers-style CPU-expert execution could reach
   low teens against 205GB/s DDR4 — but its fast paths want AVX-512/AMX, and **Milan is AVX2** (native
   AVX2 support still maturing as of Feb 2026). Honest number: **unknown; plan at ~5 tok/s** until
   benchmarked on this exact combo. The workhorse is a hypothesis, not a validated build.
2. **Cache semantics were backward** (fixed in N-1): cache reads = hits, not re-prefill. Persistence/
   hit-rate/capacity are first-order; prefill bandwidth is the miss path. Corollary gotcha: Claude
   Code-style harnesses can mutate the prompt prefix and silently kill prefix-cache reuse.
3. **KV-cache headroom ignored:** Qwen3-480B at 100K context ≈ 24GiB BF16 (12 FP8) of KV — the 96GB
   card really offers ~60–75GB for weights, pushing MORE experts to RAM. Compounds #1.
4. **Feral cats under-engineered:** ROMED8-2T has **7** x16 slots — my table put 8 GPUs on it with no
   bifurcation plan, lane map, Above-4G/MMIO validation, or power engineering. Also: 3090s DO have
   pairwise NVLink (no 8-way fabric) — my "no NVLink" line was too flat.
5. **Capability tiers not decision-grade:** "Sonnet-4-era" conflates full-precision benchmarks with an
   aggressive quant on an exotic execution path. The metric that matters: **successful agent turns per
   hour** on OUR repos. And test the **80B-A3B first** (≈96% quality) — if it holds, one GPU suffices.
6. **Economics:** subscription makes marginal Claude cost $0 — the box is justified by privacy/control/
   unlimited background tokens, not savings. Add TCO: ~$0.15/kWh → 700W continuous ≈ $920/yr.
   (My pushback: our duty cycle isn't continuous; idle ~100–150W. Real number is a few hundred $/yr.)

**Pushback recorded (minor):** the 2–3 tok/s "upper bound" assumes uniformly distributed expert
selection with zero locality; hot-expert pinning skews that upward in practice — but direction stands
and the burden of proof is ours. **New lean (replaces N-4):** *benchmark before buying* — the friend's
strongest frame: **one RTX Pro 6000 running a strong 80–120B local model, with cloud escalation for
hard turns, is more defensible than either full build.** Benchmark gate lives in BUILDS.md.