commit c46ff918a20c6cc7f809c71ddecd68648bb5f1e7 Author: Silas Date: Fri Jul 17 11:02:54 2026 -0400 local-horse born: $20k local agentic-coding builds priced (workhorse vs feral cats), model tiers, DRAM-crisis gotcha Rumination 2026-07-17, not a purchase order. Front-door README + NOTES N-1..N-4 + BUILDS.md. [ezra] diff --git a/BUILDS.md b/BUILDS.md new file mode 100644 index 0000000..43ef85e --- /dev/null +++ b/BUILDS.md @@ -0,0 +1,78 @@ +# local-horse — the two priced builds (as of 2026-07-17) + +⚠️ **Prices dated 2026-07-17, ±15% on used parts and worse on RAM/GPU (see NOTES N-3). Re-price before buying.** +Budget target: ~$20k. Constraint: no Mac. Both shapes serve an OpenAI-compatible endpoint on the tailnet +(llama.cpp / vLLM, headless Debian, Dockge like everything else). + +--- + +## Build 1 — The Workhorse (one big card + cheap capacity) + +Quiet, <1kW under load, standard 120V outlet, racks next to the Mill. The pick (NOTES N-4). + +| Part | Spec | Est. price (2026-07-17) | +|---|---|---| +| GPU | RTX Pro 6000 Blackwell **Max-Q** 96GB (300W blower, 1.8TB/s) | $9,500–13,000 | +| CPU | EPYC 7763 used (64c Milan, 8-ch DDR4; full 8 CCDs = full bandwidth) | ~$1,200 | +| Board | Supermicro H12SSL-i | ~$650 | +| RAM | 8× 64GB DDR4-3200 RDIMM refurb (512GB, ~205GB/s) | ~$2,000–2,800 | +| Storage | 4TB NVMe | ~$450 | +| Chassis/PSU | 4U + 1300W + cooler + fans | ~$900 | +| **Total** | | **~$15k–19k** | + +What it runs (MoE trick: hot path/attention in VRAM, cold experts in system RAM): + +| Model | Fit | Speed (est.) | Capability, plainly | +|---|---|---|---| +| GPT-OSS-120B (5B active) | Entirely in VRAM @ 4-bit | 100+ tok/s | Reliable daily hand — summaries, scripts, small fixes. ~Sonnet-3.5 tier. | +| **Qwen3-Coder 480B-A35B** @ 4-bit (~270GB) | Hybrid VRAM+RAM | ~15–25 tok/s | **The prize.** Real agentic coding on a known repo. Sonnet-4-era. | +| GLM 5.2 quantized | Hybrid (footprint unverified) | similar | Current open SWE-Bench champ; same tier or a notch up. | +| Kimi K2.6 (1T) @ Q3 | Barely (608GB total, tight) | single digits | Party trick, not a daily driver. | +| K3 (2.8T) | No | — | Cloud only. Its **distills** should land in this envelope. | + +## Build 2 — The Feral Cats (8× used RTX 3090, 192GB VRAM) + +Everything model-resident → prefill screams. Also screams literally; needs a **240V circuit** (~3kW load), +~150W combined idle 24/7, eight points of used-silicon failure. + +| Part | Spec | Est. price (2026-07-17) | +|---|---|---| +| GPUs | 8× used RTX 3090 24GB (~$850 ea; GDDR6X dodged the DRAM spike) | ~$6,800 | +| CPU | EPYC 7402 used (Rome — just needs the PCIe lanes) | ~$300 | +| Board | ASRock Rack ROMED8-2T (7× PCIe x16) | ~$700 | +| RAM | 256GB DDR4 refurb | ~$1,000 | +| Risers/frame | Bifurcation risers + open frame or 4U conversion | ~$500 | +| Power | 2× 1600W PSU + sync board, 240V | ~$650 | +| Storage/misc | NVMe, fans, zip ties, regret | ~$550 | +| **Total** | | **~$10.5k–12k** | + +| Model | Fit | Speed (est.) | Capability, plainly | +|---|---|---|---| +| **Qwen3-235B-A22B** @ 4-bit (~130GB) | Fully in VRAM — the rig's sweet spot | fast decode AND prefill | Strong all-rounder one tier below Coder-480B. Best agentic feel per dollar here. | +| GPT-OSS-120B | Fully in VRAM, headroom | blazing | Same as workhorse, faster. | +| 70B-class dense (Llama/Qwen) | Trivially | blazing | Helper tier. | +| Qwen3-Coder 480B / GLM 5.2 | **Doesn't fit** at useful quant | offload defeats the rig's point | The one thing the workhorse does that this can't. | +| K2.6 / K3 | No | — | — | + +## The trade in one sentence + +Workhorse reaches the **best** open coding models slowly-but-surely and behaves like an appliance; the +cats run one tier down blisteringly fast for half the money, paid for in a 240V run, fan roar, and jank. +For agentic loops (99% context re-reads — NOTES N-1) all-VRAM prefill genuinely matters, but model +ceiling matters more: a smarter slow model wastes fewer turns than a fast one going in circles. + +## Why one big card and not 8 (the physics, kept for re-derivation) + +1. **Power/noise:** 8×350W = 2.8kW load — over a 15A/120V circuit's ceiling; plus ~150W idle, forever. +2. **Interconnect tax:** split models cross PCIe (~32GB/s) every token vs 1.8TB/s on-card; consumer cards + have no P2P/NVLink (disabled on 4090+). Multi-GPU wins at batch *throughput* (many users), not + single-user latency — and the counter has one user. +3. **MoE changed the buy:** ~35B active params need one fast GPU + cheap capacity, not 8 GPUs of compute. + +## Sources (as read 2026-07-17) + +- Open-model landscape: digitalapplied.com (models↔hardware matching) · llmconfigurator.com (local coding + report) · modal.com (SWE-bench open models) · mindstudio.ai (agentic open-source 2026) +- Hardware/pricing: videocardz.com (RTX Pro 6000 list $13,250) · thundercompute.com (build pricing) · + corewavelabs.com + pcserverandparts.com (DRAM crisis) · tomshardware.com RAM price index · memory.net +- K3: venturebeat.com · cnbc.com · marktechpost.com · simonwillison.net · platform.kimi.ai quickstart diff --git a/NOTES.md b/NOTES.md new file mode 100644 index 0000000..ff67284 --- /dev/null +++ b/NOTES.md @@ -0,0 +1,57 @@ +# local-horse — findings log + +Running, numbered (`N-1`, `N-2`, …). Each entry = something we **learned and want to keep**: a signature, +a gotcha, a decision, a constraint — the stuff we'd otherwise re-derive. NOT the session blow-by-blow (that +lives in the weave thread); this is the distilled, durable record a fresh hand reads to catch up fast. + +Newest at the bottom. When a truth changes, **update every place it lives** — a stale note is worse than none. + +--- + +## N-1 (2026-07-17) — Origin: the K3 repricing that started this + +Kimi K3 dropped 2026-07-16 (Moonshot AI, 2.8T-param open MoE, weights due **2026-07-27**; API $0.30/M +cached-in · $3/M in · $15/M out). Repricing the adventure game's real build tokens (10 sessions, +2026-07-05..17, summed from `~/.claude/projects/-home-obx/*.jsonl`, deduped by message id): + +- **1,147M cache-read** + 18M uncached-in + 3.5M out → **≈ $451 at K3 rates** (±30%, session-level + granularity) vs ≈ $1,307 at Claude list prices. Actual marginal cost: $0 (subscription). +- The load-bearing fact: agentic coding is **~99% context re-reads, ~1% output**. Cached-input price + (cloud) / prefill speed + KV-cache persistence (local) dominate everything. +- K3 itself is never local: ~1.4TB weights at 4-bit. "Local like that" = best open model that fits a box. + +## N-2 (2026-07-17) — The open-model landscape, mid-2026 + +- **GLM 5.2** (Zhipu, MIT, released 2026-06-13): first open model to beat GPT-5.5 on SWE-Bench Pro + (62.1; Terminal-Bench 2.1: 81.0). Current open coding champ. **Exact memory footprint unverified — check + before sizing hardware.** +- **Qwen3-Coder 480B-A35B**: best downloadable coder with known footprint (~270GB @ 4-bit; 35B active). + The 80B-A3B variant ≈ 96% of the quality at single-workstation size. +- **Kimi K2.6**: strongest for agentic *stability* (consistent tool calling, recoverable failures). ~1T + params — barely/slowly fits the workhorse at aggressive quant. +- **The lag law** (jhestyr's intuition, confirmed): open/distilled models run ≈ **12 months behind + frontier**, and the lag is stable. A $20k box today ≈ mid-2025 frontier (Sonnet-4-era agentic coding). + K3 weights (07-27) will spawn distills sized for the workhorse envelope. +- Capability ceiling, honestly: scoped features / refactors / tests / review on a known repo = credible. + Long-horizon overnight autonomy, judgment, Opus/Fable-tier storekeeping = not at any local price. + +## N-3 (2026-07-17) — GOTCHA: the 2026 DRAM crisis broke the classic build + +Server DDR5 RDIMM prices up **>400% since mid-2025** (64GB: ~$255 Q3'25 → >$900 Q1'26 → $1,200–2,300 +mid-2026; SK Hynix sold out through 2026, fabs pivoted to HBM). Consequences: + +- The classic "EPYC + 768GB DDR5 + one big GPU" build went from ~$16k to **$28–42k**. Dead at $20k. +- Workaround: drop to **used Milan + refurb DDR4-3200** (~half the bandwidth, ~1/6 the price) — see BUILDS.md. +- **Used 3090s got *relatively* cheaper** (GDDR6X didn't spike) — the multi-card rig gained ground. +- RTX Pro 6000 Blackwell 96GB: Nvidia list jumped 55% in 16 months to $13,250; street/Max-Q hunts lower. +- Relief not expected before **2027**. Waiting is a legitimate strategy; re-price at buy time. + +## N-4 (2026-07-17) — Decision lean (not a decision) + +Ezra's recommendation if/when the itch turns real: **the workhorse** (Milan + 512GB DDR4 refurb + one +RTX Pro 6000 Max-Q, ~$15–19k) over the 8×3090 rig — a smarter model that's slower wastes fewer turns +than a fast one that circles. Quiet, <1kW, 120V, racks by the Mill, and the skeleton upgrades (RAM, +second card slot) without replacing bones. Integration shape: headless Debian, llama.cpp/vLLM serving an +OpenAI-compatible endpoint on the tailnet; per [[mill-is-home-for-workloads]] it's a Mill annex, not a +new species. Sensible trigger to revisit: **after 2026-07-27** (K3 weights + first distills) or when +DDR5 unclenches. diff --git a/README.md b/README.md new file mode 100644 index 0000000..3be22d9 --- /dev/null +++ b/README.md @@ -0,0 +1,27 @@ +# local-horse — pricing & planning a local LLM box for agentic coding + +> ### 📖 READ THESE FIRST — don't re-derive, don't build blind +> Before suggesting, changing, or buying anything on this project: +> - **[`NOTES.md`](NOTES.md)** — the running findings log (`N-1…`): the K3 cost math that started this, the RAM crisis, the decision lean. +> - **[`BUILDS.md`](BUILDS.md)** — the two priced builds (workhorse / feral cats) with the models each can run. **Prices dated 2026-07-17; the market is volatile — re-price before buying.** +> - **Weave** — `admin/silas/SCRATCHPAD.md` threads + ledger lines = live state / open lines. + +## What it is + +A **rumination, not a purchase order** (status as of 2026-07-17). jhestyr asked what it would take to +run local models — first "something like Kimi K3," then specifically **agentic coding like we do at the +counter**, best-in-class around **$20k**. This repo holds the priced-out answer so future-us doesn't +re-derive it: two build shapes, the models each runs, honest capability tiers, and the market caveats +that will invalidate the numbers over time. + +The trigger was real data: repricing the lassiter-creek-adventure-game's build (10 sessions, Jul 5–17, +from local transcripts) showed **1.15B cache-read tokens vs 3.5M output** — agentic coding is ~99% +re-reading context. That shape drives every hardware conclusion here (prefill/memory-bandwidth is the +bottleneck, not generation speed). + +Standing constraints from jhestyr: **no Mac** ("not interested in the mac approach"). Not that fancy a +man. Any coffee is coffee as long as it's not burnt. + +## How we work +Read-first · write the note where next-me reads it · **verify it landed** · talk before building · one thing +at a time · don't over-fix. The state of truth is this repo — keep it self-narrating.