istrain-public/APOCALYPSE-EDITION.md

# istrain — APOCALYPSE EDITION (public build guide)

*The complete from-scratch rebuild. This guide assumes you can install Linux and are comfortable
in a terminal, but it spells out every railroad-, SDR-, and container-specific step — because
those are the parts nobody tells you. If a step feels obvious to you, skip it; it's here for the
person behind you who needs the hint.*

*You are building a **passive** RF receiver. It only listens. No transmit, no license required in
the US to receive, no railroad cooperation needed — the trains announce themselves on public
frequencies, and you just have to be set up to hear them.*

> **Two ways to use this repo:**
> - **As a human:** read top to bottom, run the commands.
> - **As an agent** (point Claude Code / an LLM coding agent at this repo): the whole working tree
>   is here — `scripts/` (the DSP + decoders), `dashboard/` (the web UI + API server),
>   `docker/` (the container definitions), `config/` (templates). Every script has a header
>   comment explaining what it does and how it's operated. The design rules that keep it robust
>   are in §11. Start by reading this file, then `dashboard/serve.py` and `scripts/iq_hop.py`.

---

## 0. What you're building, in one minute

A train is a chain of radio transmitters. istrain listens passively to three of them and fuses
the result into one question: **is a train blocking my crossing right now?**

1. **Rail VHF voice (160–162 MHz, narrowband FM, unencrypted).** Crews, dispatchers, and trackside
   **defect detectors** speaking plain English: *"Norfolk Southern detector, milepost 148.9, no
   defects, total axles 460, speed 52, detector out."* A detector hit on your road channel means a
   train just passed that milepost — the most direct "train coming" signal there is, in words.
2. **End-of-Train telemetry (457.9375 MHz, 1200-baud FFSK).** The flashing box on the last car
   (EOT / FRED / marker) transmits its unit ID, **brake-pipe pressure**, and a **motion flag**
   every few seconds. Decode it and the air tells you: passing, stopped, brakes cut, or leaving.
3. **Head-of-Train (452.9375 MHz).** The locomotive's half of the same conversation. Decode the
   EOT unit it's addressing and you can **join head to tail** — proof of a whole train, plus
   warning from the front.

Bonus ears: **±12.5 kHz off the EOT/HOT centers** (distributed-power / mid-train repeaters live
there) and **NOAA weather radio (162.4–162.55 MHz)** as an always-on, known-good test signal.

**The two facts that make this work at ANY North American crossing:**
- **457.9375 / 452.9375 MHz is continent-wide.** Every EOT and HOT in North America uses this
  pair. You change nothing when you move.
- **The voice channels are public.** The AAR (Association of American Railroads) channel plan is
  published per railroad and per subdivision. Only your voice channel(s) change with location.

The synthesis: any one signal can be too weak to decode, but a real train lights several bands in
the same time window. Co-firing bands = a train, even on a modest antenna.

---

## 1. Hardware — the shopping list

| Item | What we run | Notes |
|---|---|---|
| SDR #1 — voice | **[RTL-SDR Blog V4](https://www.rtl-sdr.com/buy-rtl-sdr-dvb-t-dongles/)** | The V4's front-end filtering genuinely helps at VHF. This is the ears for 160–162 MHz. |
| SDR #2 — telemetry | Any RTL-SDR (R820T/R828D-class) | Runs the 452⇄457 hop. Generic sticks drift; the decoder compensates (§8). A TCXO stick is nicer, not required. |
| Antennas | ¼-wave verticals to start | **18.3 in (46.5 cm)** whip for 161 MHz; **6.4 in (16.3 cm)** for 457 MHz. Length is physics — get it right and a desk whip works day one. Upgrade: a tuned railroad-band base antenna (DPD TrainTenna, KB9VBR J-Pole) mounted high and far from your computers. Height beats gain; distance from noisy electronics beats both. |
| USB | A **powered** hub | Two dongles on one host; a powered hub prevents brownout resets. |
| Host | Any always-on 64-bit Linux box that runs Docker | Ours is a TrueNAS SCALE machine. A NUC, a Raspberry Pi 4/5, a Pi-like SBC, or an old desktop all work. 2+ cores, 2+ GB RAM. The whole rig is containers. |
| A second computer | Your laptop | For setup, watching waterfalls, and reading the dashboard. |

**Starting-from-nothing cost floor:** two dongles + whips ≈ US$70–100. Everything else is software
in this repo.

---

## 2. Operating system & base software

Any modern 64-bit Linux works. We'll give commands for **Debian/Ubuntu-family** (apt); translate
to your package manager as needed. Three viable host shapes:

- **A regular Linux box (recommended for a first build)** — Debian 12, Ubuntu 22.04+, Linux Mint,
  Raspberry Pi OS (64-bit). You control everything; nothing fights you. This guide's default.
- **An appliance NAS OS (what we ended on)** — TrueNAS SCALE, Unraid, etc., running Docker/Dockge.
  More convenient long-term, but the OS resists you (see the §4 DVB note — appliance `/etc` can
  reset on update). Do your *first* build on a regular box, move to the appliance once it works.
- **Bare metal vs. VM:** bare metal is simplest. A VM works but USB passthrough adds a failure
  mode and caps your sample rate — we ran in VirtualBox for a month and it was fine at 1.024 MS/s,
  but a Pi or NUC on bare metal is less fuss.

Install the base tools:

```bash
sudo apt update
sudo apt install -y git rtl-sdr librtlsdr-dev build-essential cmake pkg-config \
                    libusb-1.0-0-dev ffmpeg python3 python3-numpy docker.io docker-compose-plugin
```

- `rtl-sdr` gives you `rtl_test`, `rtl_eeprom`, `rtl_fm`, `rtl_sdr` — the osmocom tools that drive
  the dongles.
- `ffmpeg` is the audio plumbing (and the comms band-pass filter in front of transcription).
- `python3` + `numpy` run every DSP and decode script here — **no other Python packages are
  required** for the core rig (the scripts are deliberately stdlib + numpy only).
- Docker runs the whole stack. Optionally add **[Dockge](https://github.com/louislam/dockge)** — a
  light web UI for compose stacks; it's what we use, but plain `docker compose` is identical.

Clone this repo onto the host:

```bash
git clone <this-repo-url> istrain && cd istrain
```

---

## 3. Know your crossing — 30 minutes of desk research

This is the only location-specific work. Do it once.

1. **Identify the railroad and subdivision** at your crossing. Read the signage at the crossing
   itself (the operating railroad's name is on the signal equipment and the blue emergency-notify
   sign — that sign also lists a **DOT crossing number** and a phone number). Cross-check on
   [OpenRailwayMap](https://www.openrailwaymap.org/).
2. **Find the road (dispatch) channel.** Use two of these three and confirm they agree:
   - [RadioReference — US Railroads DB](https://www.radioreference.com/db/aid/7625)
   - [Railroad-Frequencies.com](https://www.railroad-frequencies.com/)
   - your region's railfan forum / site.
   You want the **AAR channel for the subdivision your crossing sits on**. Note its frequency and
   AAR channel number. Grab the **neighboring subdivisions' channels too** — pin 4–6 channels in a
   cluster; RTLSDR-Airband demodulates them all at once inside a single ~1 MHz window if they're
   within ~500 kHz of each other (most road channels cluster in 160.2–161.6 MHz).
3. **Find your defect detectors.** Railfan detector logs and forums list detector mileposts per
   subdivision. The detectors within ~10 miles either side of your crossing are your early-warning
   tripwires — write down their milepost numbers so the transcripts make sense.
4. **Note your NOAA weather frequency** (162.400–162.550, whichever is strongest near you). This is
   your receive-chain sanity signal forever — if you can hear NOAA, your antenna→dongle→software
   path works.

Write it all down. See `config/istrain.conf.example` — it's our worked example (a Norfolk Southern
subdivision in central Ohio); replace the five channel frequencies with yours.

---

## 4. Free the dongles from the TV driver (the step that bites everyone)

The Linux kernel grabs RTL-SDR dongles as **DVB television tuners** the instant they're plugged in,
and then no SDR software can open them. Blacklist the DVB driver on the **host** (this repo ships
the file at `config/blacklist-rtl-sdr.conf`):

```bash
sudo cp config/blacklist-rtl-sdr.conf /etc/modprobe.d/blacklist-rtl-sdr.conf
sudo modprobe -r dvb_usb_rtl28xxu 2>/dev/null   # unload it now
lsmod | grep dvb                                 # should print NOTHING
```

Reboot to make it stick. **The symptom of losing this fight later:** rtl_airband hangs at
"Allocating zero-copy buffers," or `rtl_test` says the device is busy or open failed. It is
*always* the DVB driver reloading — re-run the check above.

> ⚠ **On appliance OSes (TrueNAS/Unraid/etc.), `/etc` can be reset by a system update.** Re-verify
> this blacklist after every OS update. This is a permanent standing chore, not a one-time step.

**Serialize your dongles** so every service can pick the right one by name (index order changes on
reboot; serials don't). With only ONE dongle plugged in at a time:

```bash
rtl_eeprom -s 1002    # plug in dongle A alone, give it serial 1002 (your voice radio)
# unplug, plug in dongle B alone:
rtl_eeprom -s 1001    # dongle B = 1001 (your telemetry radio)
rtl_test               # with both in: lists both by serial; confirms they enumerate
```

(We use `1001`/`1002`; any string works — just match it in the configs.)

---

## 5. The stack — what runs

Everything is one Docker Compose stack (`docker/compose.example.yaml` + the Dockerfiles beside it):

```
▣ airband   RTLSDR-Airband (rtl-sdr-blog fork, V4-capable) on dongle 1002
            → demodulates your 4–6 pinned voice channels at once, squelch-gated
            → one mp3 per squelch opening + a line in heard.jsonl
▣ scanhop   rtl_tcp + scripts/iq_hop.py on dongle 1001
            → retunes in place: 452.9375 (10 s) ⇄ 457.9375 (10 s), forever
            → scripts/iq_channelize.py splits each dwell into center ±12.5 kHz sub-channels
            → burst → bot.jsonl / eot.jsonl / midtrain.jsonl + a raw .s16 clip
            → level.json written continuously (the live meter + "is capture alive" signal)
▣ eot-decode    scripts/eot-decode.py → scripts/eot/eot_scan.py : two-pass FFSK decode of EOT clips
                → unit ID · brake psi · motion flag, enriched back into eot.jsonl
▣ bot-recover   scripts/bot-recover.py : head-end frames → addressed-unit → head/tail join
▣ worker    scripts/transcribe-worker.py : voice clips >5 KB → ffmpeg comms band-pass
            → a Whisper server (§7) → transcript sidecars; real speech → transcripts.jsonl
▣ trim      scripts/trim-captures.py : prunes voice audio >24 h (the jsonl records are kept forever)
▣ web       dashboard/serve.py : stdlib Python, zero frameworks — the dashboard + all /api/* endpoints
```

Copy the compose template and edit the two host-specific things — your stack directory path and
your dongle serials (already `1001`/`1002` if you followed §4):

```bash
cp docker/compose.example.yaml docker/compose.yaml
cp config/istrain.conf.example config/istrain.conf   # then edit YOUR channels into it (§3)
```

---

## 6. Bring-up, step by step

1. **Selftest the DSP before any radio.** These run the whole signal chain on synthetic data — no
   dongle needed. Green here means the math works on your machine; anything failing after this is
   radio/USB, not code:
   ```bash
   python3 scripts/iq_hop.py --selftest
   python3 scripts/iq_channelize.py --selftest
   ```
2. **Prove the receive chain with NOAA first.** Before you wait for a train, confirm the whole
   antenna→dongle→software path works on a signal that's always there. Point a dongle at your NOAA
   frequency:
   ```bash
   rtl_fm -d 0 -f 162.550M -M fm -s 12k -g 40 - | aplay -r 12k -f S16_LE
   ```
   Weather robot voice = your chain works end to end. Silence = fix the antenna/gain/frequency
   before going further. (This is the single most useful debugging move; NOAA is your oscilloscope.)
3. **Edit `config/istrain.conf`** — replace the example channels with your voice frequencies from
   §3. Keep the shape: one `channels` entry per frequency, `squelch_snr_threshold` starting near
   the example's value (tune per §8), mp3 outputs into the captures dir, and your dongle serial in
   the device stanza.
4. **Build and start the stack** (from `docker/`):
   ```bash
   docker compose --profile radio --profile radio1001 build   # airband image builds first;
                                                              # scanhop copies rtl_tcp out of it
   docker compose up -d
   docker compose ps                                          # all containers Up
   ```
5. **Verify capture is alive by DATA, not by "is the process running."** Watch `data/level.json` —
   it should be rewritten every second or two (that's the hop breathing). Watch `data/captures/`
   for mp3s. This data-freshness habit is the whole health model (§11).
6. **Open the dashboard:** `http://<host>:3456`. The live meter should bounce; the correlate
   heatmap fills in 15-minute bins as bands fire.
7. **Wait for a train.** The first EOT decode is unmistakable: a unit ID, a brake pressure, and a
   motion flag from a machine a mile away, pulled out of static by a US$30 dongle. That's the
   moment the project becomes real.

---

## 7. Ears (optional, and the best part): transcription

Any Whisper-family ASR server works. We run **[faster-whisper](https://github.com/SYSTRAN/faster-whisper)
(model `small.en`) with VAD enabled** as its own small container/service, and
`scripts/transcribe-worker.py` posts each filtered voice clip to it (set `WHISPER_URL` /
`MILL_ASR` env to point at your server; the template default is `http://whisper:9000`).

Two rules learned the hard way:
- **Band-pass the audio first.** The worker runs an ffmpeg ~300–3400 Hz comms filter before ASR —
  Whisper hallucinates confident nonsense on raw squelch noise.
- **VAD on, always.** Without voice-activity detection, an hour of static becomes an hour of
  imaginary dispatcher chatter. Only clips with *detected speech* earn a line in
  `transcripts.jsonl` — the project's durable memory of everything the railroad said out loud
  near your crossing.

---

## 8. Getting the truth out of cheap hardware

- **Generic dongles drift off frequency.** Our EOT decoder (`scripts/eot/eot_scan.py`) is
  **two-pass**: an exact fast path, then an adaptive pass with fuzzy sync + tone tracking (±800 Hz)
  that recovers the off-frequency, short-burst emitters an exact decoder throws away as noise.
  Roughly half our real decodes come from the adaptive pass. If you write your own decoder, plan
  for drift from the start.
- **Squelch sits just above YOUR noise floor, not a textbook number.** Watch a quiet hour's
  squelch-opening rate. If you're capturing hundreds of empty clips, you're squelching on your own
  house's RF hash — we measured **~⅔ of desk-whip squelch openings were never radio at all.** Raise
  `squelch_snr_threshold` until the empties stop; lower it if you're missing weak transmissions.
- **Gain does not fix weak.** More gain amplifies the same noise. An A/B at gain 25 vs 40 vs auto on
  a buried signal sounds identical. The levers that actually work, in order: **antenna height,
  distance from electronics, a tuned antenna, better coax.**
- **Beware standing carriers.** We chased a "mystery mid-train transmitter" for days; it was a
  **fixed wayside EOT signal booster** (an FCC-licensed repeater) parked on 457.925 that never
  moves. Card it as furniture and subtract its floor from your correlation logic, or it fakes a band
  forever.
- **Trust the decoder, not quick statistics.** Every shortcut we tried to characterize undecodable
  bursts by spectral/envelope stats failed its own control test (a measure that couldn't tell
  modulated from unmodulated on *known-good* clips proves nothing). The decoder is the only
  instrument that doesn't lie.

---

## 9. Reading a train (the payoff)

From decoded EOT frames, per train, the **brake-pipe pressure curve is the story silence can't
tell:**

| Air curve | Meaning |
|---|---|
| ~85–90 psi + motion flag set | **Passing** through |
| Drops to a held ~55–65, motion 0 | **Dwelling** — brakes set, possibly blocking the crossing |
| Bleeds to 0 and holds, head still polling | **Cut & standing** — crew cut the crossing; the road may be OPEN even though a train is "there" |
| Recharges toward ~88, motion 0→1 | **Departing** — the roll-out is your event time |

**Never read a bleed-to-zero as a departure** — that mistake reports a blocked crossing as clear.
This taxonomy came from ~50 ground-truthed passages; yours may differ by railroad and operating
pattern, so watch and record before you trust it.

---

## 10. Security — before you point this at the internet

The dashboard (`dashboard/serve.py`) is safe to run on your LAN as-is. Before exposing it publicly
(a reverse proxy, a Cloudflare tunnel, a port-forward), know these:

- **Run it read-only when public.** Set the `SUPERVISOR_SNAPSHOT` env (any path) to mark the node
  as a **presentation node** — that disables clip deletion and refuses control POSTs. Leave it
  unset only on a private box you alone reach.
- **Leave `PUSH_KEY` unset unless you actually use the VM→server push path.** Unset = the
  `/api/push` and `/api/push-clip` write endpoints are disabled entirely. If you set a key, treat
  it as a real secret (env only, never committed) — it can overwrite the data your site displays.
- **The control endpoints are speed-bumped, not authenticated.** They require an `X-Ops` header
  (so blind scanners bounce), but that's a bump, not a password. Real control should stay on your
  LAN/tailnet, never bare on the public name. Put auth at your proxy if you want remote control.
- **`/api/errors/report` is intentionally public** (browser error beacon) — it's rate-limited and
  length-capped server-side; that's the only endpoint that accepts unauthenticated writes, and it
  only appends bounded strings to a log.
- **Never expose the raw container port to the internet.** Front it with a proxy that terminates
  TLS and, ideally, gates writes.

There are **no credentials, keys, or personal data in this repository.** The frequencies are
public; the code is stdlib. Keep it that way in your fork — put secrets in env, not in files.

---

## 11. The design rules that keep it robust (read if you're modifying it)

These were bought with outages and false readings. Keep them:

- **Every capture entrypoint exits nonzero when its dongle is missing.** The container restart
  policy (`restart: unless-stopped`) does the waiting — so a replug or a cold reboot self-heals
  with no human. (We lost a morning to a process that exited *zero* on a missing dongle and sat
  there looking alive.)
- **No startup ordering between containers.** Each retries until its own dependencies appear —
  impossible to deadlock on boot order.
- **Decoders never delete clips.** Capture, decode, and cleanup are separate jobs; a decoder bug
  can't eat your evidence.
- **Health is read from data freshness, never from "is the process up."** `level.json`'s timestamp
  age, the last capture's age per band, the transcript trickle — a fresh timestamp is the only
  health signal a radio can't fake.
- **Keep one append-only, numbered observations log.** Every "mystery signal" we resolved, we
  resolved by writing down what we actually *measured* — and the two findings we later retracted
  taught more than most that stood. Number the entries. Record the retractions. Future-you (or the
  next forker) is who you're writing for.

---

## 12. What's in this repo

```
APOCALYPSE-EDITION.md   ← you are here
README.md               ← the front door + pointers
LICENSE                 ← GPLv3
ATTRIBUTION.md          ← third-party code (PyEOT) + the shoulders this stands on
scripts/                ← the signal engine + decoders (stdlib + numpy only)
  iq_hop.py             ← retune-in-place 452⇄457 hop (the mission radio)
  iq_channelize.py      ← splits a dwell into center + ±12.5 kHz sub-channels
  eot-decode.py, eot/   ← two-pass FFSK EOT decoder (eot_scan.py + vendored PyEOT for BCH)
  bot-recover.py        ← head-end frame recovery + head/tail join
  transcribe-worker.py  ← voice clip → comms filter → Whisper → transcript
  vumon.py              ← the live level meter
  correlate.py          ← the cross-band "why" engine behind the heatmap
  trim-captures.py      ← the 24 h voice-audio janitor
  scan-hop.py, iq_retune.py, envelope.py, probe-band.py ← the hop wrapper + analysis helpers
dashboard/              ← serve.py (the API + static server) + the web UI
docker/                 ← compose template + Dockerfiles (web/airband/scanhop/worker)
config/                 ← istrain.conf.example (channels), blacklist-rtl-sdr.conf
```

---

*Built at a Norfolk Southern crossing in central Ohio, 2026, on RTL-SDR Blog V4 hardware, by a
human and his AI pair-programmer. The live instance runs at
[istrain.jhestyr.net](https://istrain.jhestyr.net). This repository is the whole thing — read it,
run it, fork it, and if you build one, tell us what you heard.*