diff --git a/docs/superpowers/specs/2026-10-30-vt1200-video-streaming-design.md b/docs/superpowers/specs/2026-10-30-vt1200-video-streaming-design.md new file mode 100644 index 0000000..b484c43 --- /dev/null +++ b/docs/superpowers/specs/2026-10-30-vt1200-video-streaming-design.md @@ -0,0 +1,305 @@ +# VT1200 Ethernet video streaming — design + +Date: 2026-10-30 +Status: **rev 3** — corrected after a builder review; the open question is answered +(§14). Awaiting approval. +Author: (session). Decisions marked "**call:**" are the builder's proposals, not +something the human partner has agreed to. + +--- + +## 1. Intent + +Play **Bad Apple!!** on a real VT1200, **full screen (1280×1024, 1 bpp)**, streamed +over the terminal's Ethernet port at a rate that reads as motion, using only code we +ship at runtime — the ROM stays byte-identical +(`md5 b46a3e442ccb86134d2356bd70d22fd2`). + +"Optimized" means: per frame the VT1200 CPU does a handful of *token* operations and +the **graphics engine moves every output pixel**. No per-pixel or per-bit CPU work +anywhere. + +Secondary outcome: the transport and decoder are a **reusable streaming pipeline** +(any 1-bpp animation, not just Bad Apple), with Bad Apple as the acceptance test. + +## 2. What exists today (grounding, all measured on the unit) + +* **Payload toolchain**: `harness/udpstream/` builds a RAM payload with the 1987 TI + SDK (`.text` at bit `0x018C0000`, `.bss` at `0x01C00000`, 256 KB — i.e. bytes + `0x380000`–`0x3C0000`) that talks raw Ethernet through its own LANCE driver. +* **Transport**: raw LANCE RX ring, **16 descriptors × 1526 B**, buffers at bit + `B(0x4000)`. Sustains **856–858 fps** of 1514-byte frames = **1.27 MB/s**, 99.9 % + delivery, on a **10 Mbit half-duplex hub** (`NETWORK_REPORT` §72). +* **Graphics engine**: `FILL L` **0.40 µs per destination word**, `PIXBLT L,XY` + **0.62 µs per destination word** on the unit; `vram_fill_()`/`vram_blit_()` exist + (`harness/udpstream/vram.asm`), including the `SPTCH=0` row-replication trick. +* **Firmware precedent** (§74.3): the ROM blits text with `PIXBLT B,XY` (packed + 1-bpp source → XY destination) and rectangles with `PIXBLT L,L`, both chunked at + `0x7FC0` pixels. +* **CPU ceiling**: a C byte-store loop costs **12.2–13.9 µs per byte**; a hand + written asm word loop measures **0.97 µs/word** (MAME). The C number is codegen, + but even asm is 1.5–3× worse per word than the blitter and shares its bus. +* **Display**: 1280×1024 mono at **72 Hz** (13.9 ms), 256-byte VRAM row pitch, VRAM + at bit `0x04000000`. +* **Heap**: the firmware's allocator is reachable from a payload (`ROM_MALLOC`, + bit `0x3F701230`, "verified live"): bit-addressed first-fit, returns a **bit** + pointer (`0` on failure), block header is 8 bytes — never write below `ptr-8`. + `dwt/notes/HEAP.md` gives the safe pattern: **one allocation at startup**, from + the payload's own process, never from an ISR, keep the pointer in a callee-saved + register or a fixed marker. +* **Cost is per destination word.** A blit of a 1280-px strip replicated down *S* + rows still writes 1280×*S* destination pixels. The `SPTCH=0` trick saves **wire + bytes, not blitter time**. + +## 3. Goals + +1. Full-screen 1280×1024 1-bpp playback of Bad Apple on the real unit. **30 fps is a + best case, not a requirement**: it is what the budget allows when the blit path + and the wire cooperate. The requirement is that the picture keeps moving and + stays legible when the rate drops — no scheme is rejected merely for landing at + 20–25 fps. +2. Per-frame CPU work is O(tokens), not O(pixels); pixel data reaches memory by + **LANCE DMA**, never through the CPU. +3. A **bring-up path** (scheme A) that can be built and verified before any + compression machinery exists, and that remains as a full-screen fallback. +4. Small, testable pieces: frame format, host encoder, terminal decoder, transport; + everything that does not need the unit is tested under MAME. + +## 4. Non-goals + +* Audio (no audio path in this harness). +* Colour or grey (the display is 1 bpp). +* Any general codec (MJPEG/H.264) or on-terminal transcoding. +* A player UI. Changing the firmware, the ROM, or the boot path. + +## 5. Anti-goals + +* **Cost that is per output pixel on the CPU.** A byte store is ~1 µs in asm, so a + 163,840-byte frame is >100 ms of CPU before any arithmetic. +* Pixel data through the ROM's BSD sockets: `recvfrom_` copies every byte on the CPU + (~12 µs/byte), ~18 ms per 1500-byte packet. +* Bit-serial or range decoding in the terminal (badderapple's method): its MCU + decodes ~70 bytes/frame of symbols at 64×48; our CPU cannot afford per-bit work at + 1280×1024. +* Buying frame rate by shrinking the picture and calling it full screen. + +## 6. Constraints and budget + +| quantity | value | source | +|---|---|---| +| link | 10 Mbit half-duplex, 1.27 MB/s of frame bytes | measured | +| per-frame wire @30 fps | **42 KB** | derived | +| per-frame wire @24 fps | 53 KB | derived | +| display refresh | 72 Hz (13.9 ms) | raster registers | +| VRAM | 1280×1024 1 bpp, 256-byte pitch | driver | +| **full-screen repaint** | 81,920 destination words | derived | +| …via `PIXBLT L,XY` | **~51 ms** (unit: 0.62 µs/word) | measured | +| …via `FILL L` | **~33 ms** (unit: 0.40 µs/word) | measured | +| …via `PIXBLT B,XY` | **~26 ms** (MAME: 0.32 µs/word) | benchmark, **not unit-verified** | +| call overhead | ~2.6–3.3 µs per blit/fill call | benchmark | +| CPU asm word copy | 0.97 µs/word | benchmark | +| RAM we may claim | **not a hardware limit**: the firmware's `malloc` is callable and at boot has **3,099,414 free bytes** in 9 blocks, the largest **2,967,694 B** ending at byte `0x400000`. Our `.bss` (byte `0x380000`) sits *inside* that free block, so it is unallocated heap the OS may reuse | `dwt/notes/HEAP.md` | +| memory traps | a 64-deep ring at `0x4000` crashed the unit (§71.2); a hand-picked address between `MARK` and `.bss` hung it and cost a power cycle (§74.5) | measured | + +**Consequence: a full-screen repaint costs 26–51 ms no matter which scheme drives +it.** 30 fps is only reachable by (a) using `B,XY` (or `FILL L`) rather than `L,XY`, +and/or (b) repainting only what changed. + +## 7. Approaches considered + +**A. Row strips, hardware-stretched.** Host sends *N* unique rows of 1280 px; the +terminal blits each row with `SPTCH=0` and `DY=1024/N`. +* Wire: N=128 → 20 KB/frame (16 ms); N=256 → 41 KB (32 ms). +* Blit: **a full repaint either way — 26 ms (B,XY) or 51 ms (L,XY)**. +* Therefore: A with `L,XY` ≈ 19 fps; A with `B,XY` ≈ 30 fps *at the wire limit*. +* Simplest possible decoder; no dictionary; the natural bring-up. + +**B. RLE rows, decoded by `FILL L`.** Same transport, runs instead of rows; one +`FILL L` per run (0.32–0.40 µs/word + ~2.3 µs/call). Compresses flat anime frames +hard, so more unique rows fit the wire budget. Still a full-screen repaint of +≈33 ms unless runs are *delta* runs. + +**C. Tile dictionary, blitter-decoded.** A 256-entry dictionary of 32×32 1-bpp tiles +(128 B each, 32 KB), uploaded once; then tile **indices** (1,280 B/frame full, +≪1 KB for deltas), one `PIXBLT B,XY` per tile. +* Full repaint: 1,280 × (2.6 µs + 64 × 0.32 µs) ≈ **29 ms**. +* **Delta repaint (the normal case): far less — this is the only scheme whose cost + scales with change rather than with screen area.** +* Wire: 1.3 KB/frame, i.e. 30× under budget: the budget is available for fidelity. +* 16×16 tiles rejected: 5,120 calls × 8.4 µs = 43 ms even before deltas. + +**Rejected: MJPEG, raw full frames, ROM sockets** — see anti-goals (raw full frame is +129 ms on the wire, 7.8 fps). + +**Decision (human partner, this session): C, with A built first as bring-up and as +the full-screen fallback.** + +## 8. Design + +### 8.1 Transport — fixed slots, small ring, double buffered + +Frames are **not** large: A's strip frame is 14×1500 B at N=128, C's full frame is +1.3 KB, a dictionary upload is 32 KB. So the 128-descriptor ring proposed in rev 1 is +unnecessary: it would need 190 KB of buffers for a frame the design never sends, and +a 64-deep ring at `0x4000` already crashed the unit. + +**Call:** the big buffers are **allocated at startup with `ROM_MALLOC`**, not linked +into `.bss`: one `malloc` for the whole working set (frame slots + tile dictionary, +plus the 160 KB framebuffer only when scheme A is in use), and pointer stored in a +fixed marker word. Reasons: it removes the 256 KB linker cap, and — more importantly +— it **pins** the memory, where today's `.bss` is unallocated heap the OS is free to +hand out (that is a real risk over a 3.5-minute run, not a theoretical one). +`.bss` keeps only the small statics (stack, counters, the pointer itself). If the +allocation returns `0`, fall back to the static `.bss` buffers and report it; A's +framebuffer is the first thing to give up. + +The ring itself stays at **16 descriptors** (its buffers are small and static). + +Also for the transport: **two frame slots, alternating**, so the chip can DMA frame +*N+1* while we blit frame *N* — the overlap that makes `max(wire, blit)` the +per-frame cost instead of `wire + blit`; and descriptors are re-armed per slot at +frame start, so a frame never straddles the ring wrap and slot ownership is explicit. +* a frame header carries `(magic, kind, frame_no, payload_len, crc)`; the terminal + rejects a frame whose `payload_len` disagrees with the bytes received, so + **`payload_len` + descriptor count are authoritative — not `RMD3`**, whose + per-descriptor semantics on a chained frame differ between MAME (`RMD3` = whole + frame length, and intermediate descriptors get no `RMD3`) and any assumption we + might make about the chip; +* oversize payloads (dictionary upload, 32 KB = 22 packets) are split into + consecutive `kind = DICT_CHUNK` frames with an offset in the header. + +### 8.2 Scheme A — row strips (bring-up) + +* Host: scale to 1280 × N (N = 128 default, 256 if the wire allows), threshold to + 1 bpp, send row-major as a **packed B-format bitmap** (one 1280-px row = 80 words). +* Terminal: per source row, `PIXBLT B,XY` with `SADDR = row base`, `SPTCH = 0`, + `DYDX = (S, 1280)`, destination XY `(r*S, 0)`, `S = 1024/N`. +* Accepts ~30 fps only if the `B,XY` rate holds on the unit; with `L,XY` it is + ~19 fps. **M2 measures both and picks the helper.** + +### 8.3 Scheme C — tiles + +* **Dictionary**: 256 × 32×32 1-bpp (128 B each = 32 KB), built offline by k-means + over the video's tiles, uploaded in chunks, held in the startup allocation. +* **Frame**: header + 1,280 index bytes (40×32 grid, row major). `kind = TILES_FULL`. +* **Delta**: `kind = TILES_DELTA` + a changed-tile list (run-length list of tile + indices, 1–2 bytes per entry) and their new indices. Deltas are the normal case. +* Decoder: one cached loop; `PIXBLT B,XY` from `dict + index*128`, `DYDX=(32,32)`, + `SPTCH=32` (a 32-px row is **two** words), destination XY `(ty*32, tx*32)`. +* **Repair**: a `TILES_FULL` keyframe every 2 s (and on the host's first frame), so + a lost delta repairs itself within 60 frames instead of desynchronising forever. + +### 8.4 Host ↔ terminal back-channel + +The payload already has TX. It sends a small status frame every 64 frames: +frames received, `crc` failures, drops, `CSR0` error bits, and the worst decode time. +The host uses it to pace the stream and to force a keyframe when drops are seen. +**Call:** status every 64 frames (~2 s at 30 fps); no per-frame ack. + +### 8.5 Decoder loop and cost control + +Per tile: load index, form source address (`index << 7` **bits**), load `B0/B1/B7`, +execute. Kept small so the instruction cache (user's guide §5.4.7: 2.25 states per +instruction word with cache, 3 extra states per fetch without) holds the loop. The +blit dominates (23 µs vs ~3–5 µs of CPU), so the CPU still has room to service the +ring and the timer. + +### 8.6 Host side + +Python 3 driver in `harness/udpstream/`, next to `sendraw.py`/`vtcmd.py`: decode the +source video to 1-bpp frames (ffmpeg or PyAV), emit strips (A) or tile indices (C), +pace onto the NIC the unit is plugged into, log bytes/fps/drops. The dictionary +builder is offline and its output is a committed data file, so the demo is +reproducible. + +### 8.7 Synchronisation + +Display 72 Hz vs target 24–30 fps, so frames land mid-scan. **Call:** accept tearing +for v1 and land each frame's blits in one burst so the tear is a single band; revisit +with a start-of-frame delay if it looks bad on the unit. + +### 8.8 Failure behaviour + +* Short/torn/CRC-bad frame: discard, keep the screen, count it, report it. +* Ring overrun (`MISS` bit in CSR0): report; the host lowers its rate. +* Missing dictionary: fall back to scheme A automatically. + +### 8.9 Interference from the resident OS + +The OS is not stopped: its drivers noticed the LANCE takeover before (§64.3) and its +ISRs keep touching the chip. If it repaints or re-inits the port mid-demo the stream +breaks. **Call:** mask interrupts during a frame burst (`DINT` around the blit loop, +`EINT` after) and have the decoder detect a VRAM area it did not write (a cheap +sentinel tile) so an OS repaint is *detected* rather than silently mixed into the +picture. A 3.5-minute continuous run is an explicit M2/M4 acceptance item. + +## 9. Budgets (design targets) + +| scheme | wire/frame | blit/frame | CPU/frame | per-frame cost | rate | +|---|---|---|---|---|---| +| A, N=128, L,XY | 20 KB (16 ms) | 51 ms | ~1 ms | ~52 ms | ~19 fps | +| A, N=128, B,XY | 20 KB (16 ms) | 26 ms | ~1 ms | ~27 ms | ~36 fps | +| A, N=256, B,XY | 41 KB (32 ms) | 26 ms | ~2 ms | ~34 ms | ~29 fps | +| C, full repaint | 1.3 KB | 29 ms | ~6 ms | ~35 ms | ~28 fps | +| C, delta (typical) | <1 KB | <10 ms | <2 ms | <12 ms | wire-bound | +| raw full frame | 164 KB (129 ms) | 51 ms | — | 129 ms | 7.8 fps | + +Wire and blit overlap (slot double buffering), so the per-frame cost is +`max(wire, blit) + parse`, not the sum; the table is conservative. + +## 10. Milestones + +1. **M1 — transport in MAME**: two-slot ring, header validation, a synthetic frame + reaches the slot intact; wrong-length and wrong-CRC frames are rejected; the + startup `malloc` is called, its pointer logged, and the firmware's free-byte + counter (bit `0x1E8BA0`) is read before and after to prove the working set was + pinned. +2. **M2 — A on the unit**: row strips at N=128 and N=256, `L,XY` vs `B,XY` measured + on real silicon; a **3.5-minute continuous run** to establish whether the OS + interferes. This milestone reports the real rate; it does not have to reach 30 fps + to pass (see §3 goal 1). +3. **M3 — dictionary + tiles in MAME**: the same frame rendered by tiles and by + strips is pixel-identical; dictionary upload in chunks. +4. **M4 — Bad Apple end to end on the unit**: the best rate the budget allows, + 30 fps as the target, with counters (bytes, fps, drops, worst decode) reported. +5. **M5 — delta tiles**: only changed tiles; measure fps and worst-case frame. + +## 11. Verification + +* **MAME**: each path headless; the no-env regression baseline + (`PROG_FRAMES=200 → PC=3FDCE400 samples=198 (3 distinct PCs)`) must not move; new + Lua checks load the payload, feed one frame, dump VRAM and compare against the + host's source. +* **Unit**: `bench-sweep.sh` methodology (delivery %, fps, CSR0 errors) plus a + frame-level test that plays a known still and dumps VRAM. +* **Faithfulness**: no ROM patch; the payload arrives over the network exactly as + `lance_cmd` already does; any stand-in is named. + +## 12. Risks + +| risk | why it matters | mitigation | +|---|---|---| +| `B,XY` at 0.32 µs/word is MAME-only | A's 30 fps and C's 29 ms both rest on it | M2 measures `B,XY` on the unit before the tile path is built on it; fallback is `FILL L` (0.40) or deltas | +| OS interference during a 3.5-min run | a repaint or LANCE re-init breaks the stream | `DINT` around bursts, sentinel detection, long-run acceptance in M2 | +| buffer lifetime | today's `.bss` is unallocated heap: the allocator splits its big block from the start, so ~1.9 MB of OS allocation reaches the payload | one startup `malloc` pins the whole working set (and the free-byte counter is reported in the status frame); fall back to static buffers and say so if it returns `0` | +| allocation size | framing guess: A wants 160 KB framebuffer + 2 slots; C wants 32 KB dictionary + 2 small slots | allocate `max(A, C) + 32 KB` slack, check the returned pointer against `0`, and report the free-byte counter before/after | +| descriptor semantics on chained frames | MAME and the chip may disagree on `RMD3` | header `payload_len` + count is authoritative | +| 10 Mbit shared hub | throughput varies | strip/delta budgets leave 2–30× headroom | + +## 13. Left to the builder + +* Tile quantisation (k-means details) and whether the dictionary is 256 or 512 + entries; whether it is refreshed on scene change. +* Delta list encoding (run-length vs bitmask). +* Frame header bit layout and CRC polynomial. +* Default N for A, and whether M1 uses strips or a solid-block pattern. +* Whether M2's `DINT` masking is per burst or for the whole demo. + +## 14. Answered question (human partner, 2026-10-30) + +*"30 fps is just best case if there is enough budget. Real fps can drop."* + +So the acceptance bar is: **playback stays smooth-looking and legible at whatever +rate the budget yields**, with 30 fps as the target the design aims at, not a gate. +Scheme A is therefore allowed to land around 19–25 fps with `L,XY`; `B,XY` is then an +optimisation to measure, not a prerequisite.