From cd4dbe2317feaded4869212a9d009f7df111b60f Mon Sep 17 00:00:00 2001 From: "opencode.energia.arti.ee" Date: Sat, 31 Oct 2026 01:28:05 +0000 Subject: [PATCH] =?UTF-8?q?plan:=20tile=20codec=20(scheme=20C)=20=E2=80=94?= =?UTF-8?q?=20dictionary,=20tile=20blit,=20deltas,=20bad=20apple?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../2026-10-30-vt1200-video-streaming.md | 253 ++++++++++++++++++ 1 file changed, 253 insertions(+) create mode 100644 docs/superpowers/plans/2026-10-30-vt1200-video-streaming.md diff --git a/docs/superpowers/plans/2026-10-30-vt1200-video-streaming.md b/docs/superpowers/plans/2026-10-30-vt1200-video-streaming.md new file mode 100644 index 0000000..25f0191 --- /dev/null +++ b/docs/superpowers/plans/2026-10-30-vt1200-video-streaming.md @@ -0,0 +1,253 @@ +# VT1200 video streaming — tile codec Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace row strips with scheme C — a 256-entry dictionary of 32×32 1-bpp tiles and a stream of tile indices plus deltas — so a full-screen frame costs ~1.3 KB on the wire and only changed tiles are redrawn. + +**Architecture:** The host quantises video frames into 32×32 tiles, uploads the dictionary once in chunks, then sends index bytes (full frames) and changed-tile lists (deltas). The terminal stores the dictionary in its malloc'd pool and paints each accepted tile with one `PIXBLT B,XY` straight into VRAM, reusing the transport, parser, slot ring and status frames that plan 1 built. + +**Tech Stack:** Python 3 host encoder; C (1987 TI TMS34010 SDK, K&R) + TMS34010 asm payload; MAME 0.289 + Lua; the DOSBox build wrapper from plan 1. + +**Spec:** `docs/superpowers/specs/2026-10-30-vt1200-video-streaming-design.md` (§7 C, §8.3, §8.4) + +**Depends on:** `docs/superpowers/plans/2026-10-30-vt1200-stream-transport-and-row-strips.md` (complete) + +## Global Constraints + +- ROM byte-identical (`md5 b46a3e442ccb86134d2356bd70d22fd2`); payloads arrive over the network or the debugger. +- Bit addresses everywhere in payload code; `malloc` takes bytes, returns a bit pointer. +- Payload C is K&R: no ANSI prototypes, no `//`, no declarations after statements. +- Frame CRC covers the **12 header bytes only** (ruling R14); body integrity is the Ethernet FCS. +- Tiles are **32×32 1 bpp**: 32 px = **two** 16-bit words per row, `SPTCH=32` bits, one `PIXBLT B,XY` per tile with `DYDX=(32,32)`. +- A full-screen repaint is 1,280 tiles; the ring caps one frame at ~144 rows of strip data, so tile frames must stay ≤ 24 KB (they are ~1.3 KB). +- No CPU copy of pixel data; the blitter moves every output pixel. +- The no-env MAME baseline must not move: `PROG_FRAMES=200` → `PC=3FDCE400 samples=198 (3 distinct PCs)`. +- Shared repo: stage only your own paths (`git add `), never `git add -A`. +- The unit run needs the physical terminal; it is human-assisted and recorded as such. + +## Review Focus + +- A tile index above 255 (or a dictionary that was never uploaded) must not paint garbage or read outside the pool. +- A partial delta must never leave a region showing stale content from two different frames without a keyframe to repair it. +- A keyframe must arrive after N frames or after a reported drop, whichever comes first, or a lost delta desynchronises the screen forever. +- The tile blit must not write outside its 32×32 destination (a wrong `DYDX` or `SPTCH` smears a row across the screen). +- The dictionary upload must be atomic from the decoder's point of view: no frame may be painted against a half-uploaded dictionary. + +--- + +## File structure + +| path | responsibility | +|---|---| +| `harness/udpstream/tilecost.py` | k-means tile dictionary builder + frame quantiser (host) | +| `harness/udpstream/tests/test_tilecost.py` | dictionary determinism, index-range, reconstruction error | +| `harness/udpstream/tiledict.py` | dictionary upload protocol (chunking, offsets) and delta frame encoding | +| `harness/udpstream/tests/test_tiledict.py` | chunk layout, delta list, keyframe cadence | +| `harness/udpstream/vram_tile.asm` | `PIXBLT B,XY` 32×32 helper, `SPTCH=32` | +| `harness/udpstream/vidstream.c` | dictionary storage, `KIND_DICT_CHUNK`/`TILES_FULL`/`TILES_DELTA` dispatch | +| `harness/mame/vidstream.lua` | `dict` and `tiles` modes | +| `harness/mame/tile_check.py` | compare a VRAM dump against a host-rendered reference frame | + +--- + +### Task 1: Dictionary builder and quantiser + +**Files:** +- Create: `harness/udpstream/tilecost.py`, `harness/udpstream/tests/test_tilecost.py` + +**Interfaces:** +- `build_dictionary(frames: list[bytes], k: int = 256, seed: int = 0) -> bytes` — returns `k` tiles of 128 bytes each (32×32 1 bpp, row-major, two bytes per row), ordered deterministically. +- `quantise(frame: bytes, dictionary: bytes) -> bytes` — returns 1,280 index bytes (40×32 grid, row major), choosing for each tile the dictionary entry with the smallest Hamming distance. + +- [ ] **Step 1: Write the failing test** + +```python +import random +from harness.udpstream.tilecost import build_dictionary, quantise + +def _frame(seed): + r = random.Random(seed) + return bytes(r.randrange(256) for _ in range(128 * 1280 // 8)) + +def test_dictionary_is_deterministic_and_indexable(): + frames = [_frame(i) for i in range(4)] + a = build_dictionary(frames, k=8, seed=7) + b = build_dictionary(frames, k=8, seed=7) + assert a == b and len(a) == 8 * 128 + +def test_quantise_returns_1280_indices_within_range(): + frames = [_frame(i) for i in range(4)] + d = build_dictionary(frames, k=8, seed=7) + idx = quantise(frames[0], d) + assert len(idx) == 1280 and max(idx) < 8 +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `python3 -m pytest harness/udpstream/tests/test_tilecost.py -v` +Expected: FAIL — no module `tilecost`. + +- [ ] **Step 3: Implement `tilecost.py`** + +Tiles are extracted from each frame in 32×32 blocks (40×32 per frame), k-means over Hamming distance with a fixed seed and a deterministic tie-break (lowest tile id), at most 8 iterations. + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `python3 -m pytest harness/udpstream/tests/test_tilecost.py -v` +Expected: PASS (2 passed) + +- [ ] **Step 5: Commit** + +```bash +git add harness/udpstream/tilecost.py harness/udpstream/tests/test_tilecost.py +git commit -m "feat(tiles): deterministic dictionary builder and quantiser" +``` + +--- + +### Task 2: Dictionary upload + +**Files:** +- Create: `harness/udpstream/tiledict.py`, `harness/udpstream/tests/test_tiledict.py` +- Modify: `harness/udpstream/frame.py` (add `KIND_DICT_CHUNK = 4`, `KIND_TILES_FULL = 2`, `KIND_TILES_DELTA = 3`) + +**Interfaces:** +- `dict_chunks(dictionary: bytes, offset_base: int = 0) -> list[bytes]` — each packet carries `offset u16` then ≤1450 bytes of dictionary. +- `payload_offset(packet: bytes) -> int`, `payload_body(packet: bytes) -> bytes`. + +- [ ] **Step 1: Write the failing test** + +```python +from harness.udpstream.tiledict import dict_chunks, payload_offset, payload_body + +def test_chunks_cover_the_dictionary_once_in_order(): + d = bytes(range(256)) * 128 # 32 KB + pk = dict_chunks(d) + assert b"".join(payload_body(p) for p in pk) == d + offs = [payload_offset(p) for p in pk] + assert offs == sorted(offs) and offs[0] == 0 + assert offs[-1] + len(payload_body(pk[-1])) == len(d) +``` + +- [ ] **Step 2: Run it to verify it fails**, then implement (Step 3), then verify it passes (Step 4). + +Commands: `python3 -m pytest harness/udpstream/tests/test_tiledict.py -v`. + +- [ ] **Step 5: Commit** + +```bash +git add harness/udpstream/tiledict.py harness/udpstream/tests/test_tiledict.py harness/udpstream/frame.py +git commit -m "feat(tiles): chunked dictionary upload protocol" +``` + +--- + +### Task 3: Dictionary storage and dispatch in the payload + +**Files:** +- Modify: `harness/udpstream/vidstream.c`, `harness/udpstream/frame_parse.h` (kinds), `harness/mame/vidstream.lua` (`dict` mode) + +**Interfaces:** +- Consumes: `fp_parse`, the slot ring, the pool (`POOL_PTR`), markers. +- Produces: `DICT_READY` (cell on the marker page), `DICT_BYTES` (bytes accepted), `DICT_BAD` (out-of-order or oversize chunks). + +- [ ] **Step 1: Write the failing MAME check** + +`VT1200_MODE=dict`: inject a dictionary in 23 chunks, then a `TILES_FULL` frame. Assert `DICT_READY == 1`, `DICT_BYTES == 32768`, a chunk with a wrong offset increments `DICT_BAD`, and a `TILES_FULL` frame that arrives before `DICT_READY` is rejected without painting. + +- [ ] **Step 2: Run to verify it fails** — `VT1200_MODE=dict harness/mame/run-vidstream.sh` (the kind is unknown, nothing is stored). + +- [ ] **Step 3: Implement.** Dictionary lives at `POOL_PTR + slot bytes`; chunks must arrive in ascending offset with no gaps; `TILES_FULL`/`TILES_DELTA` are rejected until `DICT_READY`. + +- [ ] **Step 4: Run to verify it passes.** + +- [ ] **Step 5: Commit** + +```bash +git add harness/udpstream/vidstream.c harness/udpstream/frame_parse.h harness/mame/vidstream.lua +git commit -m "feat(tiles): dictionary storage and kind dispatch in the payload" +``` + +--- + +### Task 4: Tile blit + +**Files:** +- Create: `harness/udpstream/vram_tile.asm` +- Modify: `harness/udpstream/vidstream.c`, `harness/udpstream/build-stream.sh`, `harness/mame/vidstream.lua` (`tiles` mode) +- Create: `harness/mame/tile_check.py` + +**Interfaces:** +- `tile_blit(tile_bitaddr, tx, ty)` — one `PIXBLT B,XY`, `SPTCH=32`, `DYDX=(32,32)`, destination XY `(ty*32, tx*32)`; reloads all implied operands per call (ruling R13 style: values published in `.bss`, wrapper is the only entry). + +- [ ] **Step 1: Write the failing test** + +`VT1200_MODE=tiles`: upload a dictionary whose entry *i* is a known pattern, send a `TILES_FULL` frame with index `i` in every position for four corners, dump VRAM, and assert each 32×32 block equals `dictionary[i]` bit-for-bit (`tile_check.py` renders the expected screen from the indices). + +- [ ] **Step 2: Run to verify it fails** (screen unchanged). + +- [ ] **Step 3: Implement** the asm helper and the 1,280-call loop; `dint` around the burst like the strip path. + +- [ ] **Step 4: Run to verify it passes** (each corner tile matches; a deliberately wrong `SPTCH` must fail the check — run that as a negative control). + +- [ ] **Step 5: Commit** + +```bash +git add harness/udpstream/vram_tile.asm harness/udpstream/vidstream.c harness/udpstream/build-stream.sh harness/mame/vidstream.lua harness/mame/tile_check.py +git commit -m "feat(tiles): 32x32 PIXBLT B,XY tile blit and full-frame paint" +``` + +--- + +### Task 5: Deltas and keyframes + +**Files:** +- Modify: `harness/udpstream/tiledict.py`, `harness/udpstream/vidstream.c`, `harness/mame/vidstream.lua` + +**Interfaces:** +- `delta_frame(prev: bytes, cur: bytes) -> bytes | None` — `None` when nothing changed; otherwise a run-length list of changed tile indices followed by their new values. +- Payload: `KIND_TILES_DELTA` paints only the listed tiles; a `TILES_FULL` keyframe is sent when `frame_no % KEYFRAME_EVERY == 0` (default 60, ~2 s at 30 fps) or when the status frame reports a drop. + +- [ ] **Step 1: Write the failing test** + +Host: `test_tiledict.py` — a delta of two changed tiles lists exactly those two indices and no others; identical frames produce `None`; a keyframe is due at frame 60 and 120. +MAME: `VT1200_MODE=delta` — paint a full frame, then a delta touching one tile, and assert only that 32×32 block changed (compare dumps before/after). + +- [ ] **Step 2: Run to verify it fails.** + +- [ ] **Step 3: Implement** encoder and decoder; deltas must not paint a tile index above 255 (guard) and must not run past the frame length. + +- [ ] **Step 4: Run to verify it passes.** + +- [ ] **Step 5: Commit** + +```bash +git add harness/udpstream/tiledict.py harness/udpstream/vidstream.c harness/mame/vidstream.lua harness/udpstream/tests/test_tiledict.py +git commit -m "feat(tiles): delta frames and periodic keyframes" +``` + +--- + +### Task 6: Bad Apple end to end + +**Files:** +- Create: `harness/udpstream/make_tiles.py` (video → dictionary + per-frame index/delta stream, cached on disk) +- Modify: `harness/udpstream/vidsend.py` (tile mode), `harness/mame/NETWORK_REPORT.md` (new section) + +- [ ] **Step 1: Write the failing test** — a 30-frame synthetic clip quantised and replayed in MAME must reproduce each frame's rendered bitmap within the dictionary's error bound (report both PSNR and the per-frame tile-change count). +- [ ] **Step 2: Run to verify it fails.** +- [ ] **Step 3: Implement** `make_tiles.py` and `vidsend.py --tiles`; print per-frame bytes, changed tiles, and the keyframe schedule. +- [ ] **Step 4: Run the MAME replay; record the numbers.** +- [ ] **Step 5: Unit run (human-assisted).** `sudo ./test-stream.sh --tiles` for 300 frames, then the 6,300-frame soak. Record fps, drops, `OS_TOUCH`, and whether the OS repainted. +- [ ] **Step 6: Commit** + +```bash +git add harness/udpstream/make_tiles.py harness/udpstream/vidsend.py harness/mame/NETWORK_REPORT.md +git commit -m "feat(tiles): bad apple tile stream end to end" +``` + +--- + +## Execution handoff + +Plan 1's unit measurements (fps for `L,XY` and `B,XY`, heap headroom, `OS_TOUCH`) size this plan's expectations; nothing here depends on them being good, only on them existing.