forked from arti/vt1200
plan: streaming transport + row strips (M1/M2), tasks 1-7
This commit is contained in:
opencode.energia.arti.ee committed
1 parent
eba6165a19
commit
f29ee5db2d
1 file changed
+359
@@ -0,0 +1,359 @@
|
||||
# VT1200 streaming — transport + row strips Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Stream full-screen 1-bpp frames to a real VT1200 over raw Ethernet and draw them with the graphics engine, using scheme A (row strips) as the transport's first real payload.
|
||||
|
||||
**Architecture:** The host encodes a frame into a small binary header plus packed 1280-px rows and sends it as ordinary Ethernet frames. The terminal's payload receives them into two alternating fixed slots through the LANCE RX ring (DMA, no CPU copy), validates the header, and blits each source row down `S` screen rows with `SPTCH=0`. Big buffers come from one startup `malloc` of the firmware heap so the OS cannot reclaim them.
|
||||
|
||||
**Tech Stack:** C (1987 TI TMS34010 SDK, K&R — no ANSI prototypes) + TMS34010 assembly, built through the existing DOSBox wrapper; Python 3 for the host encoder/sender; MAME 0.289 (debug build) + Lua for headless verification.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-10-30-vt1200-video-streaming-design.md`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- The ROM stays byte-identical: `md5 b46a3e442ccb86134d2356bd70d22fd2`. Payloads arrive over the network or the debugger; never patch `vt1200_v22.bin`.
|
||||
- All addresses in payload code are **bit** addresses (`byte = bit >> 3`). `malloc` takes **bytes** and returns a **bit** pointer.
|
||||
- Payload C is K&R: `int f(a) int a; { }`. No ANSI prototypes anywhere.
|
||||
- `.text` at bit `0x018C0000`; the build fails if `.text` runs past `MARK` (`0x01A00000`).
|
||||
- One `malloc` at startup, from the payload's own process, never from an ISR; never write below `ptr-8`.
|
||||
- Render path only: VRAM at bit `0x04000000`, 1280×1024, 1 bpp, 256-byte row pitch (`0x800` bits).
|
||||
- The no-env MAME regression must not move: `PROG_FRAMES=200` → `PC=3FDCE400 samples=198 (3 distinct PCs)`.
|
||||
- No CPU copy of pixel data: pixels travel LANCE → RAM → blitter.
|
||||
|
||||
## Review Focus
|
||||
|
||||
- A frame truncated at a slot boundary or lost mid-DMA must **not** paint partially; the previous screen stays.
|
||||
- A CRC/length mismatch must be detected *before* any blit, not after.
|
||||
- `malloc` returning 0 must degrade gracefully (static buffers, A's framebuffer dropped) rather than jump through a null pointer.
|
||||
- An `S` that does not divide 1024 exactly must be rejected by the host, not silently leave unpainted scanlines.
|
||||
- A ring overrun (`MISS` in CSR0) must be counted and reported, and playback must recover without a power cycle.
|
||||
|
||||
---
|
||||
|
||||
## File structure
|
||||
|
||||
| path | responsibility |
|
||||
|---|---|
|
||||
| `harness/udpstream/frame.py` | frame header pack/unpack, CRC16, frame kinds — the single definition both ends use |
|
||||
| `harness/udpstream/vidsrc.py` | source video → packed 1-bpp rows → strip frames |
|
||||
| `harness/udpstream/vidsend.py` | paced sender + status-frame reader |
|
||||
| `harness/udpstream/tests/test_frame.py` | header/CRC/round-trip tests |
|
||||
| `harness/udpstream/tests/test_vidsrc.py` | strip encoding, `S` divisibility, row packing tests |
|
||||
| `harness/udpstream/vidstream.c` | the payload: slots, ring, validation, blit loop, status |
|
||||
| `harness/udpstream/vram_b.asm` | `vram_blit_b_()` — `PIXBLT B,XY` with `SPTCH=0` |
|
||||
| `harness/udpstream/build-stream.sh` | DOSBox build for the above (mirrors `build-lance.sh`) |
|
||||
| `harness/mame/vidstream.lua` | load payload, feed frames, dump VRAM, assert counters |
|
||||
| `harness/mame/vidstream_check.py` | compare a MAME VRAM dump against the host frame |
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Frame format and CRC
|
||||
|
||||
**Files:**
|
||||
- Create: `harness/udpstream/frame.py`
|
||||
- Test: `harness/udpstream/tests/test_frame.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `HEADER_SIZE = 16`, `KIND_STRIP = 1`, `KIND_STATUS = 5`, `crc16(data: bytes) -> int`, `pack_header(kind: int, frame_no: int, payload_len: int, rows: int, stretch: int) -> bytes`, `unpack_header(buf: bytes) -> dict` returning keys `kind, frame_no, payload_len, rows, stretch, crc_ok`.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```python
|
||||
from harness.udpstream.frame import (HEADER_SIZE, KIND_STRIP, crc16,
|
||||
pack_header, unpack_header)
|
||||
|
||||
def test_header_is_16_bytes_and_round_trips():
|
||||
h = pack_header(KIND_STRIP, frame_no=7, payload_len=12800, rows=80, stretch=12)
|
||||
assert len(h) == HEADER_SIZE
|
||||
d = unpack_header(h)
|
||||
assert d["kind"] == KIND_STRIP and d["frame_no"] == 7
|
||||
assert d["payload_len"] == 12800 and d["rows"] == 80 and d["stretch"] == 12
|
||||
assert d["crc_ok"] is True
|
||||
|
||||
def test_crc_detects_a_single_bit_flip():
|
||||
b = bytearray(pack_header(KIND_STRIP, 1, 10, 8, 128))
|
||||
b[5] ^= 0x01
|
||||
assert unpack_header(bytes(b))["crc_ok"] is False
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `python3 -m pytest harness/udpstream/tests/test_frame.py -v`
|
||||
Expected: FAIL — `ModuleNotFoundError: harness.udpstream.frame`
|
||||
|
||||
- [ ] **Step 3: Implement `frame.py`**
|
||||
|
||||
Layout (little-endian): `'V','S'`, version `0x01`, `kind`, `frame_no` u16, `payload_len` u16, `rows` u8, `stretch` u8, 2 reserved, `crc16` u16 over header bytes 0..11 plus the payload. CRC is CCITT-FALSE: poly `0x1021`, init `0xFFFF`, no reflection.
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `python3 -m pytest harness/udpstream/tests/test_frame.py -v`
|
||||
Expected: PASS (2 passed)
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add harness/udpstream/frame.py harness/udpstream/tests/test_frame.py
|
||||
git commit -m "feat(stream): frame header and CRC16 shared by both ends"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Strip encoder
|
||||
|
||||
**Files:**
|
||||
- Create: `harness/udpstream/vidsrc.py`
|
||||
- Test: `harness/udpstream/tests/test_vidsrc.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `frame.pack_header`, `frame.KIND_STRIP`.
|
||||
- Produces: `class StripSource` with `__init__(self, rows: int, stretch: int)`, `encode(self, frame_no: int, rows_1bpp: bytes) -> list[bytes]` (header+payload split into ≤1450-byte chunks, one `bytes` per packet), and `class Rows` with `pack_row(pixels: list[int]) -> bytes` (1280 px → 160 bytes, MSB first).
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```python
|
||||
from harness.udpstream.vidsrc import Rows, StripSource
|
||||
|
||||
def test_pack_row_is_msb_first_and_160_bytes():
|
||||
px = [0] * 1280
|
||||
px[0] = 1
|
||||
px[7] = 1
|
||||
px[8] = 1
|
||||
b = Rows.pack_row(px)
|
||||
assert len(b) == 160
|
||||
assert b[0] == 0b11000001 and b[1] == 0b10000000
|
||||
|
||||
def test_strip_source_rejects_stretch_that_does_not_divide_1024():
|
||||
import pytest
|
||||
with pytest.raises(ValueError):
|
||||
StripSource(rows=100, stretch=7)
|
||||
|
||||
def test_strip_frame_round_trips_through_the_header():
|
||||
from harness.udpstream.frame import unpack_header
|
||||
src = StripSource(rows=8, stretch=128)
|
||||
rows = bytes([0xAA]) * (8 * 160)
|
||||
pkts = src.encode(frame_no=3, rows_1bpp=rows)
|
||||
head = unpack_header(pkts[0][:16])
|
||||
assert head["kind"] == 1 and head["rows"] == 8 and head["stretch"] == 128
|
||||
assert head["payload_len"] == 8 * 160
|
||||
assert b"".join(pkts)[16:] == rows
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `python3 -m pytest harness/udpstream/tests/test_vidsrc.py -v`
|
||||
Expected: FAIL — no module `vidsrc`
|
||||
|
||||
- [ ] **Step 3: Implement `Rows.pack_row` and `StripSource`**
|
||||
|
||||
`pack_row` sets bit `i` of the byte stream for pixel `i` (MSB first within each byte). `StripSource.encode` prepends `pack_header(KIND_STRIP, frame_no, len(rows), rows, stretch)` and chunks the result at 1450 bytes.
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `python3 -m pytest harness/udpstream/tests/test_vidsrc.py -v`
|
||||
Expected: PASS (3 passed)
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add harness/udpstream/vidsrc.py harness/udpstream/tests/test_vidsrc.py
|
||||
git commit -m "feat(stream): 1-bpp row packing and strip frame encoder"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Payload build + startup allocation
|
||||
|
||||
**Files:**
|
||||
- Create: `harness/udpstream/build-stream.sh`, `harness/udpstream/vidstream.c`
|
||||
- Modify: `harness/udpstream/romapi.h` (only if a declaration is missing)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `ROM_MALLOC` (bit `0x3F701230`), `malloc_()` thunk from `entry.asm`.
|
||||
- Produces: payload binary `harness/udpstream/build/vidstream.bin`; in-guest marker word `POOL_PTR` at bit `0x01810020` (bit pointer to the pool) and `POOL_FREE` at `0x01810024` (free bytes before/after, low/high).
|
||||
|
||||
- [ ] **Step 1: Install the build prerequisites**
|
||||
|
||||
Run: `sudo apt-get install -y --no-install-recommends dosbox mtools`
|
||||
Expected: `dosbox` and `mcopy` on PATH (neither is present on this machine today).
|
||||
|
||||
- [ ] **Step 2: Write `build-stream.sh`**
|
||||
|
||||
Copy `build-lance.sh`, change `SRC=${LANCE_SRC:-$DIR/lance.c}` to `SRC=${STREAM_SRC:-$DIR/vidstream.c}`, replace `VRAM.ASM` with `VRAM.ASM` + `VRAMB.ASM`, and keep the existing `MARK` overflow check. Name the outputs `build/vidstream.*`.
|
||||
|
||||
- [ ] **Step 3: Write the failing check**
|
||||
|
||||
Add to `harness/mame/vidstream.lua` a mode that loads the payload, enters it, waits 60 frames, then prints `POOL_PTR` and computes `POOL_FREE`. Assertion: `POOL_PTR` is non-zero and inside byte range `0x100000`–`0x400000`.
|
||||
|
||||
- [ ] **Step 4: Run it to verify it fails**
|
||||
|
||||
Run: `BENCH=0 harness/mame/run-vidstream.sh` (wrapper added in Step 5)
|
||||
Expected: `VID: pool pointer 00000000 -> FAIL` (no allocation yet)
|
||||
|
||||
- [ ] **Step 5: Implement the allocation in `vidstream.c`**
|
||||
|
||||
`_start` reads the free-byte counter at bit `0x1E8BA0`, calls `malloc_(POOL_BYTES)` with `POOL_BYTES = 233472` (228 KB: two 24 KB slots + 160 KB framebuffer + 32 KB dictionary + 4 KB slack), stores the returned bit pointer at `POOL_PTR`, reads the counter again into `POOL_FREE`. If the pointer is 0, fall back to a static `.bss` buffer and set `POOL_PTR` to it with the low bit of `POOL_FREE` set as a "static" flag.
|
||||
|
||||
- [ ] **Step 6: Run the check to verify it passes**
|
||||
|
||||
Run: `harness/mame/run-vidstream.sh`
|
||||
Expected: `VID: pool pointer in range` and `VID: heap free before/after differ by ~POOL_BYTES`.
|
||||
|
||||
- [ ] **Step 7: Prove the allocation-failure path**
|
||||
|
||||
Build with `STREAM_POOL_BYTES=0x7FFF000` (larger than free heap), run the check, and assert `POOL_PTR` equals the static buffer address and the low bit of `POOL_FREE` is set (the "static fallback" flag) — i.e. the payload keeps running instead of jumping through a null pointer.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add harness/udpstream/build-stream.sh harness/udpstream/vidstream.c harness/mame/vidstream.lua harness/mame/run-vidstream.sh
|
||||
git commit -m "feat(stream): payload skeleton with one startup malloc"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Two-slot ring with header validation
|
||||
|
||||
**Files:**
|
||||
- Modify: `harness/udpstream/vidstream.c`
|
||||
- Test: `harness/mame/vidstream.lua` (mode `rx`)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: in-guest counters `RX_OK` (`0x01810030`), `RX_BAD` (`0x01810034`), `RX_LAST_LEN` (`0x01810038`), `RX_SLOT` (`0x0181003C`).
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
In `harness/mame/vidstream.lua` mode `rx`: send one valid frame (80 rows, `S=12`) and one frame whose CRC byte is flipped, via `vidsend.py --fake`. Assert `RX_OK == 1` and `RX_BAD == 1`.
|
||||
|
||||
- [ ] **Step 2: Run to verify it fails**
|
||||
|
||||
Run: `VT1200_MODE=rx harness/mame/run-vidstream.sh`
|
||||
Expected: `RX_OK=0 RX_BAD=0 -> FAIL` (nothing parses yet)
|
||||
|
||||
- [ ] **Step 3: Implement slot receive + validation**
|
||||
|
||||
Ring of 16 descriptors, two slots of 24 KB at `POOL_PTR` and `POOL_PTR + 24576`; arm 16 descriptors per slot on entry to that slot; on completion compute `bytes = SUM(len)`; accept only if `bytes >= 16`, `magic == 'VS'`, `payload_len + 16 <= bytes`, and `crc16(header[0:12] + payload) == header_crc`. Reject → `RX_BAD++`, re-arm the same slot, no blit.
|
||||
|
||||
- [ ] **Step 4: Run to verify it passes**
|
||||
|
||||
Run: `VT1200_MODE=rx harness/mame/run-vidstream.sh`
|
||||
Expected: `RX_OK=1 RX_BAD=1 -> PASS`
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add harness/udpstream/vidstream.c harness/mame/vidstream.lua
|
||||
git commit -m "feat(stream): two-slot rx ring with header and crc validation"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Strip blit path
|
||||
|
||||
**Files:**
|
||||
- Create: `harness/udpstream/vram_b.asm`
|
||||
- Modify: `harness/udpstream/vidstream.c`, `harness/mame/vidstream.lua`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `vram_blit_b_(src_bitaddr, daddr_xy, dydx)` — `PIXBLT B,XY`, `SPTCH=0`, `PSIZE=1`, `CONVSP/CONVDP=0x14`, `CONTROL` raster op 0; reloads `B2`/`B7` per call.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
`harness/mame/vidstream.lua` mode `draw`: feed a frame whose 80 rows are a known pattern (row `r` = `0xF0F0` repeated, every 3rd row `0x0000`), dump VRAM rows 0..1023, and assert every screen row `r*12..r*12+11` equals source row `r` exactly (`harness/mame/vidstream_check.py`).
|
||||
|
||||
- [ ] **Step 2: Run to verify it fails**
|
||||
|
||||
Run: `VT1200_MODE=draw harness/mame/run-vidstream.sh`
|
||||
Expected: FAIL — VRAM still holds the previous frame
|
||||
|
||||
- [ ] **Step 3: Implement `vram_blit_b_` and the C loop**
|
||||
|
||||
For `r in 0..rows-1`: `vram_blit_b_(slot_payload + r*1280, (r*S) << 16 | 0, ((S) << 16) | 1280)`.
|
||||
Wrap the loop in `dint`/`eint` (spec §8.9) so the resident OS cannot repaint mid-burst, and keep a `sentinel` word at a VRAM corner outside the blitted area: if it changes, the OS painted over us — bump `OS_TOUCH` (`0x01810044`) and repaint the whole frame next time.
|
||||
|
||||
- [ ] **Step 4: Run to verify it passes**
|
||||
|
||||
Run: `VT1200_MODE=draw harness/mame/run-vidstream.sh`
|
||||
Expected: PASS, 1024/1024 rows match, and `OS_TOUCH == 0` for the run
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add harness/udpstream/vram_b.asm harness/udpstream/vidstream.c harness/mame/vidstream.lua harness/mame/vidstream_check.py
|
||||
git commit -m "feat(stream): row-strip blit with SPTCH=0 vertical stretch"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Sender, pacing and status
|
||||
|
||||
**Files:**
|
||||
- Create: `harness/udpstream/vidsend.py`
|
||||
- Modify: `harness/udpstream/vidstream.c` (status frame every 64 frames)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `vidsend.py --iface IFACE --rows 80 --stretch 12 --fps 30 --pattern bars` emitting frames and parsing `KIND_STATUS`; exit status 0 only if `drops == 0` and `crc_err == 0`.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
MAME with a tap interface: run `vidsend.py` for 200 frames at 30 fps; the terminal must emit ≥3 status frames, and the last must report `frames_rx == 200`, `drops == 0`.
|
||||
|
||||
- [ ] **Step 2: Run to verify it fails**
|
||||
|
||||
Run: `VT1200_MODE=stream harness/mame/run-vidstream.sh`
|
||||
Expected: FAIL — no status frames
|
||||
|
||||
- [ ] **Step 3: Implement the status frame and the sender's pacing**
|
||||
|
||||
Terminal: every 64 accepted frames send `KIND_STATUS` with `frames_rx u32, crc_err u16, drops u16, csr0 u16, heap_free_kb u16`. Sender: send frame *n* at `t0 + n/fps`, sleep to pace, print one CSV line per 30 frames.
|
||||
|
||||
- [ ] **Step 4: Run to verify it passes**
|
||||
|
||||
Run: `VT1200_MODE=stream harness/mame/run-vidstream.sh`
|
||||
Expected: PASS, `frames_rx=200 drops=0`
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add harness/udpstream/vidsend.py harness/udpstream/vidstream.c harness/mame/vidstream.lua
|
||||
git commit -m "feat(stream): paced sender and terminal status frames"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Unit run and measurement
|
||||
|
||||
**Files:**
|
||||
- Create: `harness/udpstream/test-stream.sh`
|
||||
- Modify: `harness/mame/NETWORK_REPORT.md` (new section: scheme A on the unit)
|
||||
|
||||
- [ ] **Step 1: Write `test-stream.sh`**
|
||||
|
||||
Same shape as `bench-sweep.sh`: `sudo ./test-stream.sh <iface> <mac> [rows] [stretch]`, driving `vidsend.py` at 30 fps for 300 frames and recording fps, delivery %, drops, CSR0 error bits.
|
||||
|
||||
- [ ] **Step 2: Run it on the unit**
|
||||
|
||||
Run: `sudo ./test-stream.sh enp197s0f3u1 08:00:2b:16:4b:5a 80 12`
|
||||
Expected: a CSV row per 30 frames, no `MISS`, and a measurable fps.
|
||||
|
||||
- [ ] **Step 3: Measure `B,XY` versus `L,XY`**
|
||||
|
||||
Run the same test with the helper forced to `vram_blit_` (L,XY) via `--blit=l`, and record both fps values. This is the number the tile plan needs.
|
||||
|
||||
- [ ] **Step 4: 3.5-minute soak**
|
||||
|
||||
Run 6,300 frames continuously; assert the terminal still answers and the picture is still updating (status frames keep arriving).
|
||||
|
||||
- [ ] **Step 5: Commit the numbers**
|
||||
|
||||
```bash
|
||||
git add harness/udpstream/test-stream.sh harness/mame/NETWORK_REPORT.md
|
||||
git commit -m "test(stream): scheme A measured on the unit (B,XY vs L,XY, 3.5 min soak)"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Execution handoff
|
||||
|
||||
Plan 2 (tile codec) is written only after Task 7 reports the real `B,XY` rate, the sustained fps and the heap headroom.
|
||||
Reference in new issue
Block a user