forked from arti/vt1200
spec: VT1200 ethernet video streaming (transport + tile codec), rev 3
This commit is contained in:
opencode.energia.arti.ee committed
1 parent
1d8656ca10
commit
eba6165a19
1 file changed
+305
@@ -0,0 +1,305 @@
|
||||
# VT1200 Ethernet video streaming — design
|
||||
|
||||
Date: 2026-10-30
|
||||
Status: **rev 3** — corrected after a builder review; the open question is answered
|
||||
(§14). Awaiting approval.
|
||||
Author: (session). Decisions marked "**call:**" are the builder's proposals, not
|
||||
something the human partner has agreed to.
|
||||
|
||||
---
|
||||
|
||||
## 1. Intent
|
||||
|
||||
Play **Bad Apple!!** on a real VT1200, **full screen (1280×1024, 1 bpp)**, streamed
|
||||
over the terminal's Ethernet port at a rate that reads as motion, using only code we
|
||||
ship at runtime — the ROM stays byte-identical
|
||||
(`md5 b46a3e442ccb86134d2356bd70d22fd2`).
|
||||
|
||||
"Optimized" means: per frame the VT1200 CPU does a handful of *token* operations and
|
||||
the **graphics engine moves every output pixel**. No per-pixel or per-bit CPU work
|
||||
anywhere.
|
||||
|
||||
Secondary outcome: the transport and decoder are a **reusable streaming pipeline**
|
||||
(any 1-bpp animation, not just Bad Apple), with Bad Apple as the acceptance test.
|
||||
|
||||
## 2. What exists today (grounding, all measured on the unit)
|
||||
|
||||
* **Payload toolchain**: `harness/udpstream/` builds a RAM payload with the 1987 TI
|
||||
SDK (`.text` at bit `0x018C0000`, `.bss` at `0x01C00000`, 256 KB — i.e. bytes
|
||||
`0x380000`–`0x3C0000`) that talks raw Ethernet through its own LANCE driver.
|
||||
* **Transport**: raw LANCE RX ring, **16 descriptors × 1526 B**, buffers at bit
|
||||
`B(0x4000)`. Sustains **856–858 fps** of 1514-byte frames = **1.27 MB/s**, 99.9 %
|
||||
delivery, on a **10 Mbit half-duplex hub** (`NETWORK_REPORT` §72).
|
||||
* **Graphics engine**: `FILL L` **0.40 µs per destination word**, `PIXBLT L,XY`
|
||||
**0.62 µs per destination word** on the unit; `vram_fill_()`/`vram_blit_()` exist
|
||||
(`harness/udpstream/vram.asm`), including the `SPTCH=0` row-replication trick.
|
||||
* **Firmware precedent** (§74.3): the ROM blits text with `PIXBLT B,XY` (packed
|
||||
1-bpp source → XY destination) and rectangles with `PIXBLT L,L`, both chunked at
|
||||
`0x7FC0` pixels.
|
||||
* **CPU ceiling**: a C byte-store loop costs **12.2–13.9 µs per byte**; a hand
|
||||
written asm word loop measures **0.97 µs/word** (MAME). The C number is codegen,
|
||||
but even asm is 1.5–3× worse per word than the blitter and shares its bus.
|
||||
* **Display**: 1280×1024 mono at **72 Hz** (13.9 ms), 256-byte VRAM row pitch, VRAM
|
||||
at bit `0x04000000`.
|
||||
* **Heap**: the firmware's allocator is reachable from a payload (`ROM_MALLOC`,
|
||||
bit `0x3F701230`, "verified live"): bit-addressed first-fit, returns a **bit**
|
||||
pointer (`0` on failure), block header is 8 bytes — never write below `ptr-8`.
|
||||
`dwt/notes/HEAP.md` gives the safe pattern: **one allocation at startup**, from
|
||||
the payload's own process, never from an ISR, keep the pointer in a callee-saved
|
||||
register or a fixed marker.
|
||||
* **Cost is per destination word.** A blit of a 1280-px strip replicated down *S*
|
||||
rows still writes 1280×*S* destination pixels. The `SPTCH=0` trick saves **wire
|
||||
bytes, not blitter time**.
|
||||
|
||||
## 3. Goals
|
||||
|
||||
1. Full-screen 1280×1024 1-bpp playback of Bad Apple on the real unit. **30 fps is a
|
||||
best case, not a requirement**: it is what the budget allows when the blit path
|
||||
and the wire cooperate. The requirement is that the picture keeps moving and
|
||||
stays legible when the rate drops — no scheme is rejected merely for landing at
|
||||
20–25 fps.
|
||||
2. Per-frame CPU work is O(tokens), not O(pixels); pixel data reaches memory by
|
||||
**LANCE DMA**, never through the CPU.
|
||||
3. A **bring-up path** (scheme A) that can be built and verified before any
|
||||
compression machinery exists, and that remains as a full-screen fallback.
|
||||
4. Small, testable pieces: frame format, host encoder, terminal decoder, transport;
|
||||
everything that does not need the unit is tested under MAME.
|
||||
|
||||
## 4. Non-goals
|
||||
|
||||
* Audio (no audio path in this harness).
|
||||
* Colour or grey (the display is 1 bpp).
|
||||
* Any general codec (MJPEG/H.264) or on-terminal transcoding.
|
||||
* A player UI. Changing the firmware, the ROM, or the boot path.
|
||||
|
||||
## 5. Anti-goals
|
||||
|
||||
* **Cost that is per output pixel on the CPU.** A byte store is ~1 µs in asm, so a
|
||||
163,840-byte frame is >100 ms of CPU before any arithmetic.
|
||||
* Pixel data through the ROM's BSD sockets: `recvfrom_` copies every byte on the CPU
|
||||
(~12 µs/byte), ~18 ms per 1500-byte packet.
|
||||
* Bit-serial or range decoding in the terminal (badderapple's method): its MCU
|
||||
decodes ~70 bytes/frame of symbols at 64×48; our CPU cannot afford per-bit work at
|
||||
1280×1024.
|
||||
* Buying frame rate by shrinking the picture and calling it full screen.
|
||||
|
||||
## 6. Constraints and budget
|
||||
|
||||
| quantity | value | source |
|
||||
|---|---|---|
|
||||
| link | 10 Mbit half-duplex, 1.27 MB/s of frame bytes | measured |
|
||||
| per-frame wire @30 fps | **42 KB** | derived |
|
||||
| per-frame wire @24 fps | 53 KB | derived |
|
||||
| display refresh | 72 Hz (13.9 ms) | raster registers |
|
||||
| VRAM | 1280×1024 1 bpp, 256-byte pitch | driver |
|
||||
| **full-screen repaint** | 81,920 destination words | derived |
|
||||
| …via `PIXBLT L,XY` | **~51 ms** (unit: 0.62 µs/word) | measured |
|
||||
| …via `FILL L` | **~33 ms** (unit: 0.40 µs/word) | measured |
|
||||
| …via `PIXBLT B,XY` | **~26 ms** (MAME: 0.32 µs/word) | benchmark, **not unit-verified** |
|
||||
| call overhead | ~2.6–3.3 µs per blit/fill call | benchmark |
|
||||
| CPU asm word copy | 0.97 µs/word | benchmark |
|
||||
| RAM we may claim | **not a hardware limit**: the firmware's `malloc` is callable and at boot has **3,099,414 free bytes** in 9 blocks, the largest **2,967,694 B** ending at byte `0x400000`. Our `.bss` (byte `0x380000`) sits *inside* that free block, so it is unallocated heap the OS may reuse | `dwt/notes/HEAP.md` |
|
||||
| memory traps | a 64-deep ring at `0x4000` crashed the unit (§71.2); a hand-picked address between `MARK` and `.bss` hung it and cost a power cycle (§74.5) | measured |
|
||||
|
||||
**Consequence: a full-screen repaint costs 26–51 ms no matter which scheme drives
|
||||
it.** 30 fps is only reachable by (a) using `B,XY` (or `FILL L`) rather than `L,XY`,
|
||||
and/or (b) repainting only what changed.
|
||||
|
||||
## 7. Approaches considered
|
||||
|
||||
**A. Row strips, hardware-stretched.** Host sends *N* unique rows of 1280 px; the
|
||||
terminal blits each row with `SPTCH=0` and `DY=1024/N`.
|
||||
* Wire: N=128 → 20 KB/frame (16 ms); N=256 → 41 KB (32 ms).
|
||||
* Blit: **a full repaint either way — 26 ms (B,XY) or 51 ms (L,XY)**.
|
||||
* Therefore: A with `L,XY` ≈ 19 fps; A with `B,XY` ≈ 30 fps *at the wire limit*.
|
||||
* Simplest possible decoder; no dictionary; the natural bring-up.
|
||||
|
||||
**B. RLE rows, decoded by `FILL L`.** Same transport, runs instead of rows; one
|
||||
`FILL L` per run (0.32–0.40 µs/word + ~2.3 µs/call). Compresses flat anime frames
|
||||
hard, so more unique rows fit the wire budget. Still a full-screen repaint of
|
||||
≈33 ms unless runs are *delta* runs.
|
||||
|
||||
**C. Tile dictionary, blitter-decoded.** A 256-entry dictionary of 32×32 1-bpp tiles
|
||||
(128 B each, 32 KB), uploaded once; then tile **indices** (1,280 B/frame full,
|
||||
≪1 KB for deltas), one `PIXBLT B,XY` per tile.
|
||||
* Full repaint: 1,280 × (2.6 µs + 64 × 0.32 µs) ≈ **29 ms**.
|
||||
* **Delta repaint (the normal case): far less — this is the only scheme whose cost
|
||||
scales with change rather than with screen area.**
|
||||
* Wire: 1.3 KB/frame, i.e. 30× under budget: the budget is available for fidelity.
|
||||
* 16×16 tiles rejected: 5,120 calls × 8.4 µs = 43 ms even before deltas.
|
||||
|
||||
**Rejected: MJPEG, raw full frames, ROM sockets** — see anti-goals (raw full frame is
|
||||
129 ms on the wire, 7.8 fps).
|
||||
|
||||
**Decision (human partner, this session): C, with A built first as bring-up and as
|
||||
the full-screen fallback.**
|
||||
|
||||
## 8. Design
|
||||
|
||||
### 8.1 Transport — fixed slots, small ring, double buffered
|
||||
|
||||
Frames are **not** large: A's strip frame is 14×1500 B at N=128, C's full frame is
|
||||
1.3 KB, a dictionary upload is 32 KB. So the 128-descriptor ring proposed in rev 1 is
|
||||
unnecessary: it would need 190 KB of buffers for a frame the design never sends, and
|
||||
a 64-deep ring at `0x4000` already crashed the unit.
|
||||
|
||||
**Call:** the big buffers are **allocated at startup with `ROM_MALLOC`**, not linked
|
||||
into `.bss`: one `malloc` for the whole working set (frame slots + tile dictionary,
|
||||
plus the 160 KB framebuffer only when scheme A is in use), and pointer stored in a
|
||||
fixed marker word. Reasons: it removes the 256 KB linker cap, and — more importantly
|
||||
— it **pins** the memory, where today's `.bss` is unallocated heap the OS is free to
|
||||
hand out (that is a real risk over a 3.5-minute run, not a theoretical one).
|
||||
`.bss` keeps only the small statics (stack, counters, the pointer itself). If the
|
||||
allocation returns `0`, fall back to the static `.bss` buffers and report it; A's
|
||||
framebuffer is the first thing to give up.
|
||||
|
||||
The ring itself stays at **16 descriptors** (its buffers are small and static).
|
||||
|
||||
Also for the transport: **two frame slots, alternating**, so the chip can DMA frame
|
||||
*N+1* while we blit frame *N* — the overlap that makes `max(wire, blit)` the
|
||||
per-frame cost instead of `wire + blit`; and descriptors are re-armed per slot at
|
||||
frame start, so a frame never straddles the ring wrap and slot ownership is explicit.
|
||||
* a frame header carries `(magic, kind, frame_no, payload_len, crc)`; the terminal
|
||||
rejects a frame whose `payload_len` disagrees with the bytes received, so
|
||||
**`payload_len` + descriptor count are authoritative — not `RMD3`**, whose
|
||||
per-descriptor semantics on a chained frame differ between MAME (`RMD3` = whole
|
||||
frame length, and intermediate descriptors get no `RMD3`) and any assumption we
|
||||
might make about the chip;
|
||||
* oversize payloads (dictionary upload, 32 KB = 22 packets) are split into
|
||||
consecutive `kind = DICT_CHUNK` frames with an offset in the header.
|
||||
|
||||
### 8.2 Scheme A — row strips (bring-up)
|
||||
|
||||
* Host: scale to 1280 × N (N = 128 default, 256 if the wire allows), threshold to
|
||||
1 bpp, send row-major as a **packed B-format bitmap** (one 1280-px row = 80 words).
|
||||
* Terminal: per source row, `PIXBLT B,XY` with `SADDR = row base`, `SPTCH = 0`,
|
||||
`DYDX = (S, 1280)`, destination XY `(r*S, 0)`, `S = 1024/N`.
|
||||
* Accepts ~30 fps only if the `B,XY` rate holds on the unit; with `L,XY` it is
|
||||
~19 fps. **M2 measures both and picks the helper.**
|
||||
|
||||
### 8.3 Scheme C — tiles
|
||||
|
||||
* **Dictionary**: 256 × 32×32 1-bpp (128 B each = 32 KB), built offline by k-means
|
||||
over the video's tiles, uploaded in chunks, held in the startup allocation.
|
||||
* **Frame**: header + 1,280 index bytes (40×32 grid, row major). `kind = TILES_FULL`.
|
||||
* **Delta**: `kind = TILES_DELTA` + a changed-tile list (run-length list of tile
|
||||
indices, 1–2 bytes per entry) and their new indices. Deltas are the normal case.
|
||||
* Decoder: one cached loop; `PIXBLT B,XY` from `dict + index*128`, `DYDX=(32,32)`,
|
||||
`SPTCH=32` (a 32-px row is **two** words), destination XY `(ty*32, tx*32)`.
|
||||
* **Repair**: a `TILES_FULL` keyframe every 2 s (and on the host's first frame), so
|
||||
a lost delta repairs itself within 60 frames instead of desynchronising forever.
|
||||
|
||||
### 8.4 Host ↔ terminal back-channel
|
||||
|
||||
The payload already has TX. It sends a small status frame every 64 frames:
|
||||
frames received, `crc` failures, drops, `CSR0` error bits, and the worst decode time.
|
||||
The host uses it to pace the stream and to force a keyframe when drops are seen.
|
||||
**Call:** status every 64 frames (~2 s at 30 fps); no per-frame ack.
|
||||
|
||||
### 8.5 Decoder loop and cost control
|
||||
|
||||
Per tile: load index, form source address (`index << 7` **bits**), load `B0/B1/B7`,
|
||||
execute. Kept small so the instruction cache (user's guide §5.4.7: 2.25 states per
|
||||
instruction word with cache, 3 extra states per fetch without) holds the loop. The
|
||||
blit dominates (23 µs vs ~3–5 µs of CPU), so the CPU still has room to service the
|
||||
ring and the timer.
|
||||
|
||||
### 8.6 Host side
|
||||
|
||||
Python 3 driver in `harness/udpstream/`, next to `sendraw.py`/`vtcmd.py`: decode the
|
||||
source video to 1-bpp frames (ffmpeg or PyAV), emit strips (A) or tile indices (C),
|
||||
pace onto the NIC the unit is plugged into, log bytes/fps/drops. The dictionary
|
||||
builder is offline and its output is a committed data file, so the demo is
|
||||
reproducible.
|
||||
|
||||
### 8.7 Synchronisation
|
||||
|
||||
Display 72 Hz vs target 24–30 fps, so frames land mid-scan. **Call:** accept tearing
|
||||
for v1 and land each frame's blits in one burst so the tear is a single band; revisit
|
||||
with a start-of-frame delay if it looks bad on the unit.
|
||||
|
||||
### 8.8 Failure behaviour
|
||||
|
||||
* Short/torn/CRC-bad frame: discard, keep the screen, count it, report it.
|
||||
* Ring overrun (`MISS` bit in CSR0): report; the host lowers its rate.
|
||||
* Missing dictionary: fall back to scheme A automatically.
|
||||
|
||||
### 8.9 Interference from the resident OS
|
||||
|
||||
The OS is not stopped: its drivers noticed the LANCE takeover before (§64.3) and its
|
||||
ISRs keep touching the chip. If it repaints or re-inits the port mid-demo the stream
|
||||
breaks. **Call:** mask interrupts during a frame burst (`DINT` around the blit loop,
|
||||
`EINT` after) and have the decoder detect a VRAM area it did not write (a cheap
|
||||
sentinel tile) so an OS repaint is *detected* rather than silently mixed into the
|
||||
picture. A 3.5-minute continuous run is an explicit M2/M4 acceptance item.
|
||||
|
||||
## 9. Budgets (design targets)
|
||||
|
||||
| scheme | wire/frame | blit/frame | CPU/frame | per-frame cost | rate |
|
||||
|---|---|---|---|---|---|
|
||||
| A, N=128, L,XY | 20 KB (16 ms) | 51 ms | ~1 ms | ~52 ms | ~19 fps |
|
||||
| A, N=128, B,XY | 20 KB (16 ms) | 26 ms | ~1 ms | ~27 ms | ~36 fps |
|
||||
| A, N=256, B,XY | 41 KB (32 ms) | 26 ms | ~2 ms | ~34 ms | ~29 fps |
|
||||
| C, full repaint | 1.3 KB | 29 ms | ~6 ms | ~35 ms | ~28 fps |
|
||||
| C, delta (typical) | <1 KB | <10 ms | <2 ms | <12 ms | wire-bound |
|
||||
| raw full frame | 164 KB (129 ms) | 51 ms | — | 129 ms | 7.8 fps |
|
||||
|
||||
Wire and blit overlap (slot double buffering), so the per-frame cost is
|
||||
`max(wire, blit) + parse`, not the sum; the table is conservative.
|
||||
|
||||
## 10. Milestones
|
||||
|
||||
1. **M1 — transport in MAME**: two-slot ring, header validation, a synthetic frame
|
||||
reaches the slot intact; wrong-length and wrong-CRC frames are rejected; the
|
||||
startup `malloc` is called, its pointer logged, and the firmware's free-byte
|
||||
counter (bit `0x1E8BA0`) is read before and after to prove the working set was
|
||||
pinned.
|
||||
2. **M2 — A on the unit**: row strips at N=128 and N=256, `L,XY` vs `B,XY` measured
|
||||
on real silicon; a **3.5-minute continuous run** to establish whether the OS
|
||||
interferes. This milestone reports the real rate; it does not have to reach 30 fps
|
||||
to pass (see §3 goal 1).
|
||||
3. **M3 — dictionary + tiles in MAME**: the same frame rendered by tiles and by
|
||||
strips is pixel-identical; dictionary upload in chunks.
|
||||
4. **M4 — Bad Apple end to end on the unit**: the best rate the budget allows,
|
||||
30 fps as the target, with counters (bytes, fps, drops, worst decode) reported.
|
||||
5. **M5 — delta tiles**: only changed tiles; measure fps and worst-case frame.
|
||||
|
||||
## 11. Verification
|
||||
|
||||
* **MAME**: each path headless; the no-env regression baseline
|
||||
(`PROG_FRAMES=200 → PC=3FDCE400 samples=198 (3 distinct PCs)`) must not move; new
|
||||
Lua checks load the payload, feed one frame, dump VRAM and compare against the
|
||||
host's source.
|
||||
* **Unit**: `bench-sweep.sh` methodology (delivery %, fps, CSR0 errors) plus a
|
||||
frame-level test that plays a known still and dumps VRAM.
|
||||
* **Faithfulness**: no ROM patch; the payload arrives over the network exactly as
|
||||
`lance_cmd` already does; any stand-in is named.
|
||||
|
||||
## 12. Risks
|
||||
|
||||
| risk | why it matters | mitigation |
|
||||
|---|---|---|
|
||||
| `B,XY` at 0.32 µs/word is MAME-only | A's 30 fps and C's 29 ms both rest on it | M2 measures `B,XY` on the unit before the tile path is built on it; fallback is `FILL L` (0.40) or deltas |
|
||||
| OS interference during a 3.5-min run | a repaint or LANCE re-init breaks the stream | `DINT` around bursts, sentinel detection, long-run acceptance in M2 |
|
||||
| buffer lifetime | today's `.bss` is unallocated heap: the allocator splits its big block from the start, so ~1.9 MB of OS allocation reaches the payload | one startup `malloc` pins the whole working set (and the free-byte counter is reported in the status frame); fall back to static buffers and say so if it returns `0` |
|
||||
| allocation size | framing guess: A wants 160 KB framebuffer + 2 slots; C wants 32 KB dictionary + 2 small slots | allocate `max(A, C) + 32 KB` slack, check the returned pointer against `0`, and report the free-byte counter before/after |
|
||||
| descriptor semantics on chained frames | MAME and the chip may disagree on `RMD3` | header `payload_len` + count is authoritative |
|
||||
| 10 Mbit shared hub | throughput varies | strip/delta budgets leave 2–30× headroom |
|
||||
|
||||
## 13. Left to the builder
|
||||
|
||||
* Tile quantisation (k-means details) and whether the dictionary is 256 or 512
|
||||
entries; whether it is refreshed on scene change.
|
||||
* Delta list encoding (run-length vs bitmask).
|
||||
* Frame header bit layout and CRC polynomial.
|
||||
* Default N for A, and whether M1 uses strips or a solid-block pattern.
|
||||
* Whether M2's `DINT` masking is per burst or for the whole demo.
|
||||
|
||||
## 14. Answered question (human partner, 2026-10-30)
|
||||
|
||||
*"30 fps is just best case if there is enough budget. Real fps can drop."*
|
||||
|
||||
So the acceptance bar is: **playback stays smooth-looking and legible at whatever
|
||||
rate the budget yields**, with 30 fps as the target the design aims at, not a gate.
|
||||
Scheme A is therefore allowed to land around 19–25 fps with `L,XY`; `B,XY` is then an
|
||||
optimisation to measure, not a prerequisite.
|
||||
Reference in new issue
Block a user