oc-ls-stats/README.md

# oc-ls-stats

A TUI plugin for OpenCode that displays live prefill rate (PP) and generation rate (TG) from a llama.cpp-based llama-server.

There are many plugins to show tokens per second, but the reason for this one is that when running a local model I often found myself not knowing what the server was currently doing, which meant constantly switching over to a console where I could see its output. Especially during prefill/prompt processing which can take more than a minute with no feedback in the UI from other plugins I tried. 

Another motivation was to display the data of interest, but in a non-intrusive way with no UI elements jumping around. This plugin thus only shows a single numeric value, tokens per second, with an indicator as to whether the model is currently doing prompt processing or inference (token generation).

I'm using the llama-server /slots endpoint to get the needed data, which means if you connect opencode to another provider the plugin will just display "-" since it's not getting any data to display.

Note: As explained in further detail below the data that's displayed has to be deduced from llama-server's output. Sometimes the plugin might display PP for prompt processing while in reality the model is doing TG. If additional developments are made to the llama-server output data the plugin might be able to discern between them in a better way, but I think is as good as it gets for now.

I made this for my own usage. If you find it useful as well I'm just happy.

/Troed

_thanks to Tarquinen for their [oc-tps](https://github.com/Tarquinen/oc-tps), which I used as a base although I guess most of the code has now been replaced_

## Display Format

The plugin renders a single line in the session prompt right slot:

```
1247 tps (PP)    -- during prefill
  25 tps (TG)    -- during generation
- tps (TG)       -- idle
```

## Detection and Calculation

### Server Discovery

The plugin discovers the llama-server URL by reading the OpenCode configuration file (parsed as JSONC) and extracting `baseURL`/`base_url` fields from provider options that contain "localhost". Falls back to `http://localhost:8080`.

### Slot Polling

Every 500ms, the plugin polls `GET /slots?model=<model>` on each discovered server. The model parameter is required by the `/slots` endpoint. If the model cannot be discovered from the current route's session, the plugin skips polling.

### State Classification

Each slot is classified as prefill or generation based on the `n_decoded` counter in `next_token[0]`. The plugin tracks a per-slot baseline value:

1. When a slot first appears as processing, the current `n_decoded` is recorded as the baseline with `hasIncreased = false`.
2. If `n_decoded <= baseline` and `hasIncreased` is false, the slot is classified as prefilling.
3. If `n_decoded > baseline`, `hasIncreased` is set to true and the slot is classified as generating.

This approach handles the case where `n_decoded` drops when a new request starts on a reused slot, and prevents generation stalls (where `n_decoded` plateaus) from being misclassified as prefill.

When no slots are processing, all tracked state for those slots is cleared.

### Prefill Rate (PP)

During prefill, the plugin calculates the instantaneous prompt processing rate:

1. On first detection of a prefill slot, the current `n_prompt_tokens` is captured as the baseline.
2. On subsequent polls, the delta in `n_prompt_tokens` is divided by the elapsed time in seconds.
3. The rate is updated only when both `dt > 0` and `delta > 0`.

The per-slot `n_prompt_tokens` field is used instead of the global `llamacpp:prompt_tokens_total` from `/metrics` because the global counter includes tokens from all slots, producing inflated values when multiple slots are active simultaneously.

### Generation Rate (TG)

During generation, the plugin calculates the instantaneous token generation rate:

1. On first detection of a generation slot, the current `n_decoded` is captured as the baseline.
2. On subsequent polls (same slot ID), the delta in `n_decoded` is divided by the elapsed time in seconds.
3. The rate is updated only when both `dt > 0` and `delta > 0`.

Slot reuse is tracked via `generateSlotId` to detect when a new generation starts on a different slot.

## Limitations

### Progress Percentage

The plugin cannot display prefill progress percentage. The `/slots` endpoint returns `n_prompt_tokens` (current prompt size) and `n_prompt_tokens_processed` (tokens processed), but not the final prompt size (`task->n_tokens()` from llama.cpp). Progress requires the ratio `n_prompt_tokens_processed / task->n_tokens()`.

### What Would Improve Compatibility

The following changes to the `/slots` endpoint would improve the plugin's functionality:

1. **Expose final prompt size**: Add `n_prompt_tokens_total` (or `n_tokens`) to the `/slots` output, representing `task->n_tokens()` from llama.cpp. This would enable prefill progress percentage calculation as `(n_prompt_tokens_processed / n_prompt_tokens_total) * 100`.

2. **Per-slot metrics endpoints**: Currently, the `/metrics` endpoint provides only global counters (`llamacpp:prompt_tokens_total`, `llamacpp:prompt_tokens_seconds`). Per-slot metrics would allow independent rate tracking without relying on slot state classification.

3. **Slot transition notifications**: The plugin polls every 500ms to detect state transitions. A WebSocket or SSE-based notification system for slot state changes would reduce polling overhead and improve detection latency.

4. **Stall detection**: When generation stalls (e.g., due to context window limits), `n_decoded` remains constant while `n_remain` stops decreasing. The plugin detects this via zero delta but has no way to distinguish a stall from normal generation. An explicit `stalled` flag in the slot output would help.

5. **Model-agnostic slot data**: The `/slots` endpoint requires a model parameter. Returning all slots without model filtering, or supporting `*` as a wildcard, would simplify discovery when multiple models are loaded.

## Installation

```bash
opencode plugin @troed/oc-ls-stats@latest --global
```

Requires `opencode` `1.3.14` or newer.

TUI plugins are loaded from `~/.config/opencode/tui.json`:

```json
{"plugin": ["@troed/oc-ls-stats@latest"]}
```

## Debug Logging

Debug logging is controlled by the `DEBUG_ENABLED` constant in `tui.tsx`. When enabled, full slot state data is written to `/tmp/oc-ls-stats-debug.log` on every poll.

## License

Creative Commons Zero (CC0 1.0 Universal)