oc-ls-stats/README.md

# oc-ls-stats

A TUI plugin for OpenCode that displays live prefill rate (PP) and generation rate (TG) from a llama.cpp-based llama-server.

There are many plugins to show tokens per second, but the reason for this one is that when running a local model I often found myself not knowing what the server was currently doing, which meant constantly switching over to a console where I could see its output. Especially during prefill/prompt processing which can take more than a minute with no feedback in the UI from other plugins I tried. 

Another motivation was to display the data of interest, but in a non-intrusive way with no UI elements jumping around. This plugin thus only shows a single numeric value, tokens per second, with an indicator as to whether the model is currently doing prompt processing or inference (token generation).

I'm using the llama-server /slots endpoint to get the needed data, which means if you connect opencode to another provider the plugin will just display "-" since it's not getting any data to display.

Note: As explained in further detail below the data that's displayed has to be deduced from llama-server's output. Sometimes the plugin might display PP for prompt processing while in reality the model is doing TG. If additional developments are made to the llama-server output data the plugin might be able to discern between them in a better way, but I think is as good as it gets for now.

I made this for my own usage. If you find it useful as well I'm just happy.

/Troed

_thanks to Tarquinen for their [oc-tps](https://github.com/Tarquinen/oc-tps), which I used as a base although I guess most of the code has now been replaced_

## Installation

```bash
opencode plugin @troed/oc-ls-stats@latest --global
```

Requires `opencode` `1.3.14` or newer.

TUI plugins are loaded from `~/.config/opencode/tui.json`, which after installation should look like this:

```json
{
  "plugin": ["@troed/oc-ls-stats@latest"]
}
```

## Display Format

The plugin renders a single line in the session prompt right slot:

```
1247 tps (PP)    -- during prefill
  25 tps (TG)    -- during generation
   - tps (TG)    -- idle
          n/a    -- unable to reach llama-server
```

## Detection and Calculation

### Server Discovery

By default the plugin discovers the llama-server URL by reading the OpenCode configuration via the TUI API and extracting `baseURL`/`base_url` fields from provider options whose name contains "llama" (but excludes providers whose name contains "ollama"). Falls back to `http://localhost:8080` if no matching provider is found.

If your llama-server provider is named differently, configure the endpoint explicitly via the plugin's `server` option in `tui.json`. An explicit server replaces discovery entirely:

```json
{
  "plugin": [
    ["@troed/oc-ls-stats@latest", { "server": "http://headless.local:8080" }]
  ]
}
```

A `baseURL`-style value ending in `/v1` is also accepted.

### Slot Polling

Every 500ms, the plugin polls `GET /slots?model=<model>` on each discovered server. The model parameter is required by the `/slots` endpoint. If the model cannot be discovered from the current route's session, the plugin skips polling.

When a server returns an error (e.g. HTTP 502) or is unreachable, the plugin backs off with an exponentially increasing delay per server (1s, 2s, 4s, ...) capped at 10s, with a small random jitter to stagger multiple servers. Backoff resets as soon as the server responds successfully again.

### State Classification

Each processing slot is classified as prefill (PP) or generation (TG) by comparing `n_decoded` and `n_prompt_tokens` against per-slot/per-task baselines. The plugin tracks two baseline maps keyed by slot ID:

- `prevNdBySlot[slotId]` — stores `{ prevNd, idTask, classification }`
- `prevPromptTokensBySlot[slotId]` — stores `{ tokens, idTask }`

The `idTask` field ensures baselines are invalidated when a slot starts a new task (task reuse).

Classification logic:

1. **First time or after task change/reset**: If we have no baseline for this slot+task, or `n_decoded` or `n_prompt_tokens` dropped below baseline (indicating a new request), use a heuristic: `n_decoded > 0` → TG, otherwise → PP.
2. **Compare against baseline**: If `n_decoded > baselineNd` → TG. If `n_prompt_tokens > baselineNt` → PP.
3. **Neither increased**: Persist the previous classification. This handles frequent polling where neither field increases on every poll cycle.

When a slot stops being processed, all tracked state for that slot is cleared.

### Prefill Rate (PP)

During prefill, the plugin calculates the instantaneous prompt processing rate:

1. On first detection of a prefill slot, the current `n_prompt_tokens` is captured as the baseline.
2. On subsequent polls, the delta in `n_prompt_tokens` is divided by the elapsed time in seconds.
3. The rate is updated only when both `dt > 0` and `delta > 0`.

The per-slot `n_prompt_tokens` field is used instead of the global `llamacpp:prompt_tokens_total` from `/metrics` because the global counter includes tokens from all slots, producing inflated values when multiple slots are active simultaneously.

### Generation Rate (TG)

During generation, the plugin calculates the instantaneous token generation rate:

1. On first detection of a generation slot, the current `n_decoded` is captured as the baseline.
2. On subsequent polls (same slot ID), the delta in `n_decoded` is divided by the elapsed time in seconds.
3. The rate is updated only when both `dt > 0` and `delta > 0`.

Slot reuse is tracked via `generateSlotId` to detect when a new generation starts on a different slot.

## Limitations

### Progress Percentage

The plugin cannot display prefill progress percentage. The `/slots` endpoint returns `n_prompt_tokens` (current prompt size) and `n_prompt_tokens_processed` (tokens processed), but not the final prompt size (`task->n_tokens()` from llama.cpp). Progress requires the ratio `n_prompt_tokens_processed / task->n_tokens()`.

### What Would Improve Compatibility

The following changes to the `/slots` endpoint would improve the plugin's functionality:

1. **Expose final prompt size**: Add `n_prompt_tokens_total` (or `n_tokens`) to the `/slots` output, representing `task->n_tokens()` from llama.cpp. This would enable prefill progress percentage calculation as `(n_prompt_tokens_processed / n_prompt_tokens_total) * 100`.

2. **Per-slot metrics endpoints**: Currently, the `/metrics` endpoint provides only global counters (`llamacpp:prompt_tokens_total`, `llamacpp:prompt_tokens_seconds`). Per-slot metrics would allow independent rate tracking without relying on slot state classification.

3. **Slot transition notifications**: The plugin polls every 500ms to detect state transitions. A WebSocket or SSE-based notification system for slot state changes would reduce polling overhead and improve detection latency.

4. **Stall detection**: When generation stalls (e.g., due to context window limits), `n_decoded` remains constant while `n_remain` stops decreasing. The plugin detects this via zero delta but has no way to distinguish a stall from normal generation. An explicit `stalled` flag in the slot output would help.

5. **Model-agnostic slot data**: The `/slots` endpoint requires a model parameter. Returning all slots without model filtering, or supporting `*` as a wildcard, would simplify discovery when multiple models are loaded.

## Source code repo

For known issues, posting new ones, forking or contributing:

https://git.sync.wtf/troed/oc-ls-stats

## Debug Logging

Debug logging is controlled by the `DEBUG_ENABLED` constant in `tui.tsx`. When enabled, full slot state data is written to `/tmp/oc-ls-stats-debug.log` on every poll.

## License

Creative Commons Zero (CC0 1.0 Universal)