oc-ls-stats/README.md

# oc-ls-stats

A TUI plugin for OpenCode that displays live prefill rate (PP) and generation rate (TG) from a llama.cpp-based llama-server.

There are many plugins to show tokens per second, but the reason for this one is that when running a local model I often found myself not knowing what the server was currently doing, which meant constantly switching over to a console where I could see its output. Especially during prefill/prompt processing which can take more than a minute with no feedback in the UI from other plugins I tried. 

Another motivation was to display the data of interest, but in a non-intrusive way with no UI elements jumping around. This plugin thus only shows a single numeric value, tokens per second, with an indicator as to whether the model is currently doing prompt processing or inference (token generation).

I'm using the llama-server /slots endpoint to get the needed data, which means if you connect opencode to another provider the plugin will just display "-" since it's not getting any data to display.

Note: As explained in further detail below the data that's displayed has to be deduced from llama-server's output. Sometimes the plugin might display PP for prompt processing while in reality the model is doing TG. If additional developments are made to the llama-server output data the plugin might be able to discern between them in a better way, but I think is as good as it gets for now.

I made this for my own usage. If you find it useful as well I'm just happy.

/Troed

_thanks to Tarquinen for their [oc-tps](https://github.com/Tarquinen/oc-tps), which I used as a base although I guess most of the code has now been replaced_

## Installation

```bash
opencode plugin @troed/oc-ls-stats@latest --global
```

Requires `opencode` `1.3.14` or newer.

TUI plugins are loaded from `~/.config/opencode/tui.json`, which after installation should look like this:

```json
{
  "plugin": ["@troed/oc-ls-stats@latest"]
}
```

## Display Format

The plugin renders a single line in the session prompt right slot:

```
1247 tps (PP)    -- during prefill
  25 tps (TG)    -- during generation
   - tps (TG)    -- idle
          n/a    -- unable to reach llama-server
```

## Detection and Calculation

### Server Discovery

By default the plugin discovers the llama-server URL from OpenCode's resolved provider list (`id` or `name` containing "llama", excluding "ollama"), extracting `baseURL`/`base_url` from the provider options. This covers providers from all sources (config, environment, API). If the provider list is unavailable, it falls back to parsing the OpenCode configuration directly, and finally to `http://localhost:8080`.

When several llama-server providers are detected, stats follow the provider used by the current session; until a session has picked a model, every detected server is polled.

If your llama-server provider is named differently, configure the endpoint explicitly via the plugin's `server` option in `tui.json`. An explicit server replaces discovery entirely:

```json
{
  "plugin": [
    ["@troed/oc-ls-stats@latest", { "server": "http://headless.local:8080" }]
  ]
}
```

A `baseURL`-style value ending in `/v1` is also accepted.

### Slot Polling

Every 500ms, the plugin polls `GET /slots?model=<model>` on each discovered server. The model parameter is required by the `/slots` endpoint. If the model cannot be discovered from the current route's session, the plugin skips polling.

When a server returns an error (e.g. HTTP 502) or is unreachable, the plugin backs off with an exponentially increasing delay per server (1s, 2s, 4s, ...) capped at 10s, with a small random jitter to stagger multiple servers. Backoff resets as soon as the server responds successfully again.

### State Classification

Each processing slot is classified as prefill (PP) or generation (TG) by comparing `n_decoded` and `n_prompt_tokens` against per-slot/per-task baselines. The plugin tracks two baseline maps keyed by slot ID:

- `prevNdBySlot[slotId]` — stores `{ prevNd, idTask, classification }`
- `prevPromptTokensBySlot[slotId]` — stores `{ tokens, idTask }`

The `idTask` field ensures baselines are invalidated when a slot starts a new task (task reuse).

Classification logic:

1. **First time or after task change/reset**: If we have no baseline for this slot+task, or `n_decoded` or `n_prompt_tokens` dropped below baseline (indicating a new request), use a heuristic: `n_decoded > 0` → TG, otherwise → PP.
2. **Compare against baseline**: If `n_decoded > baselineNd` → TG. If `n_prompt_tokens > baselineNt` → PP.
3. **Neither increased**: Persist the previous classification. This handles frequent polling where neither field increases on every poll cycle.

When a slot stops being processed, all tracked state for that slot is cleared.

### Prefill Rate (PP)

During prefill, the plugin calculates the instantaneous prompt processing rate:

1. On first detection of a prefill slot, the current `n_prompt_tokens` is captured as the baseline.
2. On subsequent polls, the delta in `n_prompt_tokens` is divided by the elapsed time in seconds.
3. The rate is updated only when both `dt > 0` and `delta > 0`.

The per-slot `n_prompt_tokens` field is used instead of the global `llamacpp:prompt_tokens_total` from `/metrics` because the global counter includes tokens from all slots, producing inflated values when multiple slots are active simultaneously.

### Generation Rate (TG)

During generation, the plugin calculates the instantaneous token generation rate:

1. On first detection of a generation slot, the current `n_decoded` is captured as the baseline.
2. On subsequent polls (same slot ID), the delta in `n_decoded` is divided by the elapsed time in seconds.
3. The rate is updated only when both `dt > 0` and `delta > 0`.

Slot reuse is tracked via `generateSlotId` to detect when a new generation starts on a different slot.

## Limitations

### Progress Percentage

The plugin cannot display prefill progress percentage. The `/slots` endpoint returns `n_prompt_tokens` (current prompt size) and `n_prompt_tokens_processed` (tokens processed), but not the final prompt size (`task->n_tokens()` from llama.cpp). Progress requires the ratio `n_prompt_tokens_processed / task->n_tokens()`.

### What Would Improve Compatibility

The following changes to the `/slots` endpoint would improve the plugin's functionality:

1. **Expose final prompt size**: Add `n_prompt_tokens_total` (or `n_tokens`) to the `/slots` output, representing `task->n_tokens()` from llama.cpp. This would enable prefill progress percentage calculation as `(n_prompt_tokens_processed / n_prompt_tokens_total) * 100`.

2. **Per-slot metrics endpoints**: Currently, the `/metrics` endpoint provides only global counters (`llamacpp:prompt_tokens_total`, `llamacpp:prompt_tokens_seconds`). Per-slot metrics would allow independent rate tracking without relying on slot state classification.

3. **Slot transition notifications**: The plugin polls every 500ms to detect state transitions. A WebSocket or SSE-based notification system for slot state changes would reduce polling overhead and improve detection latency.

4. **Stall detection**: When generation stalls (e.g., due to context window limits), `n_decoded` remains constant while `n_remain` stops decreasing. The plugin detects this via zero delta but has no way to distinguish a stall from normal generation. An explicit `stalled` flag in the slot output would help.

5. **Model-agnostic slot data**: The `/slots` endpoint requires a model parameter. Returning all slots without model filtering, or supporting `*` as a wildcard, would simplify discovery when multiple models are loaded.

## Source code repo

For known issues, posting new ones, forking or contributing:

https://git.sync.wtf/troed/oc-ls-stats

## Debug Logging

Debug logging is controlled by the `DEBUG_ENABLED` constant in `tui.tsx`. When enabled, full slot state data is written to `/tmp/oc-ls-stats-debug.log` on every poll.

## License

Creative Commons Zero (CC0 1.0 Universal)