sv6160: record which descriptors the TX stall loses, and retire the 55% figure

The TX stall has been described since 2026-09-11 as hitting "~55% of
associations, a few seconds into data traffic".  That is no longer true of this
driver, and leaving the number in the notes sends the next person hunting a bug
with the wrong shape.

Measured on the current build:

  - 12 forced re-associations, each followed immediately by a 6 MB transfer --
    precisely the scenario the 55% figure describes -- produced ZERO resyncs.
    Expected six or seven.
  - 3.5 hours of continuous TX+RX on one association: 1,171 cycles, ~14.7 GB,
    1,266,409 completions, ZERO resyncs.
  - The 8-hour soak the day before saw 5 stalls, i.e. ~0.6/hour, and they were
    BURSTY: three inside 27 minutes with 1h46m clean before and 2h55m after.

So it is rare and clustered, not a per-association coin flip.  Several fixes
since that figure was taken plausibly account for it (the RX aggregate cap, the
TX double-free, and the PIO->DMA change all touched this path).

What is still true, and is the useful part: when a resync does take a descriptor
back, NO completion ever arrives for it -- tx_late 0 and tx_skip_noskb 0 across
every run.  The firmware really does drop it.  And the stall window has a wide
margin: 1,266,409 completions with tx_done_max_ms of 170 against a 1000 ms
timeout, so the resync is not firing on merely-slow completions.  (Note the
signal there is tx_skip_noskb, not tx_late: the skip check runs first and goes to
out, so a late completion for a retired descriptor can never reach the tx_late
branch.)

Add the instrumentation the next stall needs, since they are hours apart and
there is no second chance:

  tx_lost_desc  per-descriptor-id count of what the resync took back
  tx_lost_ac    the same per access category
  oldest lost   how long the worst one had been outstanding

That answers the obvious questions directly -- always the same id? always one
AC? -- which is otherwise unanswerable from "resynchronised N descriptor(s)".

Also relax sv6160_fw_show_txrx_count() to decode a short reply.  This firmware
answers FW_PRIV_DUMP_TXRX_COUNT_INFO with 92 bytes where the vendor header
describes 228, so the old length check threw the whole thing away.  Worth noting
the older warning that this dump "wedges the bus" is stale -- it does not, and
almost certainly never did after the RX aggregate cap was fixed.  It is useless
for a different reason: all 92 bytes read ZERO after thousands of descriptors.
cmd_tx_send vs event_tx_done_recv would have been exactly the measurement this
bug wants, and this firmware does not keep it, just as RPU_CMD_MIB_STATS answers
with an FW_ERROR and fw_log comes back empty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JSNniou44nN9EFGqTmZ6my
This commit is contained in:
Paul KilarandClaude Opus 5 committed 2026-09-14 03:07:18 -04:00
1 parent 24e6aa5a74
commit cde6a9de8d
2 files changed
+54 -3

No files matched your search

@@ -914,6 +914,14 @@ struct sv6160_softc {
* one host-side thing that produces exactly that ratio. */
u32 txs_body_pio; /* body writes dw_mmc must PIO */
u32 txs_desc_pio; /* descriptor writes ditto */
/* Which descriptors the stall resync had to take back, so "the firmware
* stopped completing" can be asked the obvious next questions: is it
* always the same descriptor id, always the same AC, and how long had it
* been outstanding? Per-id because a pattern only shows across many
* stalls, and they are hours apart. */
u32 txs_lost_desc[SV6160_NUM_TX_DESCS];
u32 txs_lost_ac[SV6160_NUM_ACS];
u32 txs_resync_age_max; /* ms, oldest ever lost */
/* Times the worker stopped ALL mac80211 queues because one AC had no
* descriptor left, and which AC caused it. */
/* Rate asked for vs rate the firmware reports using. */
@@ -2107,9 +2107,24 @@ static void sv6160_fw_show_txrx_count(struct sv6160_softc *sc,
{
const struct sv6160_fw_txrx_count *c = data;
if (len < sizeof(*c)) {
dev_info(sc->dev, "FWCNT short reply (%u < %zu)\n",
len, sizeof(*c));
/* Layout varies by firmware build: this one answers 92 bytes where the
* vendor header describes 228, its struct stopping partway through the
* per-path timing blocks. So decode whatever head is present rather than
* rejecting the reply for being short, and treat the timing tail as
* optional.
*
* Do not expect anything from it on THIS build: all 92 bytes come back
* ZERO, verified after thousands of descriptors. That kills the one
* measurement the TX stall really wants -- cmd_tx_send vs
* event_tx_done_recv, the firmware's own account of descriptors taken
* versus completed, plus lpw_hang / err_desc_id_* to say whether it knows
* it dropped them. It is the same story as RPU_CMD_MIB_STATS, which
* answers with an FW_ERROR (see sv6160_get_survey), and as fw_log, which
* is empty even on the vendor driver: this firmware is stripped of its
* self-reporting. Kept decoding in case a future build fills it in. */
if (len < offsetof(struct sv6160_fw_txrx_count, wifi_isr_info)) {
dev_info(sc->dev, "FWCNT short reply (%u < %zu)\n", len,
offsetof(struct sv6160_fw_txrx_count, wifi_isr_info));
return;
}
@@ -2132,6 +2147,8 @@ static void sv6160_fw_show_txrx_count(struct sv6160_softc *sc,
le16_to_cpu(c->err_desc_id_host),
le16_to_cpu(c->wifi_int_disabled),
(int)le32_to_cpu(c->lpw_rx_q_min));
if (len < sizeof(*c))
return; /* timing tail absent on this firmware */
sv6160_fw_show_time(sc, "wifi_isr", &c->wifi_isr_info);
sv6160_fw_show_time(sc, "rx_notify", &c->rx_notify);
sv6160_fw_show_time(sc, "rx_begin", &c->rx_begin);
@@ -4798,6 +4815,21 @@ static int sv6160_tx_retire_all(struct sv6160_softc *sc)
* arriving at the same moment cannot both own the batch. */
if (skb && cmpxchg(&sc->tx_pending[slot], skb, NULL) == skb) {
int k, nb = sc->tx_batch_n[id];
u8 ac = sc->tx_token_ac[id];
/* Record what we are taking back before the state is
* cleared -- this is the only evidence we get about a
* stall, and they are hours apart. */
sc->txs_lost_desc[id]++;
if (ac < SV6160_NUM_ACS)
sc->txs_lost_ac[ac]++;
if (sc->tx_desc_jiffies[id]) {
u32 ms = jiffies_to_msecs(jiffies -
sc->tx_desc_jiffies[id]);
if (ms > sc->txs_resync_age_max)
sc->txs_resync_age_max = ms;
}
for (k = 0; k < nb; k++) {
struct sk_buff *f = sc->tx_batch[id][k];
@@ -6423,6 +6455,17 @@ static int sv6160_debugfs_tx_tokens_show(struct seq_file *s, void *data)
sc->txs_retry_hist[4], sc->txs_retry_hist[5],
sc->txs_retry_hist[6], sc->txs_retry_hist[7]);
seq_printf(s, "tx_resyncs %u\n", sc->txs_resyncs);
{
int i;
seq_puts(s, "tx_lost_desc ");
for (i = 0; i < SV6160_NUM_TX_DESCS; i++)
seq_printf(s, "%u:%u ", i, sc->txs_lost_desc[i]);
seq_printf(s, "\ntx_lost_ac %u %u %u %u %u (oldest lost %u ms)\n",
sc->txs_lost_ac[0], sc->txs_lost_ac[1],
sc->txs_lost_ac[2], sc->txs_lost_ac[3],
sc->txs_lost_ac[4], sc->txs_resync_age_max);
}
seq_printf(s, "tx_resync_descs %u\n", sc->txs_resync_descs);
seq_printf(s, "tx_resync_runs %u\n", sc->txs_resync_runs);
seq_printf(s, "tx_done_avg_ms %llu\n",