sv6160: stop reusing a descriptor id the firmware still owns

The TX stall was not one event, it was a loop, and we were driving it.

When completions stop, sv6160_tx_resync_fn() retires the outstanding
descriptors and frees their tokens.  sv6160_get_token() then hands the same id
straight back: for a given AC it tries queue + 0 first, so BE always gets id 1
again.  The firmware never completed that descriptor and still owns it, and it
says so plainly in an FW_ERROR:

    RPU: firmware error!  "[19308661 F] desc_id(1) reused"

then drops the new descriptor.  That one never completes either, so the next
resync fires, and reuses the id again.

The per-descriptor counters added in cde6a9d caught it overnight, and every
number fits the loop rather than a firmware that loses bursts of traffic:

  tx_resyncs 20, tx_resync_descs 20   -- exactly ONE descriptor per resync
  tx_lost_desc  1:19  3:1             -- 19 of 20 on id 1, NEVER on id 6
  tx_lost_ac    0 19 0 1 0            -- AC1
  20 resyncs inside 13 minutes, then nothing for hours

id 1 is not special to the firmware; it is simply the first id get_token
returns for AC1, hence always the one reused immediately after a resync.  And
one descriptor per resync means the pipe was nearly idle at the time -- this
fires on light traffic, which is why an idle night produced 20 of them while
3.5 hours of saturated load produced none.

So: quarantine an id that a stall resync reclaimed.  sv6160_tx_retire_all()
gains a `stalled` flag, because the two callers mean different things -- on the
stall path the firmware still owns the descriptors, while on a link flush it has
been told to disconnect and its table is about to be rebuilt, where quarantining
would only stop the next association from putting its auth frame on the air.
get_token() skips a held id, so the AC falls through to its other descriptor and
then the spares instead of hammering the one the firmware is rejecting.  A
completion arriving for a quarantined id releases it immediately: that is the
firmware telling us it is finished, which beats any timeout.

Verified on hardware that the mechanism engages: 4 quarantines set, 3
allocations deflected, and the loss histogram spreads to ids 6, 8 and 10 where
before it was 19-on-id-1-and-nothing-else.  Normal traffic is unaffected --
TX 0.93-0.98 MB/s, RX 2.62, no new resyncs, no oops.

What is NOT verified is that this prevents a real cascade, because the primary
stall still cannot be reproduced on demand.  tx_stall_ms=1 forces resyncs but
not the fault: it retires descriptors the firmware completes normally a few ms
later, and produces no desc_id-reused errors at all.  The next natural stall
will show whether the loop is broken; tx_quarantine_ms=0 restores the old
behaviour for comparison.  This addresses the cascade -- 19 of last night's 20
events -- not whatever causes the first stall, which remains open.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JSNniou44nN9EFGqTmZ6my
This commit is contained in:
Paul KilarandClaude Opus 5 committed 2026-09-14 09:32:20 -04:00
1 parent cde6a9de8d
commit 43d2f3f15a
2 files changed
+82 -5

No files matched your search

@@ -919,6 +919,15 @@ struct sv6160_softc {
* always the same descriptor id, always the same AC, and how long had it
* been outstanding? Per-id because a pattern only shows across many
* stalls, and they are hours apart. */
/* Deadline (jiffies) before which a descriptor id must NOT be handed out
* again, or 0 when it is free. Set when a stall resync takes an id back
* WITHOUT the firmware having completed it: the firmware still owns it
* then, and reusing it immediately is what turned one stall into a
* storm (see sv6160_get_token). */
unsigned long tx_quarantine[SV6160_NUM_TX_DESCS];
u32 txs_quarantined; /* ids held back */
u32 txs_quarantine_hits; /* allocations deflected */
u32 txs_quarantine_freed; /* released by a late completion */
u32 txs_lost_desc[SV6160_NUM_TX_DESCS];
u32 txs_lost_ac[SV6160_NUM_ACS];
u32 txs_resync_age_max; /* ms, oldest ever lost */
@@ -274,6 +274,30 @@ static int mod_tx_stall_ms = 1000;
module_param_named(tx_stall_ms, mod_tx_stall_ms, int, 0644);
MODULE_PARM_DESC(tx_stall_ms, "Resynchronise the whole TX descriptor pool after N ms with no completion at all (default 1000, 0=off)");
/* How long a descriptor id is held back after a stall resync took it from the
* firmware without a completion.
*
* This is the fix for the stall CASCADE. The resync frees the token, and
* sv6160_get_token() then hands the very same id straight back -- for a given
* AC it always tries queue + 0 first, so BE always gets id 1 again. The
* firmware, which never completed that descriptor, still owns it, and says so:
*
* RPU: firmware error! "[19308661 F] desc_id(1) reused"
*
* and drops the new descriptor. That one then never completes either, which
* trips the next resync, which reuses the id again. Measured over one night:
* 20 resyncs in 13 minutes, 19 of them on id 1 and not one on id 6 (AC1's other
* id), each retiring exactly ONE descriptor -- the signature of a loop feeding
* itself, not of a firmware that lost a burst of traffic.
*
* A few seconds is enough for the firmware to finish with the descriptor on its
* own, and a completion arriving for a quarantined id clears it early, which is
* the firmware telling us directly that it is done. 0 restores the old
* reuse-immediately behaviour, for A/B. */
static int mod_tx_quarantine_ms = 3000;
module_param_named(tx_quarantine_ms, mod_tx_quarantine_ms, int, 0644);
MODULE_PARM_DESC(tx_quarantine_ms, "Hold a descriptor id back for N ms after a stall resync reclaimed it (default 3000, 0=off)");
static int mod_tx_resync_giveup;
module_param_named(tx_resync_giveup, mod_tx_resync_giveup, int, 0644);
MODULE_PARM_DESC(tx_resync_giveup, "Restart the hw after N consecutive fruitless TX resyncs: 0=never (default)");
@@ -2676,8 +2700,17 @@ static void sv6160_process_tx_done(struct sv6160_softc *sc,
tx_done = (struct rpu_event_tx_done *)skb->data;
desc_id = le32_to_cpu(tx_done->descriptor_id);
sc->txs_done_events++;
if (desc_id < SV6160_NUM_TX_DESCS)
if (desc_id < SV6160_NUM_TX_DESCS) {
sc->txs_seen_desc[desc_id]++;
/* A completion for a quarantined id means the firmware has
* finished with it after all, which is better evidence than any
* timeout -- release it now rather than idling the descriptor
* for the rest of the hold-off. */
if (sc->tx_quarantine[desc_id]) {
sc->tx_quarantine[desc_id] = 0;
sc->txs_quarantine_freed++;
}
}
dev_dbg(sc->dev,
"TX_DONE: desc_id=%u queue=%u pdout=%u status=[%u,%u,%u,%u] retries=[%u,%u,%u,%u] rate=[%u,%u,%u,%u]\n",
@@ -4597,6 +4630,22 @@ static void sv6160_mac_bss_info_changed(struct ieee80211_hw *hw,
* Layout follows rk915 get_token(): each AC gets SV6160_TX_DESCS_PER_AC
* reserved ids, then a shared spare pool. Returns -1 when the pool is
* empty, which is back-pressure, not an error. */
/* Is this descriptor id still held back after a stall resync? Expired
* quarantines clear themselves here, so nothing else has to sweep them. */
static bool sv6160_token_quarantined(struct sv6160_softc *sc, int id)
{
unsigned long until = sc->tx_quarantine[id];
if (!until)
return false;
if (time_after(jiffies, until)) {
sc->tx_quarantine[id] = 0;
return false;
}
sc->txs_quarantine_hits++;
return true;
}
static int sv6160_get_token(struct sv6160_softc *sc, u32 queue)
{
int cnt, id;
@@ -4627,6 +4676,8 @@ static int sv6160_get_token(struct sv6160_softc *sc, u32 queue)
for (cnt = 0; cnt < limit; cnt++) {
id = queue + (SV6160_NUM_ACS * cnt);
if (sv6160_token_quarantined(sc, id))
continue;
if (!test_and_set_bit(id, &sc->tx_token_bmp))
goto got;
}
@@ -4636,7 +4687,8 @@ static int sv6160_get_token(struct sv6160_softc *sc, u32 queue)
* the vendor's if the limit is ever relaxed for aggregation. */
for (id = SV6160_NUM_ACS * SV6160_TX_DESCS_PER_AC;
id < SV6160_NUM_TX_DESCS; id++)
if (!test_and_set_bit(id, &sc->tx_token_bmp))
if (!sv6160_token_quarantined(sc, id) &&
!test_and_set_bit(id, &sc->tx_token_bmp))
goto got;
return -1;
@@ -4799,7 +4851,15 @@ static void sv6160_tx_fill_rates(struct sv6160_softc *sc,
* The frames are reported as transmitted rather than failed: they did go out,
* only their completion was lost, and telling the supplicant that a good frame
* failed tears down connections that are working. */
static int sv6160_tx_retire_all(struct sv6160_softc *sc)
/* Take back every descriptor the firmware still holds.
*
* `stalled` says WHY. On the stall path the firmware never completed these and
* still owns them, so their ids must be quarantined before they can be handed
* out again (see mod_tx_quarantine_ms). On a link flush the firmware has been
* told to disconnect and its descriptor table is about to be rebuilt, so the
* ids are genuinely free and quarantining them would only stop the next
* association from putting its auth frame on the air. */
static int sv6160_tx_retire_all(struct sv6160_softc *sc, bool stalled)
{
int id, n = 0;
@@ -4823,6 +4883,11 @@ static int sv6160_tx_retire_all(struct sv6160_softc *sc)
sc->txs_lost_desc[id]++;
if (ac < SV6160_NUM_ACS)
sc->txs_lost_ac[ac]++;
if (stalled && mod_tx_quarantine_ms > 0) {
sc->tx_quarantine[id] = jiffies +
msecs_to_jiffies(mod_tx_quarantine_ms);
sc->txs_quarantined++;
}
if (sc->tx_desc_jiffies[id]) {
u32 ms = jiffies_to_msecs(jiffies -
sc->tx_desc_jiffies[id]);
@@ -4881,7 +4946,7 @@ void sv6160_tx_flush_link(struct sv6160_softc *sc, const char *why)
int n;
sc->txs_resync_runs = 0;
n = sv6160_tx_retire_all(sc);
n = sv6160_tx_retire_all(sc, false);
if (n)
dev_info(sc->dev, "%s: retired %d in-flight descriptor(s)\n",
why, n);
@@ -4930,7 +4995,7 @@ static void sv6160_tx_resync_fn(struct work_struct *work)
return;
}
n = sv6160_tx_retire_all(sc);
n = sv6160_tx_retire_all(sc, true);
sc->txs_resyncs++;
sc->txs_resync_descs += n;
@@ -6455,6 +6520,9 @@ static int sv6160_debugfs_tx_tokens_show(struct seq_file *s, void *data)
sc->txs_retry_hist[4], sc->txs_retry_hist[5],
sc->txs_retry_hist[6], sc->txs_retry_hist[7]);
seq_printf(s, "tx_resyncs %u\n", sc->txs_resyncs);
seq_printf(s, "tx_quarantine %u set, %u deflected, %u freed early\n",
sc->txs_quarantined, sc->txs_quarantine_hits,
sc->txs_quarantine_freed);
{
int i;