sv6160: the TX stall was mostly our own 1000 ms timeout

Two 10-hour runs, same build, same AP, same deliberately IDLE traffic -- the
condition this fault needs, one descriptor outstanding, which is why saturated
load never reproduced it:

  tx_stall_ms=1000   99 resyncs; 44 of 103 reclaimed descriptors (43%) were
                     completed by the firmware afterwards
  tx_stall_ms=3000    2 resyncs; tx_frames == tx_done_events (95920 == 95920)

Not one descriptor lost in ten hours.  A 50x reduction, and the two survivors
were false positives too: their completions arrived 2 ms and 4 ms after the
resync gave up, which the new tx_latecomp counter measures directly.

So the firmware does not lose descriptors.  It is occasionally slow -- a tail
reaching about 3 s where the normal worst case is tx_done_max_ms 257 ms -- and a
1000 ms window cut straight into that tail.  Each premature reclaim then handed
the id back to a firmware that still owned it, which rejected it as
"desc_id(1) reused" and caused a real stall, which tripped the next resync.  The
bug hunted since 2026-09-11 was largely manufactured by this timeout, with the
id reuse turning each instance into a burst.

Raise the default to 3000 ms.  The quarantine from 43d2f3f stays: it is what
makes a premature reclaim survivable at all, it still guards the genuinely slow
cases, and its early-release path is what proved these descriptors were merely
late rather than lost.

tx_latecomp records how long after the resync gave up a completion for a
reclaimed descriptor arrives.  It needs no extra state -- the quarantine
deadline is retire_time + hold-off, so the retire time is recoverable -- and it
turns "is the window long enough" into a measurement instead of a guess.

Cost of the longer window: a descriptor that really is dead now pins its AC for
3 s instead of 1 s.  That is the right trade against a measured zero losses in
ten hours, and the AC's other descriptor keeps working meanwhile, but it is the
thing to revisit if a genuinely lost descriptor ever turns up.

Smoke-tested on hardware at the new default: TX 1.74/1.90 MB/s, RX 2.73,
1746 frames and 1746 completions, no resyncs, no PIO fallbacks, no oops.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JSNniou44nN9EFGqTmZ6my
This commit is contained in:
Paul KilarandClaude Opus 5 committed 2026-09-15 07:37:42 -04:00
1 parent 43d2f3f15a
commit bd50428975
2 files changed
+52 -3

No files matched your search

@@ -928,6 +928,13 @@ struct sv6160_softc {
u32 txs_quarantined; /* ids held back */
u32 txs_quarantine_hits; /* allocations deflected */
u32 txs_quarantine_freed; /* released by a late completion */
/* How late a completion for a reclaimed descriptor actually was, measured
* from the moment the resync gave up on it. This is what says whether
* tx_stall_ms is set too short: if these are all a few hundred ms, the
* window is simply too tight and the "stall" is mostly our own timeout. */
u32 txs_latecomp_max_ms;
u64 txs_latecomp_sum_ms;
u32 txs_latecomp_n;
u32 txs_lost_desc[SV6160_NUM_TX_DESCS];
u32 txs_lost_ac[SV6160_NUM_ACS];
u32 txs_resync_age_max; /* ms, oldest ever lost */
@@ -269,10 +269,35 @@ MODULE_PARM_DESC(fwpoke_when, "When to apply fwpoke: 0=off (default), 1=after br
* 600/600). Earlier baselines on the same hardware ran 1-of-5 and 2-of-5, so
* the failure is intermittent and any single trial proves nothing -- which is
* exactly how the previous attempt fooled itself. Set 0 to switch it off and
* re-measure. */
static int mod_tx_stall_ms = 1000;
* re-measure.
*
* The window was 1000 ms, and that turned out to be the real "TX stall". Two
* 10-hour runs, same build, same AP, same deliberately IDLE traffic (the
* condition the fault needs -- one descriptor outstanding; saturated load
* produces none of this):
*
* tx_stall_ms=1000 99 resyncs, 44 of 103 reclaimed descriptors (43%)
* completed by the firmware afterwards
* tx_stall_ms=3000 2 resyncs, and tx_frames == tx_done_events
* (95920 == 95920), i.e. NOT ONE descriptor lost in 10 h
*
* A 50x reduction, and the two survivors were false positives as well: their
* completions landed 2 ms and 4 ms after the resync gave up (tx_latecomp). So
* the firmware does not lose descriptors -- it is occasionally slow, with a
* tail reaching ~3 s where the normal maximum is tx_done_max_ms 257 ms -- and a
* 1000 ms window was simply cutting into that tail. Worse, every premature
* reclaim then handed the id straight back to the firmware, which rejected it
* as "desc_id(1) reused" and produced a genuine stall; the bug we spent days
* hunting was mostly manufactured by this timeout. See mod_tx_quarantine_ms.
*
* The cost of the longer window is that a descriptor that really is dead now
* pins its AC for 3 s instead of 1 s. That is the right trade at a measured
* zero losses in 10 hours, and the AC's other descriptor keeps working
* meanwhile, but it is the thing to revisit if a truly lost descriptor ever
* shows up. */
static int mod_tx_stall_ms = 3000;
module_param_named(tx_stall_ms, mod_tx_stall_ms, int, 0644);
MODULE_PARM_DESC(tx_stall_ms, "Resynchronise the whole TX descriptor pool after N ms with no completion at all (default 1000, 0=off)");
MODULE_PARM_DESC(tx_stall_ms, "Resynchronise the whole TX descriptor pool after N ms with no completion at all (default 3000, 0=off)");
/* How long a descriptor id is held back after a stall resync took it from the
* firmware without a completion.
@@ -2707,6 +2732,18 @@ static void sv6160_process_tx_done(struct sv6160_softc *sc,
* timeout -- release it now rather than idling the descriptor
* for the rest of the hold-off. */
if (sc->tx_quarantine[desc_id]) {
/* The deadline is retire_time + the hold-off, so the
* retire time is recoverable without keeping a second
* array: how late this completion is tells us directly
* whether tx_stall_ms gave up too early. */
unsigned long retired = sc->tx_quarantine[desc_id] -
msecs_to_jiffies(mod_tx_quarantine_ms);
u32 late = jiffies_to_msecs(jiffies - retired);
if (late > sc->txs_latecomp_max_ms)
sc->txs_latecomp_max_ms = late;
sc->txs_latecomp_sum_ms += late;
sc->txs_latecomp_n++;
sc->tx_quarantine[desc_id] = 0;
sc->txs_quarantine_freed++;
}
@@ -6523,6 +6560,11 @@ static int sv6160_debugfs_tx_tokens_show(struct seq_file *s, void *data)
seq_printf(s, "tx_quarantine %u set, %u deflected, %u freed early\n",
sc->txs_quarantined, sc->txs_quarantine_hits,
sc->txs_quarantine_freed);
seq_printf(s, "tx_latecomp n=%u avg=%llu ms max=%u ms (after the resync gave up)\n",
sc->txs_latecomp_n,
sc->txs_latecomp_n ?
sc->txs_latecomp_sum_ms / sc->txs_latecomp_n : 0,
sc->txs_latecomp_max_ms);
{
int i;