sv6160: the TX stall was mostly our own 1000 ms timeout
Two 10-hour runs, same build, same AP, same deliberately IDLE traffic -- the
condition this fault needs, one descriptor outstanding, which is why saturated
load never reproduced it:
tx_stall_ms=1000 99 resyncs; 44 of 103 reclaimed descriptors (43%) were
completed by the firmware afterwards
tx_stall_ms=3000 2 resyncs; tx_frames == tx_done_events (95920 == 95920)
Not one descriptor lost in ten hours. A 50x reduction, and the two survivors
were false positives too: their completions arrived 2 ms and 4 ms after the
resync gave up, which the new tx_latecomp counter measures directly.
So the firmware does not lose descriptors. It is occasionally slow -- a tail
reaching about 3 s where the normal worst case is tx_done_max_ms 257 ms -- and a
1000 ms window cut straight into that tail. Each premature reclaim then handed
the id back to a firmware that still owned it, which rejected it as
"desc_id(1) reused" and caused a real stall, which tripped the next resync. The
bug hunted since 2026-09-11 was largely manufactured by this timeout, with the
id reuse turning each instance into a burst.
Raise the default to 3000 ms. The quarantine from 43d2f3f stays: it is what
makes a premature reclaim survivable at all, it still guards the genuinely slow
cases, and its early-release path is what proved these descriptors were merely
late rather than lost.
tx_latecomp records how long after the resync gave up a completion for a
reclaimed descriptor arrives. It needs no extra state -- the quarantine
deadline is retire_time + hold-off, so the retire time is recoverable -- and it
turns "is the window long enough" into a measurement instead of a guess.
Cost of the longer window: a descriptor that really is dead now pins its AC for
3 s instead of 1 s. That is the right trade against a measured zero losses in
ten hours, and the AC's other descriptor keeps working meanwhile, but it is the
thing to revisit if a genuinely lost descriptor ever turns up.
Smoke-tested on hardware at the new default: TX 1.74/1.90 MB/s, RX 2.73,
1746 frames and 1746 completions, no resyncs, no PIO fallbacks, no oops.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JSNniou44nN9EFGqTmZ6my
This commit is contained in:
1 parent
43d2f3f15a
commit
bd50428975
2 files changed
+52
-3
No files matched your search
@@ -928,6 +928,13 @@ struct sv6160_softc {
|
||||
u32 txs_quarantined; /* ids held back */
|
||||
u32 txs_quarantine_hits; /* allocations deflected */
|
||||
u32 txs_quarantine_freed; /* released by a late completion */
|
||||
/* How late a completion for a reclaimed descriptor actually was, measured
|
||||
* from the moment the resync gave up on it. This is what says whether
|
||||
* tx_stall_ms is set too short: if these are all a few hundred ms, the
|
||||
* window is simply too tight and the "stall" is mostly our own timeout. */
|
||||
u32 txs_latecomp_max_ms;
|
||||
u64 txs_latecomp_sum_ms;
|
||||
u32 txs_latecomp_n;
|
||||
u32 txs_lost_desc[SV6160_NUM_TX_DESCS];
|
||||
u32 txs_lost_ac[SV6160_NUM_ACS];
|
||||
u32 txs_resync_age_max; /* ms, oldest ever lost */
|
||||
|
||||
@@ -269,10 +269,35 @@ MODULE_PARM_DESC(fwpoke_when, "When to apply fwpoke: 0=off (default), 1=after br
|
||||
* 600/600). Earlier baselines on the same hardware ran 1-of-5 and 2-of-5, so
|
||||
* the failure is intermittent and any single trial proves nothing -- which is
|
||||
* exactly how the previous attempt fooled itself. Set 0 to switch it off and
|
||||
* re-measure. */
|
||||
static int mod_tx_stall_ms = 1000;
|
||||
* re-measure.
|
||||
*
|
||||
* The window was 1000 ms, and that turned out to be the real "TX stall". Two
|
||||
* 10-hour runs, same build, same AP, same deliberately IDLE traffic (the
|
||||
* condition the fault needs -- one descriptor outstanding; saturated load
|
||||
* produces none of this):
|
||||
*
|
||||
* tx_stall_ms=1000 99 resyncs, 44 of 103 reclaimed descriptors (43%)
|
||||
* completed by the firmware afterwards
|
||||
* tx_stall_ms=3000 2 resyncs, and tx_frames == tx_done_events
|
||||
* (95920 == 95920), i.e. NOT ONE descriptor lost in 10 h
|
||||
*
|
||||
* A 50x reduction, and the two survivors were false positives as well: their
|
||||
* completions landed 2 ms and 4 ms after the resync gave up (tx_latecomp). So
|
||||
* the firmware does not lose descriptors -- it is occasionally slow, with a
|
||||
* tail reaching ~3 s where the normal maximum is tx_done_max_ms 257 ms -- and a
|
||||
* 1000 ms window was simply cutting into that tail. Worse, every premature
|
||||
* reclaim then handed the id straight back to the firmware, which rejected it
|
||||
* as "desc_id(1) reused" and produced a genuine stall; the bug we spent days
|
||||
* hunting was mostly manufactured by this timeout. See mod_tx_quarantine_ms.
|
||||
*
|
||||
* The cost of the longer window is that a descriptor that really is dead now
|
||||
* pins its AC for 3 s instead of 1 s. That is the right trade at a measured
|
||||
* zero losses in 10 hours, and the AC's other descriptor keeps working
|
||||
* meanwhile, but it is the thing to revisit if a truly lost descriptor ever
|
||||
* shows up. */
|
||||
static int mod_tx_stall_ms = 3000;
|
||||
module_param_named(tx_stall_ms, mod_tx_stall_ms, int, 0644);
|
||||
MODULE_PARM_DESC(tx_stall_ms, "Resynchronise the whole TX descriptor pool after N ms with no completion at all (default 1000, 0=off)");
|
||||
MODULE_PARM_DESC(tx_stall_ms, "Resynchronise the whole TX descriptor pool after N ms with no completion at all (default 3000, 0=off)");
|
||||
|
||||
/* How long a descriptor id is held back after a stall resync took it from the
|
||||
* firmware without a completion.
|
||||
@@ -2707,6 +2732,18 @@ static void sv6160_process_tx_done(struct sv6160_softc *sc,
|
||||
* timeout -- release it now rather than idling the descriptor
|
||||
* for the rest of the hold-off. */
|
||||
if (sc->tx_quarantine[desc_id]) {
|
||||
/* The deadline is retire_time + the hold-off, so the
|
||||
* retire time is recoverable without keeping a second
|
||||
* array: how late this completion is tells us directly
|
||||
* whether tx_stall_ms gave up too early. */
|
||||
unsigned long retired = sc->tx_quarantine[desc_id] -
|
||||
msecs_to_jiffies(mod_tx_quarantine_ms);
|
||||
u32 late = jiffies_to_msecs(jiffies - retired);
|
||||
|
||||
if (late > sc->txs_latecomp_max_ms)
|
||||
sc->txs_latecomp_max_ms = late;
|
||||
sc->txs_latecomp_sum_ms += late;
|
||||
sc->txs_latecomp_n++;
|
||||
sc->tx_quarantine[desc_id] = 0;
|
||||
sc->txs_quarantine_freed++;
|
||||
}
|
||||
@@ -6523,6 +6560,11 @@ static int sv6160_debugfs_tx_tokens_show(struct seq_file *s, void *data)
|
||||
seq_printf(s, "tx_quarantine %u set, %u deflected, %u freed early\n",
|
||||
sc->txs_quarantined, sc->txs_quarantine_hits,
|
||||
sc->txs_quarantine_freed);
|
||||
seq_printf(s, "tx_latecomp n=%u avg=%llu ms max=%u ms (after the resync gave up)\n",
|
||||
sc->txs_latecomp_n,
|
||||
sc->txs_latecomp_n ?
|
||||
sc->txs_latecomp_sum_ms / sc->txs_latecomp_n : 0,
|
||||
sc->txs_latecomp_max_ms);
|
||||
{
|
||||
int i;
|
||||
|
||||
|
||||
Reference in new issue
Block a user