sv6160: survive the post-association wedge — 0.7s associated becomes 15s

Two independent causes, found by loading the driver under serial capture and
watching what actually ends the link.

1. The RX_SERIAS echo.  The firmware delivers a batch, then re-presents the
   batch's FIRST sub-packet length on a later lone host-wake with no data
   behind it.  Measured twice: a 352B batch of (204,28,28,28) followed ~690ms
   later by a wake advertising 204, and a 1024B batch of (408,28,28,28) by one
   advertising 408.  Reading the absent bytes times out and wedges the bus.

   Neither existing clause catches it: Case 1 compares against last_recv_len,
   which holds the aggregate rather than a sub-packet, and Case 2 stands down
   at association.  Neither can lean on the idle heuristics either -- the echo
   lands well inside 2s of the assoc request we just sent, so a "no TX for 2s"
   condition never holds.  My first attempt keyed on every length the drain
   consumed and did not fire for exactly that reason.

   So match the one diagnostic thing: precisely the length that batch handed
   us, only in the window where the echo appears, and only once.

2. Giving up on a busy card.  -EBUSY on a CMD53 write is the chip holding
   DAT0 busy, which it does for hundreds of milliseconds under load -- the
   same behaviour that limits the vendor driver.  Three 5ms retries gave up
   roughly 30x too early, and we then declared the bus wedged and tore the
   link down.  Wait it out instead, ~400ms in 25ms steps, releasing the host
   between attempts so the RX drain can interleave (the chip may be busy
   precisely because it wants us to collect frames).  A retry always restarts
   from the descriptor: desc and body must reach the chip back-to-back under
   one claim, so a failure partway through is not retryable.

   -ETIMEDOUT is left alone: that means the transfer died, and retrying only
   flogs a dead controller.

Measured progression, associated uptime with traffic flowing both ways:

  before          0.7s   -> RX read 204 failed: -110
  after fix 1     6.6s   -> sdio_write_tx failed: -16
  after fix 2    15.4s   -> sdio_write_tx failed: -16, 12 TX_DONE / 28 RX events

The link is now genuinely usable for those seconds rather than dying on the
first batch.  What ends it is the familiar one: TX_DONE stops arriving, we
keep queuing, and a write eventually hits a busy card that never comes back.
That is the same sustained-traffic wedge the vendor driver has, and it is the
next thing to chase.

Also short-circuit the new retry budget once sdio_wedged is already set -- 400ms
per write turned teardown of a dead bus into minutes.  That last bit is built
but not yet exercised on hardware.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Moos7zcsHdpW8DKMq2KM3G
This commit is contained in:
Paul KilarandClaude Opus 5 committed 2026-09-11 00:01:35 -04:00
1 parent 082f123e22
commit 66d5c416b0
2 files changed
+82 -86

No files matched your search

+10 -10
View File
@@ -815,16 +815,16 @@ struct sv6160_softc {
/* Smart RECV_LEN polling — detect stale vs new data without GPIO */
u16 last_recv_len; /* RECV_LEN of the last drained aggregate */
unsigned long last_drain_jiffies;/* when we last drained real RX data */
/* Every packet length consumed by the last drain, including the
* sub-packets inside an RX_SERIAS batch. After a chained or batched
* drain the firmware sometimes re-presents ONE of those lengths on a
* later lone host-wake with no data behind it; reading it times out and
* wedges the bus. last_recv_len alone cannot catch that, because it
* holds the aggregate (e.g. 352) while the stale wake advertises a
* sub-packet (e.g. 204). */
#define SV6160_CONSUMED_LENS 12
u16 consumed_len[SV6160_CONSUMED_LENS];
u8 consumed_n;
/* After an RX_SERIAS batch the firmware re-presents the batch's FIRST
* sub-packet length on a later lone host-wake, with no data behind it.
* Measured twice: a 352B batch of (204,28,28,28) was followed ~690ms
* later by a wake advertising 204, and a 1024B batch of (408,28,28,28)
* by one advertising 408. Reading the absent bytes times out and wedges
* the bus. last_recv_len cannot catch this -- it holds the aggregate,
* not the sub-packet -- so remember the one length that actually gets
* re-presented, and when. */
u16 serias_first_len;
unsigned long serias_jiffies;
bool recv_len_failed; /* last CMD53 at this RECV_LEN timed out */
/* SDIO bus wedge tracking — a -110 CMD53 read timeout (stale RECV_LEN)
@@ -493,7 +493,7 @@ void sv6160_trigger_recovery(struct sv6160_softc *sc)
int sv6160_sdio_write_data(struct sv6160_softc *sc, u32 addr,
void *buf, size_t len)
{
int ret, tries = 3;
int ret, tries;
bool locked = false;
/* Serialize CMD53 writes against the RX drain (sc->lock) so a command
@@ -523,14 +523,18 @@ int sv6160_sdio_write_data(struct sv6160_softc *sc, u32 addr,
32, 1, buf, len, false);
}
sdio_claim_host(sc->func);
do {
tries = sc->sdio_wedged ? 1 : 16;
/* Same patience as the TX path: a busy card is not a wedged bus. The
* whole transaction stays under one claim per attempt so an RX read
* cannot slip between retries of the same write. */
for (;;) {
sdio_claim_host(sc->func);
ret = sdio_memcpy_toio(sc->func, addr, buf, ALIGN(len, 4));
if (!ret)
sdio_release_host(sc->func);
if (!ret || ret != -EBUSY || --tries <= 0)
break;
usleep_range(5000, 6000);
} while (--tries > 0);
sdio_release_host(sc->func);
usleep_range(20000, 25000);
}
if (locked)
mutex_unlock(&sc->lock);
@@ -580,30 +584,42 @@ int sv6160_sdio_write_tx(struct sv6160_softc *sc, void *desc, size_t desc_len,
32, 1, desc, desc_len, false);
}
sdio_claim_host(sc->func);
tries = 3;
do {
/* -EBUSY here is the chip holding SDIO DAT0 busy, which it does for
* hundreds of milliseconds at a time under load -- the same behaviour
* that limits the vendor driver. Three 5ms retries gave up roughly 30x
* too early and we then declared the bus wedged and tore the link down,
* which is what ended every association a few seconds after it
* succeeded. Wait it out: -EBUSY is the card asking for time, unlike
* -ETIMEDOUT, which means the transfer itself died and retrying only
* flogs a dead controller.
*
* The descriptor and body must reach the chip back-to-back under ONE
* host claim -- it pairs them by arrival order -- so a retry restarts
* the whole pair, and only ever from the descriptor. A failure partway
* through (body) is NOT retryable: the chip has already taken a
* descriptor and is waiting for its body. Releasing the host between
* attempts lets the RX drain interleave, which matters because the chip
* may be busy precisely because it wants us to collect received frames. */
/* Once the bus has already been declared wedged there is nothing left to
* wait for, and paying 400ms per write turns teardown into minutes. */
tries = sc->sdio_wedged ? 1 : 16; /* ~400ms of patience, 25ms apart */
for (;;) {
sdio_claim_host(sc->func);
ret = sdio_memcpy_toio(sc->func, SDIO_DATA_ADDR, desc,
ALIGN(desc_len, 4));
if (!ret)
if (ret) {
sdio_release_host(sc->func);
if (ret == -EBUSY && --tries > 0) {
usleep_range(20000, 25000);
continue;
}
break;
usleep_range(5000, 6000);
} while (--tries > 0);
if (ret)
goto out;
tries = 3;
do {
}
ret = sdio_memcpy_toio(sc->func, SDIO_DATA_ADDR, body,
ALIGN(body_len, 4));
if (!ret)
break;
usleep_range(5000, 6000);
} while (--tries > 0);
out:
sdio_release_host(sc->func);
sdio_release_host(sc->func);
break;
}
if (locked)
mutex_unlock(&sc->lock);
@@ -2135,30 +2151,6 @@ void sv6160_rpu_rx_frame(struct sv6160_softc *sc, struct sk_buff *skb)
ieee80211_rx_irqsafe(sc->hw, skb);
}
/* Remember a packet length this drain actually consumed, so a later lone
* host-wake advertising the same length with nothing behind it can be
* recognised as stale instead of read (and -110-wedging the bus). */
static void sv6160_note_consumed_len(struct sv6160_softc *sc, u16 len)
{
int i;
for (i = 0; i < sc->consumed_n; i++)
if (sc->consumed_len[i] == len)
return;
if (sc->consumed_n < SV6160_CONSUMED_LENS)
sc->consumed_len[sc->consumed_n++] = len;
}
static bool sv6160_len_was_consumed(struct sv6160_softc *sc, u16 len)
{
int i;
for (i = 0; i < sc->consumed_n; i++)
if (sc->consumed_len[i] == len)
return true;
return false;
}
/*
* Process RX_SERIAS (event 30) — aggregated batch of RX frames.
* Format: cmd_rx_ctrl header (magic 0x3412ccff, pkt_count, pkt_len[])
@@ -2224,7 +2216,10 @@ void sv6160_rpu_process_rx_batch(struct sv6160_softc *sc, struct sk_buff *skb)
dev_info(sc->dev, "RX_SERIAS[%d]: id=%u len=%u\n",
i, le32_to_cpu(hdr->id), pkt_len);
sv6160_note_consumed_len(sc, pkt_len);
if (i == 0) {
sc->serias_first_len = pkt_len;
sc->serias_jiffies = jiffies;
}
/* Clone this sub-event into its own skb */
sub_skb = dev_alloc_skb(pkt_len);
@@ -2888,39 +2883,41 @@ void sv6160_rx_drain(struct sv6160_softc *sc)
time_after(jiffies, sc->last_tx_time + msecs_to_jiffies(2000)))
goto clear_and_done;
/* Case 3: a length we JUST consumed, re-presented with nothing behind
* it. Cases 1 and 2 both miss this once the link is up: Case 1 only
* matches the aggregate we stored, and Case 2 stands down at
* association (!assoc) because on a live link we must not skip real
* unsolicited RX.
/* Case 3: the RX_SERIAS echo.
*
* MEASURED (2026-09-10), ~700ms after associating: the firmware sent a
* 352-byte RX_SERIAS batch of 4 sub-packets (204, 28, 28, 28), we drained
* it, and on the next lone wake it advertised 204 -- the FIRST
* sub-packet's length -- with no data behind it. last_recv_len was 352
* so Case 1 missed it, assoc was set so Case 2 was off, and the read of
* the absent 204 bytes timed out and wedged the bus, which is what ended
* every association within a couple of seconds.
* MEASURED TWICE (2026-09-10), each time ~690ms after associating: the
* firmware delivers an RX_SERIAS batch, then re-presents the batch's
* FIRST sub-packet length on a later lone host-wake with no data behind
* it. A 352B batch of (204,28,28,28) was followed by a wake advertising
* 204; a 1024B batch of (408,28,28,28) by one advertising 408. Reading
* the absent bytes times out and wedges the bus, which is what ended
* every association a second or two after it succeeded.
*
* Keyed on the exact set of lengths this drain consumed rather than on
* link state, so it cannot swallow an arbitrary real wake: a genuine new
* frame that happens to match a recent length still needs the bus to
* have been idle 250ms AND no TX for 2s, which a live link does not
* satisfy. */
if (mod_stale_guard &&
sv6160_len_was_consumed(sc, pkt_len) &&
time_after(jiffies, sc->last_drain_jiffies + msecs_to_jiffies(250)) &&
time_after(jiffies, sc->last_tx_time + msecs_to_jiffies(2000))) {
* Cases 1 and 2 both miss it: Case 1 compares against last_recv_len,
* which holds the AGGREGATE (352/1024) rather than the sub-packet, and
* Case 2 stands down at association. Neither can use the idle
* heuristics either -- this echo lands well inside 2s of the assoc
* request we just transmitted, so a "no TX for 2s" condition never
* holds here.
*
* So match the one thing that is actually diagnostic: exactly the length
* that batch handed us, only in the window where the echo appears, and
* only once. A real frame of that length arriving in that window is
* possible but would merely be dropped; reading a phantom one takes the
* bus down. */
if (mod_stale_guard && sc->serias_first_len &&
pkt_len == sc->serias_first_len &&
time_after(jiffies, sc->serias_jiffies + msecs_to_jiffies(250)) &&
time_before(jiffies, sc->serias_jiffies + msecs_to_jiffies(2000))) {
dev_info(sc->dev,
"stale wake: RECV_LEN %u was consumed by the last drain — skipping\n",
"stale wake: RECV_LEN %u echoes the last RX_SERIAS batch — skipping\n",
pkt_len);
sc->serias_first_len = 0; /* one-shot */
goto clear_and_done;
}
sc->last_recv_len = pkt_len;
sc->last_drain_jiffies = jiffies;
sc->consumed_n = 0;
sv6160_note_consumed_len(sc, pkt_len);
/* Drain the CHAIN. RECV_LEN gives the FIRST packet's size; each packet
* header's upper 16 bits (rx_next_len) give the NEXT packet's size. The
@@ -2998,7 +2995,6 @@ void sv6160_rx_drain(struct sv6160_softc *sc)
rx_next_len > SV6160_MAX_RX_SIZE)
break;
pkt_len = rx_next_len;
sv6160_note_consumed_len(sc, pkt_len);
}
}