sv6160: recover a wedged SDIO bus by re-creating the host

A hard SDIO wedge has stranded this device repeatedly, and nothing the driver
could do brought it back -- each time it needed an unbind/bind of
ff380000.mmc typed in over the serial console.  With tx_batch>9 as a reliable
way to provoke one, the recovery ladder could finally be measured on a real
wedge rather than guessed at:

    ieee80211_restart_hw   clears a SOFT wedge well (~11 s, "attempt 1 OK"),
                           but on a hard one it does nothing; watched once
                           taking a wedge from 4 to 8 resyncs and 6 to 13
                           lost descriptors
    mmc_hw_reset           returns SUCCESS and does not help.  With a single
                           function probed it is mmc_power_cycle +
                           mmc_sdio_reinit_card, and dmesg shows the bus
                           dutifully renegotiating 400kHz -> 50MHz while
                           staying wedged
    module reload          probe still fails -16, no netdev
    unbind/bind            CLEARS it every time

The problem is the HOST, not the card: only re-running dw_mci_probe() puts
the controller back in a working state.  A function driver cannot call that,
but it can ask the driver core to re-bind its own grandparent, which is
exactly what the sysfs unbind/bind does.  sv6160_rebind_sdio_host() does so
from a STANDALONE work item, because device_release_driver() on the dw_mmc
platform device tears down the host, the card and this driver -- our
.remove() runs inside that call -- so the work cannot live in the softc it is
about to free.  It holds a reference on the platform device and on the module,
and is one-shot until a probe succeeds.

It escalates on REPEATED RUNTIME WEDGES, not on a single fault: three bus
recoveries inside 60 s (host_rebind, 0 disables) means restart_hw is being
asked to fix something it cannot, and each extra round trip is another ~11 s
of dead radio.  Three is safe to trigger on -- a recovery only fires on a
genuine -16/-110 fault, and a 10 h soak of 4.1 M descriptors produced zero --
against an ~8 s cure.  The bring-up retry path escalates too, once its four
fw_reboot attempts are spent.

VERIFIED on hardware, from a cleared dmesg so every count is provably new:

    hit 0: TimeoutError   escalations=0
    hit 1: TimeoutError   escalations=0
    hit 2: TimeoutError   escalations=1
    "3 bus recoveries in 60 s - restart_hw is not clearing it,
     re-creating the SDIO host"
    -> recovered UNATTENDED

and the mechanism separately: the netdev is renumbered (wlan15 -> wlan16, so
the card really is destroyed and re-enumerated), the bus re-inits from
400kHz, and the link is back with its IP in 8 s -- against the serial-console
rescue it needed before.

NOT hooked into probe, though that looks like the obvious place.  Once the bus
is hard-wedged our probe is never entered at all: sdio_bus_probe() calls
sdio_set_block_size() before drv->probe, that CMD52 fails -EBUSY, and the core
logs "probe failed -16" without ever calling us -- the failing reloads have no
probe entry line in dmesg.  A driver already rmmod'ed on a wedged bus is past
helping itself, and only a test harness reloads a wedged driver.  The comment
there says so, because the hook looks correct and will otherwise be re-added.

debugfs "recover" takes a level now: 1 (or empty) restarts the hw as before,
2 re-creates the host, so the deep path has a deliberate manual trigger.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JSNniou44nN9EFGqTmZ6my
This commit is contained in:
Paul KilarandClaude Opus 5 committed 2026-09-17 10:30:20 -04:00
1 parent d99b1e305b
commit 53493a14b3
3 files changed
+194 -3

No files matched your search

@@ -647,6 +647,9 @@ struct rpu_cmd_txq_params {
* 3% does not pay for standing on the edge of that cliff. Every array here
* scales off this constant (bodies[], batch[], tx_batch[][]; frm_status[] in
* the event is already 16), so changing it stays a one-line edit. */
/* Window over which repeated bus recoveries are counted. */
#define SV6160_RECOVER_WINDOW_MS 60000
#define SV6160_TX_BATCH_MAX 8
/* Most pieces mac80211 may hand us in one (A-)MSDU; we flatten them anyway,
* this just bounds the copy. */
@@ -1167,6 +1170,10 @@ struct sv6160_softc {
* so bring-up (and future runtime paths) can trigger the vendor-style
* mmc_hw_reset recovery (sv6160_sdio_fw_reboot) and retry. */
bool sdio_wedged;
/* Rolling window used to tell "one bad moment" from "restart_hw is not
* fixing this" -- see sv6160_trigger_recovery. */
unsigned long recover_window;
u32 recover_count;
/* mac80211 state */
struct ieee80211_vif *vif;
@@ -1285,6 +1292,12 @@ void sv6160_rx_drain(struct sv6160_softc *sc);
void sv6160_stop_queues(struct sv6160_softc *sc);
void sv6160_start_queues(struct sv6160_softc *sc);
/* Last-resort wedge recovery: ask the driver core to re-create the dw_mmc host
* that owns our card. Takes the sdio_func, not the softc, because the wedge
* that most needs it is the one where PROBE fails -16 and no softc exists. */
void sv6160_rebind_sdio_host(struct sdio_func *func);
void sv6160_rebind_note_probe_ok(void);
/* PM Support */
void sv6160_notify_pm(struct sv6160_softc *sc, int wakeup);
@@ -391,6 +391,23 @@ MODULE_PARM_DESC(tx_dma, "Bounce TX bodies through an aligned buffer so dw_mmc c
* matter how fast the radio is. The vendor packs whatever is queued for that
* AC into a single descriptor (rpu_if.c:1881 num_frames_per_desc =
* skb_queue_len(txq)). 1 = the old one-frame-per-descriptor behaviour. */
/* How many bus recoveries inside SV6160_RECOVER_WINDOW_MS before we stop
* retrying the thing that is not working and re-create the SDIO host instead
* (see sv6160_rebind_sdio_host). 0 disables the escalation entirely.
*
* ieee80211_restart_hw clears a soft wedge on the first go -- measured, it
* recovers in about 11 s and reports "bring-up complete (attempt 1 OK)". A
* bus that needs recovering over and over inside a minute is one restart_hw
* cannot fix, and the only cure measured on a real wedge is re-creating the
* host: module reload leaves probe failing -16, unbind/bind brings it back.
*
* Three is deliberately low. A recovery only fires on a genuine -16/-110
* bus fault, which never happened once in a 10 h soak of 4.1 M descriptors,
* so three inside a minute is already pathological, and the cure costs ~8 s. */
static int mod_host_rebind = 3;
module_param_named(host_rebind, mod_host_rebind, int, 0644);
MODULE_PARM_DESC(host_rebind, "Bus recoveries within 60s before re-creating the SDIO host (unbind/bind dw_mmc): 3=default, 0=off");
static int mod_tx_batch = SV6160_TX_BATCH_MAX;
module_param_named(tx_batch, mod_tx_batch, int, 0644);
MODULE_PARM_DESC(tx_batch, "Max frames packed into one TX descriptor (1 = off, default 8)");
@@ -609,6 +626,100 @@ int sv6160_sdio_readb(struct sv6160_softc *sc, u8 addr, u8 *val)
* mac80211 restart_hw replay the connection. We mirror that:
* ieee80211_restart_hw() drives mac_stop + mac_start, and mac_start's
* sv6160_sdio_fw_reboot() does the mmc_hw_reset that un-wedges DAT0. */
/* Deep recovery: tear down and re-create the dw_mmc host that owns our card.
*
* This is the ONLY thing measured to clear a hard SDIO wedge. Everything
* cheaper has been tried on a real one (reproduced on demand with tx_batch=16,
* which the firmware rejects -- see SV6160_TX_BATCH_MAX):
*
* ieee80211_restart_hw no effect; on a softer wedge it made things
* worse, 4 -> 8 resyncs and 6 -> 13 lost descs
* mmc_hw_reset (fw_reboot) returns SUCCESS and does not help. With one
* function probed it is just mmc_power_cycle +
* mmc_sdio_reinit_card, and dmesg shows the bus
* dutifully renegotiating 400kHz -> 50MHz while
* staying wedged
* module reload probe still fails -16, no netdev
* unbind/bind ff380000.mmc CLEARS it -- probes, associates, 1:1 counters
*
* The problem is the HOST, not the card: only re-running dw_mci_probe() puts
* the controller back in a working state. A function driver cannot call that,
* but it can ask the driver core to re-bind its own grandparent, which is
* exactly what the sysfs unbind/bind does.
*
* Runs from a STANDALONE work item on purpose. device_release_driver() on the
* dw_mmc platform device tears down the host, the card and this very driver --
* our .remove() runs inside this call -- so the work must not live in the
* softc that is about to be freed, and must not touch it afterwards. It holds
* a reference on the platform device and on this module so neither can vanish
* mid-flight. */
struct sv6160_host_rebind {
struct work_struct work;
struct device *host_dev;
};
static void sv6160_host_rebind_fn(struct work_struct *work)
{
struct sv6160_host_rebind *rb =
container_of(work, struct sv6160_host_rebind, work);
struct device *dev = rb->host_dev;
int ret;
dev_warn(dev, "sv6160: re-creating the SDIO host to clear a wedged bus\n");
device_release_driver(dev); /* takes this driver down with it */
ret = device_attach(dev);
dev_warn(dev, "sv6160: SDIO host re-bind %s (%d)\n",
ret > 0 ? "OK" : "FAILED", ret);
put_device(dev);
kfree(rb);
module_put(THIS_MODULE);
}
/* One shot per module load, reset by a probe that succeeds. The guard has to
* live here rather than in the softc: the wedge that most needs this is the one
* where PROBE itself fails -16, and at that point there is no softc at all. */
static bool sv6160_host_rebound;
void sv6160_rebind_note_probe_ok(void)
{
sv6160_host_rebound = false;
}
/* Ask for the host to be re-created. Safe from any context, and safe to call
* with no softc. Does nothing if the topology is not what we expect rather
* than guessing at a device. */
void sv6160_rebind_sdio_host(struct sdio_func *func)
{
struct sv6160_host_rebind *rb;
struct device *host_dev;
if (!mod_host_rebind)
return;
if (!func || !func->card || !func->card->host)
return;
host_dev = func->card->host->parent;
if (!host_dev)
return;
if (sv6160_host_rebound) {
dev_warn(&func->dev,
"SDIO host already re-created once - not looping on it\n");
return;
}
sv6160_host_rebound = true;
rb = kzalloc(sizeof(*rb), GFP_ATOMIC);
if (!rb)
return;
if (!try_module_get(THIS_MODULE)) {
kfree(rb);
return;
}
INIT_WORK(&rb->work, sv6160_host_rebind_fn);
rb->host_dev = get_device(host_dev);
queue_work(system_unbound_wq, &rb->work);
}
static void sv6160_recover_work_fn(struct work_struct *work)
{
struct sv6160_softc *sc = container_of(work, struct sv6160_softc,
@@ -626,6 +737,28 @@ static void sv6160_trigger_recovery(struct sv6160_softc *sc)
{
if (!sc->is_started || sc->bringup_active || sc->recovering)
return;
/* Rolling count of how often this bus has needed rescuing. One
* recovery is routine; a handful in a minute means restart_hw is being
* asked to fix something it cannot, and every extra round trip is
* another 11 s of dead radio. Escalate instead. */
if (!sc->recover_window ||
time_after(jiffies, sc->recover_window +
msecs_to_jiffies(SV6160_RECOVER_WINDOW_MS))) {
sc->recover_window = jiffies;
sc->recover_count = 0;
}
sc->recover_count++;
if (mod_host_rebind && sc->recover_count >= mod_host_rebind) {
dev_err(sc->dev,
"%u bus recoveries in %u s - restart_hw is not clearing it, re-creating the SDIO host\n",
sc->recover_count, SV6160_RECOVER_WINDOW_MS / 1000);
sc->recover_count = 0;
sv6160_rebind_sdio_host(sc->func);
return;
}
sc->recovering = true;
schedule_work(&sc->recover_work);
}
@@ -3963,9 +4096,14 @@ static int sv6160_fw_bringup(struct sv6160_softc *sc, u32 init_channel)
* SDIO fault it power-cycles the card (mmc_hw_reset, rk915_sdio_power_cycle)
* and lets mac80211 restart. We mirror that here: each attempt boots the
* firmware cold (sv6160_sdio_fw_reboot does mmc_hw_reset + sdio_enable_func
* + device_init + firmware download — the same power-cycle that un-wedges
* a stuck DAT0, proven equivalent to a dwmmc unbind/rebind), runs the
* bring-up, and retries if sc->sdio_wedged got set mid-sequence. */
* + device_init + firmware download), runs the bring-up, and retries if
* sc->sdio_wedged got set mid-sequence.
*
* This recovers a merely busy card. It is NOT equivalent to a dwmmc
* unbind/rebind, though this comment used to claim so: measured on a
* hard wedge, fw_reboot and a module reload both leave probe failing
* -16, while unbind/bind brings it straight back. When the retries are
* exhausted we therefore escalate to sv6160_rebind_sdio_host(). */
{
for (attempt = 1; attempt <= max_attempts; attempt++) {
sc->sdio_wedged = false;
@@ -4111,6 +4249,11 @@ static int sv6160_fw_bringup(struct sv6160_softc *sc, u32 init_channel)
"fw_bringup: wedged after %d attempts (ch=%u)\n",
max_attempts, init_channel);
sc->bringup_active = false;
/* Every attempt above already did a full fw_reboot, i.e.
* mmc_hw_reset + re-download. Measured: that does not
* clear a hard wedge, and neither does reloading the
* module. Escalate to the one thing that does. */
sv6160_rebind_sdio_host(sc->func);
return -EIO;
}
}
@@ -6484,6 +6627,23 @@ static ssize_t sv6160_debugfs_recover_write(struct file *file,
size_t count, loff_t *ppos)
{
struct sv6160_softc *sc = file_inode(file)->i_private;
char kbuf[8] = {};
int level = 1;
if (count && count < sizeof(kbuf) && !copy_from_user(kbuf, buf, count)) {
if (kstrtoint(strim(kbuf), 0, &level))
level = 1;
}
/* 2 = the deep one: re-create the SDIO host, the only thing measured to
* clear a hard wedge. Deliberately reachable by hand, because that is
* how it gets exercised -- the automatic path fires from probe failure,
* which is hard to provoke on demand. */
if (level >= 2) {
dev_info(sc->dev, "recover: re-creating the SDIO host via debugfs\n");
sv6160_rebind_sdio_host(sc->func);
return count;
}
if (!sc->is_started)
return -ENODEV;
@@ -317,9 +317,27 @@ static int sv6160_probe(struct sdio_func *func,
if (ret == -ETIMEDOUT)
dev_err(&func->dev,
"SDIO init timed out (-110). If the system suspended with this driver unloaded, the mmc host was not resumed; power-cycle it with unbind/bind on dwmmc_rockchip, or keep the driver loaded across suspend.\n");
/* Do NOT try to escalate to sv6160_rebind_sdio_host() here. It
* looks like the right place and it cannot work: once the bus is
* hard-wedged this probe is never entered at all. sdio_bus_probe()
* calls sdio_set_block_size() before drv->probe, that CMD52 fails
* -EBUSY, and the core logs "probe with driver sv6160 failed with
* error -16" without ever calling us -- verified on hardware, the
* failing reloads have no probe entry line in dmesg.
*
* The driver can only rescue the host while it still HOLDS the
* device, so the escalation lives in the bring-up retry path
* instead. A driver that has been rmmod'ed on a wedged bus is
* already past helping itself; that state needs the unbind/bind
* below, by hand. (It is also not something production does --
* only a test harness reloads a wedged driver.) */
goto err_disable;
}
/* Far enough in that the host is demonstrably alive: re-arm the
* one-shot so a later wedge can be rescued too. */
sv6160_rebind_note_probe_ok();
/* Allocate mac80211 hardware struct */
hw = ieee80211_alloc_hw(sizeof(struct sv6160_softc), &sv6160_ops);
if (!hw) {