sv6160: recover a wedged SDIO bus by re-creating the host
A hard SDIO wedge has stranded this device repeatedly, and nothing the driver
could do brought it back -- each time it needed an unbind/bind of
ff380000.mmc typed in over the serial console. With tx_batch>9 as a reliable
way to provoke one, the recovery ladder could finally be measured on a real
wedge rather than guessed at:
ieee80211_restart_hw clears a SOFT wedge well (~11 s, "attempt 1 OK"),
but on a hard one it does nothing; watched once
taking a wedge from 4 to 8 resyncs and 6 to 13
lost descriptors
mmc_hw_reset returns SUCCESS and does not help. With a single
function probed it is mmc_power_cycle +
mmc_sdio_reinit_card, and dmesg shows the bus
dutifully renegotiating 400kHz -> 50MHz while
staying wedged
module reload probe still fails -16, no netdev
unbind/bind CLEARS it every time
The problem is the HOST, not the card: only re-running dw_mci_probe() puts
the controller back in a working state. A function driver cannot call that,
but it can ask the driver core to re-bind its own grandparent, which is
exactly what the sysfs unbind/bind does. sv6160_rebind_sdio_host() does so
from a STANDALONE work item, because device_release_driver() on the dw_mmc
platform device tears down the host, the card and this driver -- our
.remove() runs inside that call -- so the work cannot live in the softc it is
about to free. It holds a reference on the platform device and on the module,
and is one-shot until a probe succeeds.
It escalates on REPEATED RUNTIME WEDGES, not on a single fault: three bus
recoveries inside 60 s (host_rebind, 0 disables) means restart_hw is being
asked to fix something it cannot, and each extra round trip is another ~11 s
of dead radio. Three is safe to trigger on -- a recovery only fires on a
genuine -16/-110 fault, and a 10 h soak of 4.1 M descriptors produced zero --
against an ~8 s cure. The bring-up retry path escalates too, once its four
fw_reboot attempts are spent.
VERIFIED on hardware, from a cleared dmesg so every count is provably new:
hit 0: TimeoutError escalations=0
hit 1: TimeoutError escalations=0
hit 2: TimeoutError escalations=1
"3 bus recoveries in 60 s - restart_hw is not clearing it,
re-creating the SDIO host"
-> recovered UNATTENDED
and the mechanism separately: the netdev is renumbered (wlan15 -> wlan16, so
the card really is destroyed and re-enumerated), the bus re-inits from
400kHz, and the link is back with its IP in 8 s -- against the serial-console
rescue it needed before.
NOT hooked into probe, though that looks like the obvious place. Once the bus
is hard-wedged our probe is never entered at all: sdio_bus_probe() calls
sdio_set_block_size() before drv->probe, that CMD52 fails -EBUSY, and the core
logs "probe failed -16" without ever calling us -- the failing reloads have no
probe entry line in dmesg. A driver already rmmod'ed on a wedged bus is past
helping itself, and only a test harness reloads a wedged driver. The comment
there says so, because the hook looks correct and will otherwise be re-added.
debugfs "recover" takes a level now: 1 (or empty) restarts the hw as before,
2 re-creates the host, so the deep path has a deliberate manual trigger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JSNniou44nN9EFGqTmZ6my
This commit is contained in:
1 parent
d99b1e305b
commit
53493a14b3
3 files changed
+194
-3
No files matched your search
@@ -647,6 +647,9 @@ struct rpu_cmd_txq_params {
|
||||
* 3% does not pay for standing on the edge of that cliff. Every array here
|
||||
* scales off this constant (bodies[], batch[], tx_batch[][]; frm_status[] in
|
||||
* the event is already 16), so changing it stays a one-line edit. */
|
||||
/* Window over which repeated bus recoveries are counted. */
|
||||
#define SV6160_RECOVER_WINDOW_MS 60000
|
||||
|
||||
#define SV6160_TX_BATCH_MAX 8
|
||||
/* Most pieces mac80211 may hand us in one (A-)MSDU; we flatten them anyway,
|
||||
* this just bounds the copy. */
|
||||
@@ -1167,6 +1170,10 @@ struct sv6160_softc {
|
||||
* so bring-up (and future runtime paths) can trigger the vendor-style
|
||||
* mmc_hw_reset recovery (sv6160_sdio_fw_reboot) and retry. */
|
||||
bool sdio_wedged;
|
||||
/* Rolling window used to tell "one bad moment" from "restart_hw is not
|
||||
* fixing this" -- see sv6160_trigger_recovery. */
|
||||
unsigned long recover_window;
|
||||
u32 recover_count;
|
||||
|
||||
/* mac80211 state */
|
||||
struct ieee80211_vif *vif;
|
||||
@@ -1285,6 +1292,12 @@ void sv6160_rx_drain(struct sv6160_softc *sc);
|
||||
void sv6160_stop_queues(struct sv6160_softc *sc);
|
||||
void sv6160_start_queues(struct sv6160_softc *sc);
|
||||
|
||||
/* Last-resort wedge recovery: ask the driver core to re-create the dw_mmc host
|
||||
* that owns our card. Takes the sdio_func, not the softc, because the wedge
|
||||
* that most needs it is the one where PROBE fails -16 and no softc exists. */
|
||||
void sv6160_rebind_sdio_host(struct sdio_func *func);
|
||||
void sv6160_rebind_note_probe_ok(void);
|
||||
|
||||
/* PM Support */
|
||||
void sv6160_notify_pm(struct sv6160_softc *sc, int wakeup);
|
||||
|
||||
|
||||
@@ -391,6 +391,23 @@ MODULE_PARM_DESC(tx_dma, "Bounce TX bodies through an aligned buffer so dw_mmc c
|
||||
* matter how fast the radio is. The vendor packs whatever is queued for that
|
||||
* AC into a single descriptor (rpu_if.c:1881 num_frames_per_desc =
|
||||
* skb_queue_len(txq)). 1 = the old one-frame-per-descriptor behaviour. */
|
||||
/* How many bus recoveries inside SV6160_RECOVER_WINDOW_MS before we stop
|
||||
* retrying the thing that is not working and re-create the SDIO host instead
|
||||
* (see sv6160_rebind_sdio_host). 0 disables the escalation entirely.
|
||||
*
|
||||
* ieee80211_restart_hw clears a soft wedge on the first go -- measured, it
|
||||
* recovers in about 11 s and reports "bring-up complete (attempt 1 OK)". A
|
||||
* bus that needs recovering over and over inside a minute is one restart_hw
|
||||
* cannot fix, and the only cure measured on a real wedge is re-creating the
|
||||
* host: module reload leaves probe failing -16, unbind/bind brings it back.
|
||||
*
|
||||
* Three is deliberately low. A recovery only fires on a genuine -16/-110
|
||||
* bus fault, which never happened once in a 10 h soak of 4.1 M descriptors,
|
||||
* so three inside a minute is already pathological, and the cure costs ~8 s. */
|
||||
static int mod_host_rebind = 3;
|
||||
module_param_named(host_rebind, mod_host_rebind, int, 0644);
|
||||
MODULE_PARM_DESC(host_rebind, "Bus recoveries within 60s before re-creating the SDIO host (unbind/bind dw_mmc): 3=default, 0=off");
|
||||
|
||||
static int mod_tx_batch = SV6160_TX_BATCH_MAX;
|
||||
module_param_named(tx_batch, mod_tx_batch, int, 0644);
|
||||
MODULE_PARM_DESC(tx_batch, "Max frames packed into one TX descriptor (1 = off, default 8)");
|
||||
@@ -609,6 +626,100 @@ int sv6160_sdio_readb(struct sv6160_softc *sc, u8 addr, u8 *val)
|
||||
* mac80211 restart_hw replay the connection. We mirror that:
|
||||
* ieee80211_restart_hw() drives mac_stop + mac_start, and mac_start's
|
||||
* sv6160_sdio_fw_reboot() does the mmc_hw_reset that un-wedges DAT0. */
|
||||
/* Deep recovery: tear down and re-create the dw_mmc host that owns our card.
|
||||
*
|
||||
* This is the ONLY thing measured to clear a hard SDIO wedge. Everything
|
||||
* cheaper has been tried on a real one (reproduced on demand with tx_batch=16,
|
||||
* which the firmware rejects -- see SV6160_TX_BATCH_MAX):
|
||||
*
|
||||
* ieee80211_restart_hw no effect; on a softer wedge it made things
|
||||
* worse, 4 -> 8 resyncs and 6 -> 13 lost descs
|
||||
* mmc_hw_reset (fw_reboot) returns SUCCESS and does not help. With one
|
||||
* function probed it is just mmc_power_cycle +
|
||||
* mmc_sdio_reinit_card, and dmesg shows the bus
|
||||
* dutifully renegotiating 400kHz -> 50MHz while
|
||||
* staying wedged
|
||||
* module reload probe still fails -16, no netdev
|
||||
* unbind/bind ff380000.mmc CLEARS it -- probes, associates, 1:1 counters
|
||||
*
|
||||
* The problem is the HOST, not the card: only re-running dw_mci_probe() puts
|
||||
* the controller back in a working state. A function driver cannot call that,
|
||||
* but it can ask the driver core to re-bind its own grandparent, which is
|
||||
* exactly what the sysfs unbind/bind does.
|
||||
*
|
||||
* Runs from a STANDALONE work item on purpose. device_release_driver() on the
|
||||
* dw_mmc platform device tears down the host, the card and this very driver --
|
||||
* our .remove() runs inside this call -- so the work must not live in the
|
||||
* softc that is about to be freed, and must not touch it afterwards. It holds
|
||||
* a reference on the platform device and on this module so neither can vanish
|
||||
* mid-flight. */
|
||||
struct sv6160_host_rebind {
|
||||
struct work_struct work;
|
||||
struct device *host_dev;
|
||||
};
|
||||
|
||||
static void sv6160_host_rebind_fn(struct work_struct *work)
|
||||
{
|
||||
struct sv6160_host_rebind *rb =
|
||||
container_of(work, struct sv6160_host_rebind, work);
|
||||
struct device *dev = rb->host_dev;
|
||||
int ret;
|
||||
|
||||
dev_warn(dev, "sv6160: re-creating the SDIO host to clear a wedged bus\n");
|
||||
device_release_driver(dev); /* takes this driver down with it */
|
||||
ret = device_attach(dev);
|
||||
dev_warn(dev, "sv6160: SDIO host re-bind %s (%d)\n",
|
||||
ret > 0 ? "OK" : "FAILED", ret);
|
||||
|
||||
put_device(dev);
|
||||
kfree(rb);
|
||||
module_put(THIS_MODULE);
|
||||
}
|
||||
|
||||
/* One shot per module load, reset by a probe that succeeds. The guard has to
|
||||
* live here rather than in the softc: the wedge that most needs this is the one
|
||||
* where PROBE itself fails -16, and at that point there is no softc at all. */
|
||||
static bool sv6160_host_rebound;
|
||||
|
||||
void sv6160_rebind_note_probe_ok(void)
|
||||
{
|
||||
sv6160_host_rebound = false;
|
||||
}
|
||||
|
||||
/* Ask for the host to be re-created. Safe from any context, and safe to call
|
||||
* with no softc. Does nothing if the topology is not what we expect rather
|
||||
* than guessing at a device. */
|
||||
void sv6160_rebind_sdio_host(struct sdio_func *func)
|
||||
{
|
||||
struct sv6160_host_rebind *rb;
|
||||
struct device *host_dev;
|
||||
|
||||
if (!mod_host_rebind)
|
||||
return;
|
||||
if (!func || !func->card || !func->card->host)
|
||||
return;
|
||||
host_dev = func->card->host->parent;
|
||||
if (!host_dev)
|
||||
return;
|
||||
if (sv6160_host_rebound) {
|
||||
dev_warn(&func->dev,
|
||||
"SDIO host already re-created once - not looping on it\n");
|
||||
return;
|
||||
}
|
||||
sv6160_host_rebound = true;
|
||||
|
||||
rb = kzalloc(sizeof(*rb), GFP_ATOMIC);
|
||||
if (!rb)
|
||||
return;
|
||||
if (!try_module_get(THIS_MODULE)) {
|
||||
kfree(rb);
|
||||
return;
|
||||
}
|
||||
INIT_WORK(&rb->work, sv6160_host_rebind_fn);
|
||||
rb->host_dev = get_device(host_dev);
|
||||
queue_work(system_unbound_wq, &rb->work);
|
||||
}
|
||||
|
||||
static void sv6160_recover_work_fn(struct work_struct *work)
|
||||
{
|
||||
struct sv6160_softc *sc = container_of(work, struct sv6160_softc,
|
||||
@@ -626,6 +737,28 @@ static void sv6160_trigger_recovery(struct sv6160_softc *sc)
|
||||
{
|
||||
if (!sc->is_started || sc->bringup_active || sc->recovering)
|
||||
return;
|
||||
|
||||
/* Rolling count of how often this bus has needed rescuing. One
|
||||
* recovery is routine; a handful in a minute means restart_hw is being
|
||||
* asked to fix something it cannot, and every extra round trip is
|
||||
* another 11 s of dead radio. Escalate instead. */
|
||||
if (!sc->recover_window ||
|
||||
time_after(jiffies, sc->recover_window +
|
||||
msecs_to_jiffies(SV6160_RECOVER_WINDOW_MS))) {
|
||||
sc->recover_window = jiffies;
|
||||
sc->recover_count = 0;
|
||||
}
|
||||
sc->recover_count++;
|
||||
|
||||
if (mod_host_rebind && sc->recover_count >= mod_host_rebind) {
|
||||
dev_err(sc->dev,
|
||||
"%u bus recoveries in %u s - restart_hw is not clearing it, re-creating the SDIO host\n",
|
||||
sc->recover_count, SV6160_RECOVER_WINDOW_MS / 1000);
|
||||
sc->recover_count = 0;
|
||||
sv6160_rebind_sdio_host(sc->func);
|
||||
return;
|
||||
}
|
||||
|
||||
sc->recovering = true;
|
||||
schedule_work(&sc->recover_work);
|
||||
}
|
||||
@@ -3963,9 +4096,14 @@ static int sv6160_fw_bringup(struct sv6160_softc *sc, u32 init_channel)
|
||||
* SDIO fault it power-cycles the card (mmc_hw_reset, rk915_sdio_power_cycle)
|
||||
* and lets mac80211 restart. We mirror that here: each attempt boots the
|
||||
* firmware cold (sv6160_sdio_fw_reboot does mmc_hw_reset + sdio_enable_func
|
||||
* + device_init + firmware download — the same power-cycle that un-wedges
|
||||
* a stuck DAT0, proven equivalent to a dwmmc unbind/rebind), runs the
|
||||
* bring-up, and retries if sc->sdio_wedged got set mid-sequence. */
|
||||
* + device_init + firmware download), runs the bring-up, and retries if
|
||||
* sc->sdio_wedged got set mid-sequence.
|
||||
*
|
||||
* This recovers a merely busy card. It is NOT equivalent to a dwmmc
|
||||
* unbind/rebind, though this comment used to claim so: measured on a
|
||||
* hard wedge, fw_reboot and a module reload both leave probe failing
|
||||
* -16, while unbind/bind brings it straight back. When the retries are
|
||||
* exhausted we therefore escalate to sv6160_rebind_sdio_host(). */
|
||||
{
|
||||
for (attempt = 1; attempt <= max_attempts; attempt++) {
|
||||
sc->sdio_wedged = false;
|
||||
@@ -4111,6 +4249,11 @@ static int sv6160_fw_bringup(struct sv6160_softc *sc, u32 init_channel)
|
||||
"fw_bringup: wedged after %d attempts (ch=%u)\n",
|
||||
max_attempts, init_channel);
|
||||
sc->bringup_active = false;
|
||||
/* Every attempt above already did a full fw_reboot, i.e.
|
||||
* mmc_hw_reset + re-download. Measured: that does not
|
||||
* clear a hard wedge, and neither does reloading the
|
||||
* module. Escalate to the one thing that does. */
|
||||
sv6160_rebind_sdio_host(sc->func);
|
||||
return -EIO;
|
||||
}
|
||||
}
|
||||
@@ -6484,6 +6627,23 @@ static ssize_t sv6160_debugfs_recover_write(struct file *file,
|
||||
size_t count, loff_t *ppos)
|
||||
{
|
||||
struct sv6160_softc *sc = file_inode(file)->i_private;
|
||||
char kbuf[8] = {};
|
||||
int level = 1;
|
||||
|
||||
if (count && count < sizeof(kbuf) && !copy_from_user(kbuf, buf, count)) {
|
||||
if (kstrtoint(strim(kbuf), 0, &level))
|
||||
level = 1;
|
||||
}
|
||||
|
||||
/* 2 = the deep one: re-create the SDIO host, the only thing measured to
|
||||
* clear a hard wedge. Deliberately reachable by hand, because that is
|
||||
* how it gets exercised -- the automatic path fires from probe failure,
|
||||
* which is hard to provoke on demand. */
|
||||
if (level >= 2) {
|
||||
dev_info(sc->dev, "recover: re-creating the SDIO host via debugfs\n");
|
||||
sv6160_rebind_sdio_host(sc->func);
|
||||
return count;
|
||||
}
|
||||
|
||||
if (!sc->is_started)
|
||||
return -ENODEV;
|
||||
|
||||
@@ -317,9 +317,27 @@ static int sv6160_probe(struct sdio_func *func,
|
||||
if (ret == -ETIMEDOUT)
|
||||
dev_err(&func->dev,
|
||||
"SDIO init timed out (-110). If the system suspended with this driver unloaded, the mmc host was not resumed; power-cycle it with unbind/bind on dwmmc_rockchip, or keep the driver loaded across suspend.\n");
|
||||
/* Do NOT try to escalate to sv6160_rebind_sdio_host() here. It
|
||||
* looks like the right place and it cannot work: once the bus is
|
||||
* hard-wedged this probe is never entered at all. sdio_bus_probe()
|
||||
* calls sdio_set_block_size() before drv->probe, that CMD52 fails
|
||||
* -EBUSY, and the core logs "probe with driver sv6160 failed with
|
||||
* error -16" without ever calling us -- verified on hardware, the
|
||||
* failing reloads have no probe entry line in dmesg.
|
||||
*
|
||||
* The driver can only rescue the host while it still HOLDS the
|
||||
* device, so the escalation lives in the bring-up retry path
|
||||
* instead. A driver that has been rmmod'ed on a wedged bus is
|
||||
* already past helping itself; that state needs the unbind/bind
|
||||
* below, by hand. (It is also not something production does --
|
||||
* only a test harness reloads a wedged driver.) */
|
||||
goto err_disable;
|
||||
}
|
||||
|
||||
/* Far enough in that the host is demonstrably alive: re-arm the
|
||||
* one-shot so a later wedge can be rescued too. */
|
||||
sv6160_rebind_note_probe_ok();
|
||||
|
||||
/* Allocate mac80211 hardware struct */
|
||||
hw = ieee80211_alloc_hw(sizeof(struct sv6160_softc), &sv6160_ops);
|
||||
if (!hw) {
|
||||
|
||||
Reference in new issue
Block a user