Skip to content

xtensa/espressif: fix lock-order deadlock in esp_wifi_event_handler() - #20178

Merged
xiaoxiang781216 merged 1 commit into
apache:masterfrom
FelipeMdeO:fix/esp-wifi-event-handler-netdev-lock-deadlock
Sep 18, 2026
Merged

xiaoxiang781216 merged 1 commit into
apache:masterfrom
FelipeMdeO:fix/esp-wifi-event-handler-netdev-lock-deadlock

Conversation

@FelipeMdeO

Copy link
Copy Markdown
Contributor

Summary

esp_wifi_event_handler() held esp_wifi_lock() across the entire
event switch, including the esp_wlan_*_hook() calls
(WIFI_EVENT_STA_CONNECTED/_DISCONNECTED, WIFI_EVENT_AP_START/_STOP).
Those hooks reach netdev_lower_carrier_on()/_off(), which take the
per-device netdev_lock().

Every other path into esp_wifi_lock() acquires the two locks in the
opposite order: e.g. the netdev ifdown path holds netdev_lock()
around its own call into esp_wifi_api_stop(), which calls
esp_wifi_lock(). So:

  • ifdown path: netdev_lock()esp_wifi_lock()
  • event handler (before this fix): esp_wifi_lock()netdev_lock()

An application that disconnects Wi-Fi (wpa_driver_wext_disconnect()
immediately followed by wapi_set_ifdown(), e.g.
netlib_ifdown()/wapi_set_ifdown() right after a disconnect ioctl)
races the resulting WIFI_EVENT_STA_DISCONNECTED callback against its
own ifdown call. The two lock orders wedge each other permanently — an
AB-BA deadlock, confirmed on real ESP32-S3 hardware (XIAO ESP32-S3,
CONFIG_ESPRESSIF_WIFI + CONFIG_PM + CONFIG_SCHED_TICKLESS).

Fix: esp_wifi_lock() is now taken only around the specific calls that
reach into the Wi-Fi driver API (esp_wifi_scan_event_parse(),
esp_wifi_set_ps()), never spanning a esp_wlan_*_hook() call. This
keeps netdev_lock() first (or absent) and esp_wifi_lock() last on
every path, matching the order every other caller already uses.

Impact

  • Is new feature added? No — pure bug fix.
  • Is existing feature changed? No behavior change for any config that
    worked correctly before; only removes a deadlock window that was
    always latent whenever WIFI_EVENT_STA_DISCONNECTED/_CONNECTED (or
    the AP equivalents) could race an ifdown/ifup on the same device.
  • Impact on hardware? No board/driver files touched — this is generic
    arch/xtensa/src/common/espressif/ code shared by every Xtensa
    Espressif target that enables CONFIG_ESPRESSIF_WIFI.

Testing

I confirm that changes are verified on local setup and works as intended:

  • Build Host: Ubuntu 24.04, x86_64, xtensa-esp-elf-gcc (crosstool-NG
    esp-14.2.0_20241119, 14.2.0).
  • Target: Xtensa, Seeed XIAO ESP32-S3, out-of-tree defconfig with
    CONFIG_ESPRESSIF_WIFI=y, CONFIG_PM=y, CONFIG_SCHED_TICKLESS=y,
    CONFIG_NETINIT_DHCPC=y, real Wi-Fi AP (WPA2/CCMP) + DHCP.
  • Reproduction: an application task calls wpa_driver_wext_disconnect()
    immediately followed by wapi_set_ifdown() on the STA interface,
    with no delay between the two calls.

Confirmed via JTAG (OpenOCD + GDB) that both wedged tasks were genuinely
blocked (not merely slow): the disconnecting task waited on a mutex
whose sem.val.mholder field named the kernel low-priority work-queue
thread as holder, and that work-queue thread waited on a mutex naming
the disconnecting task right back — the two lock orders above,
confirmed live in memory, not inferred from code reading alone.

Console log before the fix (ps, ~20s after the disconnecting call,
no further progress ever observed):

  TID   PID  PPID PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK            STACK COMMAND
    1     0     0 224 RR       Kthread   - Waiting  Semaphore 0000000000000000 0008104 hpwork ...
    2     0     0 100 RR       Kthread   - Waiting  Mutex:9   0000000000000000 0004016 lpwork ...
    5     0     0 253 RR       Kthread   - Waiting  MQ empty  0000000000000000 0006600 wifi
    9     9     0 100 RR       Task      - Waiting  Mutex:2   0000000000000000 0008128 collar start

(Mutex:N names the holder's TID — see
sched/sched/sched_get_stateinfo.c. TID 9 waits on a mutex held by TID
2 (lpwork); TID 2 waits on a mutex held by TID 9 — a two-task cycle.)

Console log after the fix, same reproduction, no hang, all application
worker threads alive and running:

  TID   PID  PPID PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK            STACK COMMAND
    9     9     0 100 RR       Task      - Waiting  Semaphore 0000000000000000 0008128 collar start
   10     9     0 110 RR       pthread   - Waiting  Semaphore 0000000000000000 0004064 collar_imu ...
   11     9     0 105 RR       pthread   - Waiting  Semaphore 0000000000000000 0004064 collar_sd ...
   12     9     0 100 RR       pthread   - Waiting  Semaphore 0000000000000000 0008160 collar_tx ...

Reproduced 4/4 times before the fix, 0/2 after (both clean runs reached
full application startup with the disconnect/reconnect path exercised).

@github-actions github-actions Bot added Arch: xtensa Issues related to the Xtensa architecture Size: S The size of the change in this PR is small labels Sep 17, 2026
@github-actions

github-actions Bot commented Sep 17, 2026

Copy link
Copy Markdown

MemBrowse Memory Report

No memory changes detected for:

esp_wifi_event_handler() held esp_wifi_lock() across the whole event
switch, including the esp_wlan_*_hook() calls
(WIFI_EVENT_STA_CONNECTED/_DISCONNECTED, WIFI_EVENT_AP_START/_STOP).
Those hooks reach netdev_lower_carrier_on()/_off(), which take the
per-device netdev_lock().

Every other path into esp_wifi_lock() acquires the two locks in the
opposite order -- the netdev ifdown path holds netdev_lock() around
its own call into esp_wifi_api_stop(), which calls esp_wifi_lock().
An application that disconnects Wi-Fi (wpa_driver_wext_disconnect()
immediately followed by wapi_set_ifdown()) races the resulting
WIFI_EVENT_STA_DISCONNECTED callback against its own ifdown call, and
the two lock orders wedge each other permanently.

Confirmed on real ESP32-S3 hardware (XIAO ESP32-S3,
CONFIG_ESPRESSIF_WIFI + CONFIG_PM + CONFIG_SCHED_TICKLESS): the
disconnecting task and the low-priority work-queue thread each waited
on a mutex held by the other (checked live via JTAG/GDB, not inferred
from code reading alone). Reproduced 4/4 times before this fix, 0/2
after.

Fix: esp_wifi_lock() is now taken only around the specific calls that
reach into the Wi-Fi driver API (esp_wifi_scan_event_parse(),
esp_wifi_set_ps()), never spanning a esp_wlan_*_hook() call --
netdev_lock() first (or absent), esp_wifi_lock() last, on every path.

Signed-off-by: Felipe Moura <moura.fmo@gmail.com>
Assisted-by: Claude:claude-sonnet-5
@FelipeMdeO
FelipeMdeO force-pushed the fix/esp-wifi-event-handler-netdev-lock-deadlock branch from e08ca89 to faae980 Compare September 17, 2026 17:48
@FelipeMdeO FelipeMdeO changed the title xtensa/espressif: fix esp_wifi_lock()/netdev_lock() deadlock in esp_wifi_event_handler() xtensa/espressif: fix lock-order deadlock in esp_wifi_event_handler() Sep 17, 2026
@xiaoxiang781216
xiaoxiang781216 merged commit d656cfa into apache:master Sep 18, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Arch: xtensa Issues related to the Xtensa architecture Size: S The size of the change in this PR is small

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants