hal: don't free a funct entry the running thread may still use (delf/unloadrt crash)

[yurc]

Symptom

delf of a function from a running thread, followed by unloadrt of its component, kills rtapi_app (halcmd: recv_result 1 failed: Connection reset by peer on unloadrt) or leaves the thread spinning. With a function that takes a noticeable share of the period it happens within the first few cycles.

Cause

The realtime thread walks thread->funct_list without the HAL mutex (thread_task()).

• hal_del_funct_from_thread() and free_funct_struct() (unloadrt) unlink the entry with list_remove_entry(), which points the removed entry's next/prev at itself (src/hal/hal_lib.c:2757). A thread standing on that entry keeps calling it.
• The entry is returned to the free list at once (free_funct_entry_struct()), and unloadrt then dlclose()s the module (src/rtapi/uspace_rtapi_main.cc:630) while the thread may still execute its code.

Fix

• funct_entry_unlink(): take the entry out of the list but keep the entry's own links, so a thread standing on it continues to the rest of the list.
• thread_wait_quiescent(): with the mutex held, wait until the thread completed two more passes before the entry is freed. The pass counter is the existing beatcnt, now published with a release store after the list walk.
• Threads not running: no change.
• Thread does not complete a pass within 1000 periods: the entry is leaked instead of recycled, an error is printed, delf returns -ETIMEDOUT.

Waiting alone is not enough: with the wait but the old list_remove_entry(), every run timed out because the thread looped on the self-linked entry.

Test

tests/hal-delf-live: a component whose function busy-waits 20 µs in a 100 µs thread; loadrt → addf → delf → unloadrt repeated DELF_ITER times (default 200) with threads started, then checks that t.threadbeat still advances. Only halcmd/halcompile commands are used.

Results

Built like the rip-and-test CI job (ubuntu:24.04, --with-realtime=uspace), only this test, 10 runs × 200 cycles each:

tree	failed runs
master 056f2fd + test commit	10/10 (rtapi_app dies on unloadrt, cycles 1–7)
master 056f2fd + test + fix	0/10 (2000 cycles, thread beating afterwards)
2.9 fe72abb + test (variant without threadbeat)	10/10 (unloadrt failed, cycles 1–5)

DELF_ITER=200 scripts/rip-environment runtests -v tests/hal-delf-live

Branch

Opened against master. 2.9 has the same bug (reproduced above), but the fix uses the pass counter that only exists since f9ea091 (threadbeat); a 2.9 backport needs a new field in hal_thread_t. I can prepare it if you want it in 2.9.

Why we need it

We swap HAL components in and out while the servo thread keeps running (one component's function is replaced in the same rtapi_app that runs motion and the EtherCAT master, without stopping the machine). That is where we hit this.

Limits

• uspace only; not built or run on RTAI.
• Runs were in a container without SCHED_FIFO (non-RT uspace). The race does not depend on RT scheduling, but timing will differ on an RT kernel.
• init_funct_list (initf) is not touched.

[grandixximo]

Interesting use case, just out of curiosity, what are you doing where you need swapping components with the thread running?
Ps:
test.sh fails shellcheck warning

[yurc]

2.9 backport, for reference: branch synctwin/hal-delf-quiescence-2.9 (2 commits on 2.9 fe72abb20: test 7cef18d9b, fix 8dee91099).

Differences from this PR:

• 2.9 has no pass counter, so hal_thread_t gets a private unsigned int beatcnt (no pin), incremented with a release store at the end of each pass in thread_task(). Unlink/wait/leak logic in hal_del_funct_from_thread() and free_funct_struct() is the same as here.
• Since a shared-memory struct changes, HAL_VER is bumped 0x10 -> 0x11 as the comment in hal_priv.h asks. Note 0x11 was also used on master for a while (a1c1347), so you may prefer a different value. hal_priv.h is not an installed header, so out-of-tree components are not affected.
• Test: no threadbeat pin in 2.9, so the test runs siggen.0.update in the same thread and checks its output still changes; uint param -> u32 (2.9 halcompile).

Results, same setup as above (ubuntu:24.04 container, uspace without SCHED_FIFO, --enable-werror, only this test, DELF_ITER=200, 10 runs each):

• 2.9 without the fix: 10/10 fail (unloadrt failed)
• 2.9 with the fix: 0/10 fail

If you'd rather have this in 2.9 as well, I can open a separate PR against 2.9.

[grandixximo]

@yurc did you see my text?

[yurc]

Hi Luca, sorry for the late reply, and thanks for the review! We use LinuxCNC not only for machine tools but also as the controller for automation cells: a rotary table, a conveyor, a palletizing robot, all on the same engine. Each cell's logic is an IEC 61131-3 program (SFC/ST) compiled by MATIEC into a HAL component. Motion goes through PLCopen-style blocks (MC_MoveAbsolute, MC_MoveVelocity, MC_Halt, MC_SetPosition). Cells talk to each other through PackML tags over NATS. The operator can swap a cell's program on a running station. The servo thread with the EtherCAT CiA402 drives keeps running, the drives stay in OP and hold position. So we load the new component and delf/unloadrt the old one while the thread is running. Restarting HAL just to change a program is not an option for us. Today's example: a servo rotary table following a hand gesture from a camera, with the angle coming in as a tag. That is how we hit the rtapi_app crash. The shellcheck warning (SC2034, unused loop variable) is fixed in 6af8e3b. вс, 4 окт. 2026 г. в 11:43, Luca Toniolo ***@***.***>:

…

*grandixximo* left a comment (LinuxCNC/linuxcnc#4630) <#4630 (comment)> Interesting use case, just out of curiosity, what are you doing where you need swapping components with the thread running? Ps: test.sh fails shellcheck warning — Reply to this email directly, view it on GitHub <#4630?email_source=notifications&email_token=AAU7VNU254YNFSWA25T3ROL5SIEULA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKOJXHAZDCMJYGQ4KM4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5978211848>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAU7VNRLMH5L367NARGRGMD5SIEULAVCNFSNUABDKJSXA33TNF2G64TZHMZTMNRSHEYDKO2JONZXKZJ3GU3DSNZVGUZTEMRXUF3AE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[grandixximo]

from no reply, to two in a row, nice ;-)

[grandixximo]

what's your handover sequence, do you halt motion blocks before delf, or does new comp take over nets first?

[yurc]

Thanks for the approval!

We do break-before-make, and only from a stopped node. The new component never takes over nets while the old one is still on them.

1. The swap happens only when the node is in PackML Stopped. A program swap during Execute is refused, and stopping the node first drops the enable request, so the drives go out of OP.
2. Unload: halt any in-flight motion-block player (MC_Halt on its axes) → node enable pin to 0 → delf the node's function from the servo thread → unloadrt the node comp → unload its userspace tag bridge → drop the node's nets (nets outlive unloadrt and would otherwise keep the mc-axis pins claimed).
3. Load: loadrt the new comp → net → addf → enable.

The base layer stays up the whole time: EtherCAT/cia402, the mc_axis motion blocks, the servo thread. So the swap is fast and the bus never leaves OP. Only the node's own function goes through delf, and that is exactly where the self-looped funct entry bit us.