Skip to content

Dispatch write, writev, and readv on netlink fds - #305

Merged
jserv merged 1 commit into
sysprog21:mainfrom
Xalestar:netlink-write
Aug 17, 2026
Merged

Dispatch write, writev, and readv on netlink fds#305
jserv merged 1 commit into
sysprog21:mainfrom
Xalestar:netlink-write

Conversation

@Xalestar

@Xalestar Xalestar commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

busybox ip addr under elfuse exits 1 on its first rtnetlink request:

$ build/elfuse --sysroot externals/test-fixtures/rootfs \
      externals/test-fixtures/rootfs/bin/busybox ip addr
ip: write error: Bad file descriptor

The verbose trace has the socket, the bind and the getsockname reaching the netlink handlers, and the request itself failing:

syscall 198 socket(16, 3, 0)    -> 3
syscall 200 bind(3, ...)        -> 0
syscall 204 getsockname(3, ...) -> 0
syscall 64  write(3, buf, 0x14) -> -9

netlink_socket() registers pipefd[0] as the host fd so that poll and epoll have something to wait on. sys_write() and sys_writev() carry no FD_NETLINK case, so the request reaches that pipe end, which is open read-only, and macOS returns EBADF. sys_readv() has the same gap for iovcnt > 1; a single entry already delegates to sys_read().

Linux accepts write() on a netlink socket as sendto() with no explicit destination, and busybox's libiproute sends every request that way. glibc getifaddrs() uses sendto and recvmsg, both already routed, which is why the existing coverage passes over the gap.

Scope: all eight spellings (sendto, sendmsg, write, writev, recvfrom, recvmsg, read, readv) reach one gather and one scatter, so a request split across iov entries is not truncated, a receive is not capped by its first entry, and nl_complete_span() bounds all four receive spellings alike.

tests/test-netlink.c drives each of the eight against a NETLINK_ROUTE socket and then the end-to-end getifaddrs() path that originally regressed. Every receive polls before it reads, so a regression fails an assertion instead of blocking the run, and the assertions hold for a real kernel too and run in the qemu column of the matrix.

Verification

Verified on macOS 26.6.1 (build 25G76), SDK 26.5, Apple M5 Pro, on a branch based on upstream/main at 3541d18.

make test-matrix stays within baseline in all three modes with zero failures: elfuse-aarch64 252 passed, qemu-aarch64 231 passed, elfuse-x86_64 78 passed. test-netlink is OK in the qemu-aarch64 column as well as the elfuse-aarch64 one, so all 22 assertions hold against a real 6.18 kernel.

make lint, .ci/check-format.sh, .ci/check-newline.sh, .ci/check-security.sh and .ci/check-matrix-lists.sh pass. .ci/check-cppcheck.sh needs mapfile, so it is left to the CI runner rather than this host's bash 3.2.

make check passes the unit suite and reports 81 of 84 in the BusyBox smoke, with one failure that is this host and not the branch: nslookup times out because no DNS server is reachable from here, and it fails the same way with the branch's changes backed out.

The new assertions fail without the change rather than passing vacuously. Against the branch's own sources backed out, the four that cover sendmsg and recvmsg:

FAIL: sendmsg() gathers a split request (errno=22 Invalid argument)
FAIL: recvmsg() fills past the first entry (errno=22 Invalid argument)
FAIL: the scattered recvmsg() bytes parse as RTM_NEWLINK (errno=22 Invalid argument)
FAIL: sendmsg() of an empty iovec fails with ENODATA (errno=22 Invalid argument)

With only the recvmsg scatter removed, so the two receive assertions stand on their own rather than on the send above them:

FAIL: recvmsg() fills past the first entry (errno=61 No data available)
FAIL: the scattered recvmsg() bytes parse as RTM_NEWLINK (errno=61 No data available)

Removing the sys_write and sys_writev dispatch:

FAIL: write(RTM_GETLINK) accepts the request (errno=9 Bad file descriptor)
FAIL: write() request produces an RTM_NEWLINK dump (errno=9 Bad file descriptor)
FAIL: write() of zero bytes fails with ENODATA (errno=0 Success)
FAIL: writev() of zero bytes returns 0 (errno=9 Bad file descriptor)
FAIL: writev(RTM_GETLINK) accepts a split request (errno=9 Bad file descriptor)
FAIL: writev() gathers the split request into one dump (errno=9 Bad file descriptor)

Removing the sys_readv dispatch leaves the readv drain empty, and the recvmsg pair after it fails on the buffer that drain would have emptied:

FAIL: readv() fills past the first entry (errno=61 No data available)
FAIL: the scattered readv() bytes parse as RTM_NEWLINK (errno=61 No data available)
FAIL: recvmsg() fills past the first entry (errno=61 No data available)
FAIL: the scattered recvmsg() bytes parse as RTM_NEWLINK (errno=61 No data available)

cubic-dev-ai[bot]

This comment was marked as resolved.

jserv

This comment was marked as resolved.

@jserv
jserv requested a review from henrybear327 August 17, 2026 07:38
@Xalestar
Xalestar force-pushed the netlink-write branch 2 times, most recently from 6ba89db to 105a246 Compare August 17, 2026 09:43
A netlink guest fd holds the read end of the readiness pipe as its host
fd. write() and writev() on it fall through to that pipe end, which is
open read-only, and report EBADF. readv() with more than one entry reads
the same pipe instead of the response buffer and returns nothing.

Linux accepts write() on a netlink socket as sendto() with no explicit
destination, and busybox ip sends every rtnetlink request that way, so
"busybox ip addr" dies on its first dump request with "ip: write error:
Bad file descriptor". glibc getifaddrs() uses sendto and recvmsg, both
already routed, so the gap only shows on a sender that takes the write()
spelling.

One request, and one response, spans the whole iovec.
netlink_send_iov() gathers every entry into the request it parses and
netlink_recv_iov() fills every entry from the response, so all eight
spellings agree: a request split across entries is not truncated, a
first entry too small for the reply does not cap the read, and
nl_complete_span() bounds every receive to whole messages rather than
only the two that spell it recv.

The gather reads every entry before it reports any, so the count it
returns is bytes that were validated and parsed, and an unmapped entry
anywhere in the vector is EFAULT. A request past the staging buffer is
refused with EMSGSIZE, the errno Linux reports past sk_sndbuf; that
ceiling is the emulation's own and sits well past any request
nl_process_request() answers.

The receive counts what landed rather than whole entries:
guest_write_partial() reports the bytes a faulting copy placed, and
those bytes are in guest memory whatever the caller returns, so done and
buf_pos stay exact to the byte.

Four lengths behave differently, each measured against Linux 6.18 under
qemu-aarch64 rather than derived:

  write(fd, buf, 0)     -ENODATA. netlink_sendmsg refuses an empty
                        message before it builds an skb.
  writev(fd, empty, n)  0. do_readv_writev returns on a zero total
                        before the socket is reached.
  sendmsg(fd, empty)    -ENODATA. ___sys_sendmsg hands the empty vector
                        to the socket instead of answering it above.
  1 to 15 bytes         the byte count, nothing parsed.
                        netlink_rcv_skb enters its loop only from
                        nlmsg_total_size(0) bytes up.

The netlink check in sys_writev sits ahead of the single-entry shortcut
so the second row holds for a one-entry vector too.

Verified: tests/test-netlink.c passes unchanged against elfuse and
against the qemu-aarch64 kernel, and each assertion fails when the
dispatch it covers is removed.
@jserv
jserv merged commit 4d69626 into sysprog21:main Aug 17, 2026
14 checks passed
@Xalestar
Xalestar deleted the netlink-write branch August 17, 2026 10:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants