Skip to content

tests: add real userfaultfd integration suite and stability target for the UFFD service (WIP) - #8

Draft
bergwolf with Copilot wants to merge 8 commits into
v3from
copilot/add-uffd-test-coverage
Draft

tests: add real userfaultfd integration suite and stability target for the UFFD service (WIP)#8
bergwolf with Copilot wants to merge 8 commits into
v3from
copilot/add-uffd-test-coverage

Conversation

Copilot AI commented Aug 13, 2026 •
edited by bergwolf
Loading

Copy link
Copy Markdown

The existing make test-uffd was a stateless socket smoke not wired into CI, with no real page-fault coverage. This PR adds a layered UFFD gate — capability preflight, real userfaultfd fault-completion integration, protocol negative corpus, backend-failure and lifecycle cases — plus a stability target, as the foundation for a required PR check and a daily scheduled workflow.

New: tests/e2e/uffd_fault_test.go

A minimal in-test client that creates an anonymous VMA + real userfaultfd, hands the FD to nydus uffd via HANDSHAKE, and verifies fault completion end to end:

  • Capability preflight (FR-02) — TestUffdCapabilityPreflight checks the syscall, UFFDIO_API, MISSING-mode UFFDIO_REGISTER and the COPY/ZEROPAGE/WAKE ioctls. NYDUSFS_UFFD_REQUIRE=1 (set by CI) turns missing capability into a hard unsupported-runner failure — no silent skip.
  • Real fault completion (FR-03, FR-05, NFR-01) — TestUffdRealFaultCopy (service-side UFFDIO_COPY; cold/warm byte-exact comparison against a FETCH-derived reference, EROFS superblock check, tail zero pages, loop-mounted whole-tree corpus diff), TestUffdRealFaultZeropage (/dev/zero tail hole → UFFDIO_ZEROPAGE), TestUffdRealFaultZerocopy (client maps RANGE_RESPONSE FDs with MAP_FIXED and issues UFFDIO_WAKE). Every fault has a 10s deadline; every case a 30s teardown deadline with ordered cleanup (socket → uffd → touchers → munmap) so a stuck fault can never SIGSEGV or hang the binary (FR-06).
  • Prefault + multi-batch framing (FR-04, C-02) — TestUffdPrefaultAndBatching builds a 20-blob merged image forcing >16 ranges, asserting ≥2 NEXT-linked RANGE_RESPONSE batches with correct FD counts and contents.
  • Negative corpus (FR-07) — 13 fail-closed cases (bad magic, oversized/truncated payload, unsupported version, wrong handshake FD count, malformed regions, unexpected FDs, zero-length/unaligned/out-of-bounds/overflowing FETCH) plus v1 tolerance cases (unknown msg type / header flag bits ignored). Each asserts connection termination without a success-shaped response and that the service survives.
  • Backend failure (FR-08) — truncated/removed blobs: FETCH fails closed within deadline, zero wrong bytes, unverified ranges never marked ready.
  • Lifecycle (FR-09, C-03) — client disconnect, SIGTERM with active session (clean exit, socket removed), stale-socket restart with a fresh working session.
  • Failure evidence (FR-14) — on failure, writes meta.json (case ID, commit SHA, kernel, capability state, service exit status, FD count) plus service logs to NYDUSFS_UFFD_EVIDENCE_DIR for artifact upload.

Changed: tests/e2e/uffd_test.go

  • startUffdService returns a uffdServiceProc (cmd + captured output) for diagnostics and evidence collection.
  • readUffdRangeResponses now reports the batch count.
  • Smoke test additionally preads the first FETCH-returned FD and asserts the EROFS superblock magic — FD content, not just count.

Changed: Makefile

  • test-uffd runs the whole ^TestUffd suite (was ^TestUffdServiceSmoke$ only) with UFFD_REQUIRE / UFFD_EVIDENCE_DIR passthrough; timeout raised to 600s.
  • New test-uffd-stability target: repeats the three real-fault cases UFFD_STABILITY_COUNT (default 34) times each — 102 core fault iterations per run for the daily job (FR-11), no retries (FR-12).

Status / remaining work

Verified locally: smoke, preflight, zeropage, zerocopy, negative corpus and backend-failure all pass. Three known fixes are pending, tracked in the checklist:

  1. TestUffdRealFaultCopy/Xattrs — loop-mount diff trips on the builder-added root xattr trusted.nydus.prefetch.blobs.
  2. TestUffdPrefaultAndBatching — nydus merge requires sha256-hex source filenames; layer bootstraps need renaming.
  3. TestUffdLifecycle/DisconnectWithOutstandingFault — an unresolved in-process userfaultfd fault starves Go runtime timers (confirmed with a standalone repro); the outstanding-fault case must move to a child process. Currently hangs and must be reworked before the gate is enabled.

Still to add: the uffd-tests job in ci.yml (FR-01 required check), the scheduled uffd-stability.yml workflow (FR-11/FR-12), and the docs/uffd-ci.md runbook covering the unsupported-runner procedure and the component-only evidence boundary vs internal RunD E2E (FR-15).

@gaius-qi
refactor: replace anyhow with typed error handling across nydus codebase
4168e2e 
Introduce a project-wide `Error` type in `nydus-core` using `thiserror`,
replacing `anyhow` throughout the CLI, library, and service modules. All
public APIs now return domain-specific error variants instead of opaque
`anyhow::Error`, enabling callers to match on error kinds. Go-side error
handling in nydusify is also updated accordingly.

Signed-off-by: Gaius <gaius.qi@gmail.com>
@gaius-qi
fix(error): change Display to show only outermost layer; add report()…
37ec35c 
… for full chain

`Context` variant's `Display` previously formatted the entire cause chain
inline, violating the std convention that an error exposes its cause
through either `Display` or `source()` but not both.

`Display` now prints only the outermost context string. A new `report()`
adapter walks `source()` and joins every layer with `: `, matching the
old `Display` output. All log and print call-sites that need the full
chain have been updated to use `err.report()`.

The `is_client_disconnect` text scan is updated to call `err.report()`
so protocol-level error messages nested inside a `Context` wrapper are
still detected.

Signed-off-by: Gaius <gaius.qi@gmail.com>
@gaius-qi
feat: introduce nydus workspace with split crates for format, storage…
3e97a0f 
…, backend, and build

Extract and introduce dedicated crates (nydus-error, nydus-format, nydus-backend,
nydus-storage, nydus-core, nydus-config, nydus-telemetry) covering EROFS on-disk
format structs, blob metadata with chunk/group tables and CRC32, local and registry
blob backends with HTTP proxy and Dragonfly SDK support, a group-level deduplicated
local blob cache with OFD locking and mmap-backed readiness tracking, a blob prefetch
worker, Prometheus metrics, and an EROFS bootstrap builder with merge, layout
allocation, and multi-block-device image writing. Add integration tests for ErofsReader
and NydusCore public APIs.

Signed-off-by: Gaius <gaius.qi@gmail.com>
@gaius-qi
refactor: rename ChunkAddr to ErofsChunkAddr and add architecture…
7aedbc5 
… docs

Rename `ChunkAddr` to `ErofsChunkAddr` for naming consistency with other
EROFS-prefixed types, updating all usages across the codebase. Also add
an ARCHITECTURE.md describing the overall system design.

Signed-off-by: Gaius <gaius.qi@gmail.com>
@gaius-qi
refactor: rename types, fields, and modules for consistency across co…
9ad45d6 
…debase

Rename `FormatError` to `Error`, `LocalBackend` to `Local`, `ProtoConn` to
`Connection`, and various fields (e.g. `config` to `options`) throughout
backend, format, build, storage, fanotify, and uffd modules. Introduce
`Coordinator`/`JobTable` structs in fanotify, `FileId` and `Range` types,
and convert free functions into methods. Update `XattrEntry` from tuple to
named-field struct. Align naming conventions consistently across all crates.

Signed-off-by: Gaius <gaius.qi@gmail.com>
@gaius-qi
chore: remove ARCHITECTURE.md
693b6ad 
Delete the architecture documentation file covering workspace layout, vocabulary map, verb conventions, and naming guidelines.

Signed-off-by: Gaius <gaius.qi@gmail.com>
@gaius-qi
docs: update source paths and API names to reflect crate reorganization
5e4b998 
Replace stale `nydus-core/src/metadata/` references with the correct
`nydus-format/src/erofs/` paths, rename on-disk structs to their new
`Erofs*` prefixed forms, update function names (`prepare_entries` →
`prepare_all`, `blob.fetch` → `blobs.fetch`, etc.), fix the metrics API
to `nydus_telemetry::metrics::snapshot()`, and add a Crate Architecture
section to nydus.md that documents the eight-crate workspace, the
data-plane/control-plane split, and the dependency graph.

Signed-off-by: Gaius <gaius.qi@gmail.com>
Copilot AI requested a review from bergwolf August 14, 2026 07:33
@bergwolf
bergwolf force-pushed the copilot/add-uffd-test-coverage branch from d85db89 to 9c7bda1 Compare August 14, 2026 08:18
@bergwolf
tests: add comprehensive userfaultfd integration test suite
d5e7813 
This commit introduces a layered UFFD CI gate with eight test cases that
validate the nydus UFFD service against real userfaultfd instances:

- C1: Runner capability preflight that fails closed when
  NYDUSFS_UFFD_REQUIRE is set
- C2: Managed-copy fault completion with cold/warm byte equality, EROFS
  superblock check, tail zero pages, and loop-mounted EROFS whole-tree diff
- C3: Managed zero-page completion for the device tail hole backed by
  /dev/zero
- C4: Zerocopy fault completion with RANGE_RESPONSE FDs, fixed mappings,
  and UFFDIO_WAKE
- C5: Handshake prefault plus multi-batch RANGE_RESPONSE coverage on a
  merged multi-blob image
- C6: Malformed/unsupported input fails closed without hanging
- C7: Backend corruption fails closed with zero wrong bytes
- C8: Disconnect/stop/restart with outstanding faults, bounded cleanup,
  no stale socket reuse

Every fault has an explicit deadline; no case may hang the suite. When a
case fails and NYDUSFS_UFFD_EVIDENCE_DIR is set, structured diagnostic
evidence is written for CI to upload.

Add CI workflows for the UFFD test suite and update the Makefile with
necessary test infrastructure.

Signed-off-by: Peng Tao <bergwolf@hyper.sh>
@bergwolf
bergwolf force-pushed the copilot/add-uffd-test-coverage branch from 9c7bda1 to d5e7813 Compare August 14, 2026 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bergwolf bergwolf Awaiting requested review from bergwolf

Assignees

@bergwolf bergwolf

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@bergwolf @gaius-qi