Skip to content

fix(app): preserve inherited CPU affinity when pinning workers - #89

Merged
DPS0340 merged 2 commits into
devfrom
fix/25-inherited-worker-affinity
Sep 14, 2026
Merged

DPS0340 merged 2 commits into
devfrom
fix/25-inherited-worker-affinity

Conversation

@DPS0340

@DPS0340 DPS0340 commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Keep Linux multiprocess worker pinning inside the affinity mask inherited at startup. Related to #25; this does not close the HTTP P99.999 investigation.

The current code counts allowed CPUs and then uses worker_index % count as a CPU ID. These are not equivalent for nonzero or sparse masks. On a four-CPU Linux VM:

  • taskset -c 1,3: current workers run on CPUs 0,1.
  • taskset -c 2,3: current workers run on CPUs 0,1.
  • taskset -c 3: both current workers run on CPU 0.

A taskset mask is not a cgroup cpuset boundary: a later successful sched_setaffinity can widen it within the enclosing cpuset. The old unchecked call can therefore defeat operator CPU placement, not merely fail.

Changes

  • Read the calling thread's inherited mask and select the worker-index-th allowed CPU, wrapping by the number of set bits.
  • Report affinity lookup/set failures and leave the inherited mask in place rather than guessing another CPU.
  • Keep the Linux/GNU guard, multiprocess-only behavior, process indexing, and worker/thread defaults unchanged.
  • Add a private header, a standalone regression target in make test, and explicit header dependencies for affected builds.

Validation

  • RED: the exact old startup affinity block, wrapped for the regression harness, fails {1,3} / worker index 0.
  • GREEN: make test_worker_affinity, normal, NDEBUG, and ASan+UBSan on Linux ARM64. Seven placement cases, three failure paths, and real fork-child affinity isolation. Checks remain active under NDEBUG.
  • macOS: standalone test compiles and explicitly skips the Linux-only behavior.
  • Real HTTP integration: two worker processes / four reactor threads each, four input masks, before and after. Baseline observations above reproduce; candidate observations are exactly 0,1, 1,3, 2,3, and 3. All 256 HTTP 200 responses have the expected body. Per-task identities/masks remain stable; owned process groups are removed and port 9091 is closed after each session.
  • Independent bounded correctness/security review: no blocking findings. Added-line security scan: no matches.

The real baseline server is pinned at 65632aed0a30e8aabe1519aa9b706e19685c2b4b; its app startup code is unchanged in this PR's base fbf9638edb5acd4105e903fe517fd7ac7736e4b3. The candidate used an isolated copy of its library with all four changed translation units rebuilt (app plus latest-dev http/reactor/session_manager); source hashes and archive members were checked. The only intervening public-header delta is documentation. This is not a fresh full-tree build.

Limits / pending

  • Remote CI on 9114bed276aaad767500c6a71d7b75e149afee71: 8 checks succeeded, 1 report-publish job was skipped, and the CodeQL summary is neutral. ASan/UBSan, CodeQL analysis, HTTP/2 conformance, macOS kqueue, and benchmark jobs succeeded. The ASan/UBSan log includes the new affinity test.
  • A separate local full-suite clean build was not run. The diagnostic VM had only about 153 MB free before the focused build; prior benchmark artifacts were preserved.
  • CPU-placement correctness is repaired. The source A/B below supports a bounded planned-arrival improvement, not consistent actual-write tail improvement or a universal speedup.
  • Fixed-size cpu_set_t is retained. If the kernel mask does not fit, lookup fails safely with a diagnostic; dynamic CPU_ALLOC support is outside this patch.
  • Linux affinity is per-thread. This patch concerns the existing startup pinning point; it does not add a facility to repin arbitrary pre-existing threads.

CI follow-up

The first head failed the raw-allocator gate because app.c line numbers shifted. Commit 9114bed276aaad767500c6a71d7b75e149afee71 updates only those eight baseline locations; exact call text and entry count are unchanged. Local allocator lint, test-wiring check, and all 17 scripts/ci Python tests pass. The fresh-head CI checks listed above have now completed without failures. This does not establish a latency improvement or approve merge.

CPU-placement screen

Full results. Eight runs used the same candidate server and elapsed-clock wrk2 tool at R300000, with client CPUs0-1 fixed. Only the server mask changed from0-1 to2-3. The run order was AB BA BA AB. No BPF or perf capture ran.

  • 95,733,695 requests; no reported load-generator errors.
  • Planned-arrival P99.999: median run values7.277ms shared and4.621ms isolated. All four pairs improved; median paired ratio0.646804.
  • Actual-write P99.999: median run values3.265ms and2.260ms. Three pairs improved; one was11.56% worse.
  • Largest paired rate difference0.387324%. Raw counts, task-to-mask maps, eventfds, source hashes, and cleanup checks passed.

This supports the placement experiment, not a code-change speedup, a general SLO, or closure of #25. Old invalid clock datasets remain invalid. Worker and thread defaults remain unchanged.

Source A/B under identical requested masks

PR #89 now has a source A/B result under identical requested CPU masks.

Base fbf9638edb5acd4105e903fe517fd7ac7736e4b3 vs candidate 9114bed276aaad767500c6a71d7b75e149afee71. Both requested server CPUs 2-3 and client CPUs 0-1. Base workers actually ran on 0-1; candidate workers respected 2-3.

Both app objects were rebuilt and both binaries relinked with identical normalized arguments and common dependencies. Only app.o differs among 77 archive members. This is controlled reuse, not two full clean builds. Both Linux binaries used the same host-backed filesystem.

Eight 40-second runs, AB BA BA AB, R300000/t12/c400/-U with the same elapsed-clock wrk2. No BPF/perf.

  • 95,780,004 requests; reported errors 0; 96 calibrations. Each histogram kind contains 71,600,827 post-calibration samples.
  • Planned-arrival P99.999: medians of run quantiles 6.417ms -> 5.001ms. All 4 pairs improved; median candidate/base ratio 0.832621 (16.74% reduction), passing the frozen <=0.90 plus four-win screen.
  • Actual-write countermetric: only 2/4 improved; median paired ratio 1.077422 (7.74% worse). Individual ratios 1.170819,0.828221,0.984026,1.279883. There is no consistent actual-write tail improvement.
  • Paired RPS difference at most 0.191587%.

Parent raw/source/identity-to-mask/eventfd checks passed. Postflight verified 113 files, no owned survivors, and a closed port. Independent bounded static review found no blocker.

This supports the affinity correctness fix and the planned-arrival result in this configured workload, NOT a universal server-latency improvement. Default CPU placement, the original 64/256/512 matrix, and overall issue #25 remain unresolved. No merge or issue closure is requested by this result.

@DPS0340
DPS0340 marked this pull request as ready for review September 13, 2026 18:40
@DPS0340
DPS0340 merged commit ecb157a into dev Sep 14, 2026
10 checks passed
@gg582
gg582 deleted the fix/25-inherited-worker-affinity branch September 14, 2026 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant