Skip to content

feat: prepare v3.5.0 public FIXED cache and reliable benchmarks - #91

Merged
gg582 merged 7 commits into
mainfrom
release/3.4.2-measurement
Sep 14, 2026
Merged

gg582 merged 7 commits into
mainfrom
release/3.4.2-measurement

Conversation

@DPS0340

@DPS0340 DPS0340 commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

Release hold

Do not publish v3.5.0 until the separately developed CVE fixes are included and their associated issue(s) resolved. This condition applies even if this PR and CI pass. The exact dependent issue/PR identifiers still need confirmation; missing identifiers do not waive the hold.

Summary

  • Target v3.5.0, as agreed in feat: prepare v3.5.0 public FIXED cache and reliable benchmarks #91 (comment), replacing the proposed v3.4.2 release.
  • Implement ADR docs: propose safe public FIXED response cache ADR #85 with explicit PUBLIC_FIXED opt-in, owned representations, conservative admission, process-wide limits, and invalidation. Ineligible requests use ordinary dispatch.
  • Integrate docs: propose safe public FIXED response cache ADR #85/feat(ci): add opt-in raw and corrected wrk histograms #87, including dual-histogram tools and corrected, provenance-bound benchmark reporting. Keep PUBLIC_FIXED as a separate tenth case and preserve existing workload arguments.
  • Fix blocking pipeline framing and deferred exchange ownership through middleware unwind, producer completion and inline abort.
  • Transfer scheduler and job-queue allocations before publication, including the separately selected kqueue backend. Keep opaque callback arguments caller-owned.
  • Finish a coalesced response batch synchronously when shutdown is already observed, rather than parking it on a stopped reactor. This is not a global graceful-shutdown guarantee or an absolute shutdown deadline.
  • Package version 3.5.0. Applications must rebuild and relink against matching headers and library.

Verification

  • Baseline 5b128f1178df374ee5bc3db1aba8f921a6af7d68: Linux ASan/UBSan, source distribution, interop and benchmark jobs passed; macOS C1M TCP failed with a receive timeout.
  • Reproduced the macOS failure using the exact CI source archive in an isolated native build. After the shutdown-flush fix, classic/C1M × GC on/off × normal/NDEBUG TCP runs passed (8 runs).
  • Common and kqueue job-queue component regressions passed normal/NDEBUG/ASan × GC on/off (12 runs). ASan leak detection was disabled for these native component runs: no leak-proof claim.
  • The full native library passed the job handoff target (4 runs) and the TCP target (8 runs) together, with source hashes bound to the integrated candidate.
  • Real baseline UAF and scope failures are retained. The portable creator test waits for its callback before stop without relaxing creator/donor exit ordering or GC checks.
  • Python CI helper tests: 43 passed. Allocator lint and diff check passed. Actionlint/ShellCheck reports the same pre-existing workflow findings as the baseline, with no new findings.
  • Prior source/security reviews and shutdown-flush review passed. Final platform delta is independently reviewed before commit. Exact-head full CI remains required after integration.

Remaining gates

  • Keep draft until exact-head full-library, sanitizer, source archive, native platform, interop and applicable checks pass.
  • Keep the separate CVE dependency hold, even after this PR passes.
  • Verify the main merge and its checks before any eventual release. Keep existing tags unchanged.
  • After release is permitted, verify downloaded assets and remeasure the exact released version with raw artifacts.
  • This PR does not close [BUG][Plan/CWIST 3.5] Enhance bad P99.999 latency #25 or claim a P99.999 improvement. Failed warmups and older measurements retain their original source/version labels.

Roadmap: docs/roadmap-v3.5.md. Related: #25, #85, #87.

def send_group(pgid, sig):
try:
os.killpg(pgid, sig)
except ProcessLookupError:
target = os.readlink(fd)
if target.startswith('anon_inode:'):
fds[target] = fds.get(target, 0) + 1
except (FileNotFoundError, ProcessLookupError):
response = connection.getresponse()
response.read()
ready = response.status == 200
except (OSError, http.client.HTTPException):
type(first[k]) is int and type(last[k]) is int and last[k] >= first[k]
for k in first):
csw = sum(last[k] - first[k] for k in first)
except (KeyError, TypeError, ValueError):
from pathlib import Path
import subprocess
import tempfile
import unittest
@gg582

gg582 commented Sep 14, 2026

Copy link
Copy Markdown
Member

In X.Y.Z version tag, Z is for urgent patches and critical issue. How about making a roadmap for v3.5 and unify it?

@DPS0340

DPS0340 commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator Author

Agreed, @gg582. Let us target v3.5.0 instead of v3.4.2 and put the public FIXED cache, API changes, and benchmark work into one release roadmap.

The TCP, sanitizer, and CI checks must still pass before release. We will then measure the released version and publish the results. The version change does not mean that #25 is resolved.

@DPS0340 DPS0340 changed the title feat: prepare v3.4.2 public FIXED cache and reliable benchmarks feat: prepare v3.5.0 public FIXED cache and reliable benchmarks Sep 14, 2026
@DPS0340

DPS0340 commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator Author

@gg582 unlock draft when you're ready to release :)

@gg582
gg582 marked this pull request as ready for review September 14, 2026 13:15
@gg582
gg582 merged commit 739bafc into main Sep 14, 2026
14 checks passed
@gg582
gg582 deleted the release/3.4.2-measurement branch September 14, 2026 13:16
gg582 added a commit that referenced this pull request Sep 14, 2026
… GC scope

Found while running the full test suite before cutting v3.5: the
already-merged #109 (cwist_realloc() full-GC tracking fix) made
cwist_realloc(NULL, ...) auto-track under full-GC, matching
cwist_alloc()'s behavior -- correct in general, but it exposed a real
latent bug in scheduler.c's heap_grow(), which relies on cwist_realloc()
to grow s->heap from its initial NULL.

s->heap is owned by the cwist_scheduler_t for its whole lifetime, not by
whichever caller thread happens to trigger a resize. Left tracked on
that thread's full-GC pending-sweep list, the thread exiting would have
the sweep free the scheduler's own live heap array out from under it --
a real use-after-free, just waiting on a thread-exit timing window
rather than a double-free like #109's bug.

test_gc_job_handoff.c's own comment documented the old assumption
explicitly: 'allocated by cwist_realloc, which is deliberately not
scope-tracked' -- that was only ever true by accident, because the
pre-#109 cwist_realloc() never tracked anything at all. Disown the
grown heap immediately after cwist_realloc() returns it (matching the
same handoff pattern used everywhere else this session), and update
the stale comment. Reproduced the failure (actual=2 expected=1, a
double-count from the same array being independently freed and
allocated to by the sweep) at bare 739bafc (the #91 merge point,
before any of this branch's other cherry-picks) to confirm this predates
and is independent of them. 5/5 clean runs after the fix.
gg582 added a commit that referenced this pull request Sep 14, 2026
… GC scope

Same fix as main's f2404f4, minus the test (dev doesn't have
tests/test_gc_job_handoff.c -- that's specific to #91's public-FIXED-cache
work, main-only). The underlying bug is independent of that feature: dev
already has #109's cwist_realloc() full-GC tracking fix, so
heap_grow()'s cwist_realloc(NULL, ...) call has had this exact same
latent use-after-free risk since #109 landed here too.

Cherry-pick also needed a fixup: the #include <cwist/core/mem/gc.h> line
present in main's scheduler.c wasn't in dev's version yet, so
cwist_full_gc_enabled()/cwist_gc_scope_disown() weren't declared -- added
it. Rebuilt clean; test_scheduler passes, malloc-baseline gate passes
unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG][Plan/CWIST 3.5] Enhance bad P99.999 latency

2 participants