Skip to content

Openheimer, split: land the boundary fixes, drop the hot-path guards, write OH-11 - #38

Merged
Ttimmahlax merged 3 commits into
mainfrom
openheimer-real
Sep 16, 2026
Merged

Ttimmahlax merged 3 commits into
mainfrom
openheimer-real

Conversation

@Ttimmahlax

Copy link
Copy Markdown
Contributor

What this is

The Openheimer campaign left ~1,400 uncommitted lines and a log of 202 findings against this crate. This PR is that tree split: every check on a value a caller can choose is kept; every guard on a contract this crate's own callers already uphold is dropped; and the one fix the log claimed but never wrote (OH-11) is written.

Landed

  • Checked arithmetic on sizes / alignments / offsets / option values that wrapped or debug-panicked, all on cold paths (huge_alloc, the aligned malloc/realloc family, guarded alloc, os::page_align_up / alloc_aligned, bin_size, page_under_utilized, option scaling and clamping, reserve_huge_os_pages, prim::fixed sizing, the segment-map window walk, subprocess ids, is_aligned_to(x, 0)).
  • manage_os_memory refuses null / wrapped / unmapped / already-registered ranges (prim::range_is_reserved asks the OS); arena_register claims its slot with a CAS and frees on refuse.
  • Hook re-entry guards; debug_checks guard is an assert! (it was compiled out of the build the feature exists for).
  • Thread exit abandons every heap the dying thread owns, and create_heap bootstraps the thread first.
  • C ABI: out-pointers refuse null / misaligned, dupenv clears outs on EINVAL, realpath bounded by PATH_MAX, heap handles shape-checked once in heap_ptr, mi_heap_set_default refuses a bad handle.

Withdrawn (139 of the 202 IDs)

Null / 0x1 / floor+8 probes against internal unsafe fns, chased up a ladder of six address floors and closed with a segment-map membership lookup on every internal handle plus a heap-registry walk under a global lock on the free path. At the campaign's stopping point that cost free +27, realloc +164, page_extend +68 instructions, never measured. page.rs is reverted whole; segment.rs, heap.rs, alloc.rs, segment_map.rs, lib.rs are rebuilt from main plus only the boundary hunks. Finding-by-finding: docs/plans/openheimer-run.md §4.

OH-11, written for real

A remote double free links the block onto xthread_free twice, so the chain is cyclic, and the pre-existing "n > used" check after the collect walk was unreachable: the walk never returned. remote_free now refuses a block that is already the chain head, and the collect walk is bounded by used. Getting the push-time check onto free at zero cost took three tries (a diverging callee pins a frame to free on Windows x64); the shape that works and why is documented at page::remote_double_free.

Cost, measured

Release assembly per symbol, main → branch, Windows / Linux x86-64:

symbol Windows Linux
alloc::free 62 → 66 63 → 67
alloc::malloc, malloc_aligned_at, page_extend, usable_size 0 0
alloc::realloc +7 +6
alloc::realloc_aligned_at +1 +1

The free delta is two cmp; je pairs on the cross-thread arm; the local path is byte-identical and the function keeps no frame.

Verification

  • tests/double_free.rs +4 child-process regressions, green under default / secure / blockmap / both.
  • tests/openheimer.rs keeps 37 of the campaign's 95; the FFI pack keeps 8 of 16.
  • Full suite green at default and ra_small_profile; gate-selftest 11/11 (one anchor re-based); wasm-size 20169 → 20321 gzipped (baseline updated); unsafe census 909 → 928, every site rowed in UNSAFE.md; fmt and clippy -D warnings clean.

Second commit banks the fuzz corpus the campaign's local runs produced (tracked by precedent).

🤖 Generated with Claude Code

tim-almond-house and others added 3 commits September 16, 2026 05:12
… write OH-11

The campaign left ~1,400 uncommitted lines and a log of 202 findings. This
keeps every check on a value a caller can choose and drops every guard on a
contract this crate's own callers already uphold.

Landed (cold paths, or zero cost on the local free path):
- checked arithmetic on sizes / alignments / offsets / option values that
  wrapped or debug-panicked: huge_alloc, malloc_aligned_at, realloc_aligned_at,
  guarded alloc, os::page_align_up / alloc_aligned, bin_size,
  page_under_utilized, options get_size / get_clamp, reserve_huge_os_pages,
  prim::fixed sizing, segment_map window walk, subproc ids, is_aligned_to(x, 0)
- manage_os_memory refuses null / wrapped / unmapped / already-registered
  ranges (prim::range_is_reserved: mincore / VirtualQuery); arena_register
  claims its slot with a CAS and frees the descriptor + mapping on refuse
- hook re-entry guards (deferred-free / error / output)
- debug_checks foreign-pointer guard is an assert!, not a debug_assert!
  compiled out of the build the feature exists for
- thread exit abandons every heap the dying thread owns, and create_heap
  bootstraps the thread first (first-class heaps used to stay DELAYED under
  a dead owner forever)
- C ABI: dupenv clears outs on EINVAL, realpath bounded by PATH_MAX,
  out-pointers refuse null / misaligned, heap handles are shape-checked once
  in heap_ptr, mi_heap_set_default refuses a bad handle

Withdrawn (139 of the 202 IDs): null / 0x1 / floor+8 probes against internal
unsafe fns, chased up a ladder of six address floors and closed with a
segment-map membership lookup on every internal handle plus a heap-registry
walk under a global lock on the free path. At the campaign's stopping point
that cost free +27, realloc +164, page_extend +68 instructions, unmeasured.
page.rs is reverted whole; segment.rs, heap.rs, alloc.rs, segment_map.rs
and lib.rs are rebuilt from main plus only the boundary hunks.

OH-11, written for real: the log marked it fixed and the fix did not exist
(three of its regressions failed, exit 97). A remote double free links the
block onto xthread_free twice, so the chain is CYCLIC and the pre-existing
"n > used" check after the collect walk could never be reached — the walk
never returned. remote_free now refuses a block that is already the chain
head, and the collect walk is bounded by used. Getting the push-time check
onto free for free: a diverging callee is call+ud2, and one call pins a
frame to the top of free on Windows x64 (no shrink-wrap under SEH) — +2 on
every local free. A may-return cold callee in tail position is a je; see
page::remote_double_free.

Measured (release asm, per symbol, main -> branch, Windows / Linux x86-64):
free 62->66 / 63->67 (two cmp;je on the cross-thread arm, local path
byte-identical, no frame); malloc 0/0; malloc_aligned_at 0/0; realloc +7/+6;
realloc_aligned_at +1/+1 (was +43: is_power_of_two() emitted as a SWAR
popcount without popcnt — is_aligned_to now spells the test out);
page_extend 0/0; usable_size 0/0.

Tests: double_free.rs +4 child-process regressions (abandoned page,
set_default_heap+exit, first-class heap+exit, interleaved A-B-A), green under
default / secure / blockmap / both. openheimer.rs keeps 37 of the campaign's
95 (every lie-input test went with its guard); the FFI pack keeps 8 of 16.
Full suite green at default and ra_small_profile; gate-selftest 11/11 (one
anchor re-based); wasm-size 20169 -> 20321 gzipped (baseline updated);
unsafe census 909 -> 928, every site rowed in UNSAFE.md; clippy -D warnings.

Disposition, finding by finding: docs/plans/openheimer-run.md §4. The
campaign's own log and fleet row carry a matching header.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Inputs the campaign's local fuzz runs produced; tracked by the same precedent as the 1,332 already in the tree.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The branch was lint-clean on Windows and red on every Ubuntu job at the
same three lines, none of which compile here: the mincore block in
prim/unix.rs had no SAFETY line; align_up_checked is only called by the
Windows backend, so it is dead code everywhere else (now cfg-gated with
the backend); and the Miri-only mock backend used an edition-2024-illegal
reference pattern inside an implicitly-borrowing closure pattern.

Re-checked on x86_64-unknown-linux-gnu: clippy --all-features
--all-targets, clippy at ra_small_profile, and cargo check with --cfg miri
(the mock backend), all clean, plus Windows all-features clippy and fmt.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Ttimmahlax
Ttimmahlax merged commit c9631f4 into main Sep 16, 2026
10 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants