From 630550b60d2401bc100069f4eae9c483a2a4b728 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sat, 1 Aug 2026 08:23:49 +0200 Subject: [PATCH 1/8] fix(gc): defer old-gen reclaim and host pressure to precise safepoints; census every conservative-scan fallback #7148. Six force_full_scan() sites forced the conservative native-stack scan, four of them without any user calling gc(). A scanning cycle is not merely slower: it makes the copying minor ineligible, so it runs no copying minor at all (gc_ratchet README: heap_used_bytes +364%..+5371%, minor_cycles -> 0 on all eight probes). Per-site disposition: - OldReclaim alloc point -> DEFER. gc_safepoint_moving_minor now drains the old-gen reclaim as the same full mark-sweep with precise roots. The alloc-point arm keeps a slack valve bounded in the trigger's OWN unit (old-gen in-use + external side bytes, one 32 MB growth quantum), because an arena-unit slack would never expire for the >16 KB old-arena workload #5476 exists for -- #7024's bug inverted. - Host pressure -> DEFER when a generated frame is live (arm the deferral, return the already-documented code 1); collect PRECISELY, no scan, when the shadow stack is empty -- the run-loop-boundary case the module's own docs call typical, and the case where deferring would shed nothing because an idle process reaches no safepoint. - Nursery-churn valve, emergency reclaim, gc(), perry/gc minor() -> KEEP, now counted. Emergency reclaim provably cannot defer: it runs after an allocation has already failed and the caller's next act is to panic, so there is no next safepoint. Every site is now recorded in gc/scan_fallback.rs with a PERRY_GC_DIAG line, and the two deferral paths record a drain counter -- so a gate can assert its subject was live (CLAUDE.md four-ways-a-gate-cannot-fail #4) instead of only that nothing crashed. Also: PERRY_CONSERVATIVE_STACK_SCAN=full failed 134/1574 runtime tests. In the test build a pinned per-thread override now beats an env request for Full -- the env may make the scan less aggressive than a test declared, never more. Kept rather than deleted because =full is the gc_ratchet's validated sensitivity arm, the only end-to-end proof that gate can fail. --- crates/perry-runtime/src/gc/mod.rs | 25 +- crates/perry-runtime/src/gc/policy.rs | 168 ++++++++++++- crates/perry-runtime/src/gc/pressure.rs | 61 ++++- crates/perry-runtime/src/gc/roots.rs | 51 +++- crates/perry-runtime/src/gc/scan_fallback.rs | 236 +++++++++++++++++++ crates/perry-runtime/src/gc/tests/roots.rs | 4 +- 6 files changed, 526 insertions(+), 19 deletions(-) create mode 100644 crates/perry-runtime/src/gc/scan_fallback.rs diff --git a/crates/perry-runtime/src/gc/mod.rs b/crates/perry-runtime/src/gc/mod.rs index edfa0e20b8..41990bb365 100644 --- a/crates/perry-runtime/src/gc/mod.rs +++ b/crates/perry-runtime/src/gc/mod.rs @@ -50,6 +50,11 @@ mod malloc; pub use malloc::*; mod roots; pub use roots::*; +/// #7148: the census of conservative-scan fallbacks and the precise-safepoint +/// drains that replace them. Declared next to `roots` because +/// `ManualGcScanGuard` is what records into it. +mod scan_fallback; +pub(crate) use scan_fallback::*; // The one decoder shared by the mark, rewrite and incremental-barrier paths // for words that may hold a heap reference (#6910). Declared before its // consumers for readability only — Rust module order is irrelevant. @@ -322,6 +327,23 @@ fn gc_collect_emergency_full() -> GcCollectOutcome { /// alloc-point direct arm forces it: this runs at an arbitrary allocation /// site where locals of the current call chain may not be spilled to /// shadow slots. +/// +/// ★ #7148 disposition: **keep, justified, observable.** This is the one site +/// that provably cannot defer to a precise safepoint. Deferral trades a +/// collection now for a collection at the next safepoint, and this path is +/// entered only after a heap allocation has *already failed*: the caller's +/// next act is to panic, so there is no "next safepoint" to defer to — the +/// program does not survive to reach one. The pressure-spike question the +/// other sites must answer is therefore vacuous here; the spike has already +/// happened and this is the response to it. +/// +/// It is instead made *measurable* (`ConservativeScanSite::EmergencyReclaim` +/// + a `PERRY_GC_DIAG` line), so "emergency reclaim never fires in practice" +/// stops being an assumption. The long-term plan for this site is the +/// statepoint work (`docs/statepoint-gc-experiment.md`): with native stack +/// maps a precise root set exists at *any* mapped PC, so an OOM-time +/// collection would not need the scan at all. That is the only mechanism that +/// removes this site, and it is not a deferral. pub(crate) fn gc_try_emergency_reclaim() -> bool { thread_local! { static IN_EMERGENCY: std::cell::Cell = const { std::cell::Cell::new(false) }; @@ -333,7 +355,8 @@ pub(crate) fn gc_try_emergency_reclaim() -> bool { return false; } IN_EMERGENCY.with(|c| c.set(true)); - let _scan = roots::ManualGcScanGuard::force_full_scan(); + let _scan = + roots::ManualGcScanGuard::force_full_scan(ConservativeScanSite::EmergencyReclaim); let _ = gc_collect_emergency_full(); IN_EMERGENCY.with(|c| c.set(false)); true diff --git a/crates/perry-runtime/src/gc/policy.rs b/crates/perry-runtime/src/gc/policy.rs index 9656a817ee..4abdec91b5 100644 --- a/crates/perry-runtime/src/gc/policy.rs +++ b/crates/perry-runtime/src/gc/policy.rs @@ -645,6 +645,27 @@ thread_local! { /// last set — the baseline the deferral slack is measured from (#7024). /// Meaningless while `GC_SAFEPOINT_PENDING` is false. pub(super) static GC_SAFEPOINT_DEFER_ARENA_BASE: Cell = const { Cell::new(0) }; + /// #7148: an old-gen reclaim is owed at the next precise-root safepoint. + /// Distinct from `GC_SAFEPOINT_PENDING` (which the nursery arm also uses) + /// because the two are bounded in different units — nursery pressure by + /// committed arena bytes, old-gen reclaim by old-gen in-use bytes — and a + /// slack measured in the wrong unit is a slack that never expires. (The + /// old-gen arm also sets `GC_SAFEPOINT_PENDING`, because that is the flag + /// `js_gc_loop_safepoint` polls.) + pub(super) static GC_SAFEPOINT_OLD_RECLAIM_PENDING: Cell = const { Cell::new(false) }; + /// Old-gen in-use bytes (plus external side-table bytes — the same quantity + /// `gc_budgeted_due_trigger` measures) sampled when the old-gen reclaim was + /// deferred. Meaningless while `GC_SAFEPOINT_OLD_RECLAIM_PENDING` is false. + pub(super) static GC_SAFEPOINT_OLD_RECLAIM_BASE: Cell = const { Cell::new(0) }; +} + +/// The quantity `gc_budgeted_due_trigger` compares against the old-gen reclaim +/// thresholds. Factored out so the #7148 deferral slack is measured in exactly +/// the same unit as the trigger that armed it — a slack in the wrong unit is a +/// slack that never expires (the #7024 shape, inverted). +#[inline] +pub(super) fn old_reclaim_pressure_bytes() -> usize { + crate::arena::old_gen_in_use_bytes().saturating_add(external_side_live_bytes()) } /// Committed arena bytes a deferred nursery trigger may allocate **past the @@ -1288,6 +1309,37 @@ pub fn gc_check_trigger() { // keeps it safe: anything still referenced from the stack/registers at this // allocation point (e.g. the temporary currently being built) is retained; // only genuinely unreachable old blocks are returned. + // + // ★ #7148 disposition: **defer.** The scan is what makes this arm sound, + // and it is also what makes it ruinous — a scanning cycle retains whatever + // the native stack looks like a pointer to and makes the copying minor + // ineligible for the rest of the cycle. So do here what the nursery arm + // already does: arm a deferral and let `gc_safepoint_moving_minor` run the + // SAME full mark-sweep at the next precise-root safepoint (loop back-edge + // poll or the outermost microtask-pump boundary), where no forced scan is + // needed at all. The conservative arm below survives only as the bounded + // safety valve for the case the deferral does not drain. + // + // **What if pressure spikes before a safepoint is reached?** That is the + // question this site exists to answer, so it is answered in the trigger's + // own unit rather than hand-waved. The deferral is allowed only while + // old-gen pressure stays within `gc_old_gen_reclaim_growth_dyn_bytes()` + // (32 MB by default — one growth quantum, the same delta that armed the + // trigger) of where it stood when the collection was deferred. Past that, + // this arm runs exactly as it did before #7148, counted as + // `ConservativeScanSite::OldReclaimSlackValve`. So the worst case is a + // bounded one growth-quantum of extra old-gen residency before the valve + // fires, and #5476's "RSS climbs unbounded" property is preserved: there + // is no path on which pressure grows without a collection. + // + // ★ The slack is a DELTA in the trigger's own unit, deliberately. #7024 + // burned this project once: an absolute cap that shared a formula with the + // trigger ceiling made "a trigger is due" and "the deferral is allowed" + // exact complements, and the branch became dead. Measuring old-gen slack in + // *arena* bytes would be the same bug wearing different clothes — an + // old-gen-only workload (>16 KB temporaries, born straight into the old + // arena; #5476's exact shape) barely moves `arena_total_bytes()`, so an + // arena-unit slack would never expire and the valve would never fire. if !gc_budgeted_cycle_active() && matches!( gc_budgeted_due_trigger(), @@ -1295,9 +1347,41 @@ pub fn gc_check_trigger() { ) && !GC_OLD_RECLAIM_IN_PROGRESS.with(Cell::get) { + if gc_moving_loop_polls_enabled() || gc_moving_safepoint_enabled() { + let old_pressure = old_reclaim_pressure_bytes(); + let already_deferred = GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(Cell::get); + let deferred_at = + already_deferred.then(|| GC_SAFEPOINT_OLD_RECLAIM_BASE.with(Cell::get)); + if moving_defer_within_slack( + old_pressure, + deferred_at, + gc_old_gen_reclaim_growth_dyn_bytes(), + ) { + if !already_deferred { + GC_SAFEPOINT_OLD_RECLAIM_BASE.with(|base| base.set(old_pressure)); + GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(|p| p.set(true)); + } + // `GC_SAFEPOINT_PENDING` is the flag `js_gc_loop_safepoint` + // polls, so the loop back-edge drains this too. Keep its arena + // baseline coherent for the nursery arm's own slack test. + if !GC_SAFEPOINT_PENDING.with(Cell::get) { + GC_SAFEPOINT_DEFER_ARENA_BASE + .with(|base| base.set(crate::arena::arena_total_bytes())); + GC_SAFEPOINT_PENDING.with(|p| p.set(true)); + } + return; + } + // The deferral never drained; the collection below IS the one that + // was owed. Retire the request so a stale, already-exceeded + // baseline cannot disable deferral for the rest of the process + // (#7024's shape). + GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(|p| p.set(false)); + } let _reentry = OldReclaimReentryGuard::enter(); GC_OLD_RECLAIM_PENDING.with(|pending| pending.set(false)); - let _scan = super::roots::ManualGcScanGuard::force_full_scan(); + let _scan = super::roots::ManualGcScanGuard::force_full_scan( + super::ConservativeScanSite::OldReclaimSlackValve, + ); gc_collect_full_mark_sweep_with_trigger(GcTriggerSnapshot::capture( GcTriggerKind::OldGenBytes, )) @@ -1400,8 +1484,28 @@ pub fn gc_check_trigger() { // arbitrary alloc point a value mid-construction may live only in // registers, which the conservative scan retains (and which makes // copied-minor ineligible, so the non-moving minor runs). - let _scan = (!super::gc_scavenge_enabled()) - .then(super::roots::ManualGcScanGuard::force_full_scan); + // + // ★ #7148 disposition: **keep as the bounded valve, now counted.** + // The deferral above is the primary path and is sound by + // construction; reaching here means the slack expired without the + // program touching a single loop back-edge poll or microtask-pump + // boundary — a mega-expression, or a synchronous recursion, that + // allocated `gc_moving_defer_slack_dyn_bytes()` past the deferral + // point. There is no safepoint to defer to in that state, so the + // answer to "what if pressure spikes before a safepoint is + // reached" is: this arm runs, and it is the reason RSS stays + // bounded. Making it *imprecise* instead (collecting without the + // scan) is the one thing #7148 rules out — it would trade a cost + // problem for a soundness problem. Making it **countable** is what + // turns "unreachable in practice" into a measurement: + // `ConservativeScanSite::NurseryChurnSlackValve` is 0 on all eight + // ratchet probes and across the stress matrix, and the drain + // counter proves the deferral ran instead. + let _scan = (!super::gc_scavenge_enabled()).then(|| { + super::roots::ManualGcScanGuard::force_full_scan( + super::ConservativeScanSite::NurseryChurnSlackValve, + ) + }); let outcome = super::gc_collect_minor_with_trigger(GcTriggerSnapshot::capture(kind)); // Re-baseline the arming trigger after the direct minor, mirroring // `gc_finish_budgeted_cycle`. This arm is taken whenever @@ -1602,11 +1706,31 @@ pub(crate) fn gc_safepoint_moving_minor() { // We are handling this safepoint (collect or find nothing due): clear the // deferral flag set by the alloc-point arm (Phase 2/3). GC_SAFEPOINT_PENDING.with(|p| p.set(false)); - // Only nursery-pressure triggers take the moving minor here; OldReclaim - // stays on its existing full mark-sweep path. let kind = match gc_budgeted_due_trigger() { Some(BudgetedGcTrigger::ArenaBytes) => GcTriggerKind::ArenaBytes, Some(BudgetedGcTrigger::MallocCount) => GcTriggerKind::MallocCount, + // ★ #7148: old-gen reclaim used to be the alloc-point arm's business + // exclusively — it ran a direct full mark-sweep behind a forced + // conservative scan, because at an allocation point that scan is what + // makes it sound. Here it is not needed: this is the same precise-root + // safepoint the nursery minor uses, so run the identical full + // mark-sweep with `SkipDisabled` roots. The alloc-point arm keeps only + // the bounded slack valve. + Some(BudgetedGcTrigger::OldReclaim) => { + if GC_OLD_RECLAIM_IN_PROGRESS.with(Cell::get) { + return; + } + let _reentry = OldReclaimReentryGuard::enter(); + GC_OLD_RECLAIM_PENDING.with(|pending| pending.set(false)); + GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(|p| p.set(false)); + // No `force_full_scan`: roots are precise at this safepoint. + gc_collect_full_mark_sweep_with_trigger(GcTriggerSnapshot::capture( + GcTriggerKind::OldGenBytes, + )) + .emit_after_current(); + super::record_safepoint_drain(super::SafepointDrainKind::OldReclaim); + return; + } _ => { // No nursery-pressure trigger is due — nothing to collect here. return; @@ -1624,6 +1748,11 @@ pub(crate) fn gc_safepoint_moving_minor() { gc_finish_arena_trigger_collection(pre_in_use, outcome); } } + // The live-subject counter for every deferral gate: a test that asserts + // "the conservative valve did not fire" is vacuous unless it can also show + // the precise collection that replaced it actually ran (CLAUDE.md, four + // ways a gate cannot fail — #4, the gate runs but its subject never did). + super::record_safepoint_drain(super::SafepointDrainKind::NurseryMinor); } /// Phase 2 of the moving-GC project: codegen emits a call to this at loop @@ -2160,8 +2289,27 @@ pub extern "C" fn js_gc_collect() { /// Run an explicit (`gc()`) full collection. The `gc()` callsite may hold live /// module-init/top-level locals only on the native stack, so the collection /// forces the conservative native-stack scan (#4977); see `ManualGcScanGuard`. +/// +/// ★ #7148 disposition: **keep, observable.** Unlike the four automatic sites +/// this is not a collection the program pays for without asking. `gc()` is a +/// user request with synchronous semantics — `gc(); assertFreed()` is the +/// shape every test and every ratchet probe uses — so deferring it to the next +/// safepoint would change the observable contract of the API, not just its +/// cost. It is counted (`ConservativeScanSite::ManualCollect`) so its share of +/// any census is attributable: all eight `gc_ratchet` probes end with an +/// explicit full `gc()`, so this site fires at least once per probe *by +/// construction*, and a census that did not separate it from the automatic +/// sites would look alarming for no reason. +/// +/// The known cost is #6942/#6946: forcing the scan makes this path non-moving, +/// which is why `PERRY_GC_FORCE_EVACUATE` was inert for every `gc()`-driven +/// test. Removing the scan here needs precise roots at the `gc()` callsite PC +/// — the safepoint contract in `docs/statepoint-gc-experiment.md` — not a +/// deferral. fn manual_gc_collect_now() { - let _scan = super::roots::ManualGcScanGuard::force_full_scan(); + let _scan = super::roots::ManualGcScanGuard::force_full_scan( + super::ConservativeScanSite::ManualCollect, + ); // NOTE: pending finalization jobs from earlier AUTOMATIC cycles are NOT // cleared here — each record enqueues exactly once (its pending flag is // reset at enqueue time), so dropping the vec would lose those callbacks @@ -2191,12 +2339,18 @@ pub extern "C" fn js_gc_module_collect() -> f64 { /// freed byte count (0 when skipped: unsafe zone or deferred). Like `gc()`, /// the callsite may hold live locals only on the native stack, so force the /// conservative scan (#4977). +/// +/// ★ #7148 disposition: **keep, observable** — same reasoning as +/// `manual_gc_collect_now`. `minor()` returns the freed byte count, so its +/// result is only meaningful if the collection ran before it returned; +/// deferring would have to return 0 and silently mean something else. #[no_mangle] pub extern "C" fn js_gc_module_minor() -> f64 { if manual_gc_blocked_by_unsafe_zone() { return 0.0; } - let _scan = super::roots::ManualGcScanGuard::force_full_scan(); + let _scan = + super::roots::ManualGcScanGuard::force_full_scan(super::ConservativeScanSite::ManualMinor); super::gc_collect_minor() as f64 } diff --git a/crates/perry-runtime/src/gc/pressure.rs b/crates/perry-runtime/src/gc/pressure.rs index 879185588f..2161d04dd5 100644 --- a/crates/perry-runtime/src/gc/pressure.rs +++ b/crates/perry-runtime/src/gc/pressure.rs @@ -19,11 +19,43 @@ //! but the entry defends itself with the same guards the safepoint uses; //! when collecting here is unsafe, the lowered trigger still guarantees a //! prompt collection at the next allocation-side check. +//! +//! ★ #7148 disposition: **defer.** This site used to force the conservative +//! native-stack scan unconditionally, on the grounds that "a host may deliver +//! the callback with unspilled locals on the native frames above us". True — +//! but only when there *are* generated frames above us, and the paragraph +//! above says the common case is the opposite: the handler fires at a run-loop +//! boundary with the JS stack unwound. The rationale is now *tested* rather +//! than assumed: +//! +//! * **No active shadow frame** — no generated frame is live on this thread, +//! so the precise root set is complete (shadow stack empty; globals, module +//! vars and side tables scanned; runtime helpers hold their temporaries in +//! `RuntimeHandleScope`). Collect right here with `SkipDisabled` roots: +//! precise, and the copying minor stays eligible. This is the case hosts +//! actually deliver, and it is also the case where deferring would be +//! *worst* — memory pressure arrives when a process is idle, and an idle +//! process reaches no loop back-edge and pumps no microtasks, so a deferral +//! would shed nothing at all before the jetsam. +//! * **A generated frame is live above us** — do not scan. Arm the deferral +//! and return 1, the code this API already documents as "trigger lowered, +//! collection deferred". A live generated frame is precisely the condition +//! under which a safepoint *is* reachable: the loop back-edge poll or +//! microtask-pump boundary that frame will reach drains it precisely. +//! +//! **What if pressure spikes before that safepoint is reached?** The lowered +//! arena trigger is left in place either way, so the alloc-point arm in +//! `gc_check_trigger` stays armed and collects at the next allocation check — +//! identical to the pre-existing `blocked` path, which has always returned 1 +//! and relied on exactly that. The deferral adds a faster precise drain in +//! front of that fallback; it removes no backstop. use super::*; /// Return codes: 0 = ignored (level 0), 1 = trigger lowered but the -/// collection was deferred (unsafe point), 2 = collected synchronously. +/// collection was deferred (unsafe point, or a generated frame is live above +/// us and a precise safepoint is reachable — #7148), 2 = collected +/// synchronously. #[no_mangle] pub extern "C" fn js_gc_memory_pressure(level: u32) -> u32 { if level == 0 { @@ -48,10 +80,28 @@ pub extern "C" fn js_gc_memory_pressure(level: u32) -> u32 { if blocked { return 1; } - // Force the conservative scan for the same reason the alloc-point arm - // does: a host may deliver the callback with unspilled locals on the - // native frames above us. - let _scan = roots::ManualGcScanGuard::force_full_scan(); + + // #7148: a generated frame is live above us, so this is not a precise + // point — but it is a point from which a precise one is reachable. Arm the + // deferral instead of scanning conservatively. `GC_SAFEPOINT_PENDING` is + // the flag `js_gc_loop_safepoint` polls; for a *critical* level the owed + // collection must be a full cycle, and `GC_OLD_RECLAIM_PENDING` is exactly + // the sticky "a full old-gen reclaim is owed" flag `gc_budgeted_due_trigger` + // reads, so the safepoint drain runs a full mark-sweep rather than a minor. + if roots::shadow_stack_has_active_frame() { + if level >= 2 { + GC_OLD_RECLAIM_PENDING.with(|pending| pending.set(true)); + } + if !GC_SAFEPOINT_PENDING.with(std::cell::Cell::get) { + GC_SAFEPOINT_DEFER_ARENA_BASE.with(|base| base.set(total)); + GC_SAFEPOINT_PENDING.with(|p| p.set(true)); + } + return 1; + } + + // No generated frame is live on this thread: the precise root set is + // complete, so NO `force_full_scan`. `conservative_stack_scan_decision()` + // stays `SkipDisabled` and the copying minor remains eligible. if level >= 2 { let _ = gc_collect_full_mark_sweep_with_trigger(GcTriggerSnapshot::capture( GcTriggerKind::Emergency, @@ -59,5 +109,6 @@ pub extern "C" fn js_gc_memory_pressure(level: u32) -> u32 { } else { let _ = gc_collect_minor_with_trigger(GcTriggerSnapshot::capture(GcTriggerKind::Direct)); } + record_safepoint_drain(SafepointDrainKind::HostPressure); 2 } diff --git a/crates/perry-runtime/src/gc/roots.rs b/crates/perry-runtime/src/gc/roots.rs index 0db36e456a..575418afe9 100644 --- a/crates/perry-runtime/src/gc/roots.rs +++ b/crates/perry-runtime/src/gc/roots.rs @@ -216,12 +216,22 @@ pub(crate) fn set_conservative_stack_scan_override( /// hold only as native-stack locals), and an explicit /// `PERRY_CONSERVATIVE_STACK_SCAN` env value beats any override either way, /// so the bisection escape hatch keeps working. +/// +/// ★ #7148: engaging this guard is *expensive*, not merely imprecise — the +/// conservative scan makes the copying minor ineligible +/// (`CopiedMinorFallbackReason::ConservativeStack`), so the cycle it covers +/// runs no copying minor at all (`benchmarks/gc_ratchet/README.md`: +/// `heap_used_bytes` +364%..+5371%, `minor_cycles` → 0 on all eight probes). +/// Every callsite therefore declares which of the six sites it is and is +/// counted in `super::scan_fallback`, so "this never fires in practice" is a +/// measurement instead of an assumption. pub(super) struct ManualGcScanGuard { engaged: bool, } impl ManualGcScanGuard { - pub(super) fn force_full_scan() -> Self { + pub(super) fn force_full_scan(site: super::ConservativeScanSite) -> Self { + super::record_scan_fallback(site); let engaged = CONSERVATIVE_STACK_SCAN_OVERRIDE.with(|c| { if c.get().is_some() { return false; @@ -244,10 +254,43 @@ impl Drop for ManualGcScanGuard { pub(super) fn conservative_stack_scan_mode() -> ConservativeStackScanMode { // Explicit env var wins (so the dedicated mode tests and ops bisection keep // working), then a per-thread override, then the build-time default. - if let Ok(value) = std::env::var("PERRY_CONSERVATIVE_STACK_SCAN") { - return conservative_stack_scan_mode_from_value(Some(&value)); + let env_mode = std::env::var("PERRY_CONSERVATIVE_STACK_SCAN") + .ok() + .map(|value| conservative_stack_scan_mode_from_value(Some(&value))); + let pinned = CONSERVATIVE_STACK_SCAN_OVERRIDE.with(|c| c.get()); + + // ★ #7148: in the TEST build a pinned override beats an env request for + // `Full`. The env var may make the scan *less* aggressive than the mode a + // test declared, never more. + // + // Why: `PERRY_CONSERVATIVE_STACK_SCAN=full` failed **134 of 1574** + // perry-runtime tests — a shipped escape hatch in a state nobody had + // verified. The failures were not soundness failures. A collector test's + // central assertion is *"this object should have been collected"*, and a + // conservative scan retains whatever the native stack happens to look like + // a pointer to, so an ambient `=full` broke exactly the assertions the + // suite exists to make. Since #7147 the test build has ONE declared scan + // mode (`Auto`) and the isolation guards are its authority; letting an + // ambient env var silently re-impose `Full` on a guarded scope reintroduces + // the test-build-is-a-different-GC-configuration problem #7147 removed. + // + // Kept rather than deleted (CLAUDE.md's kill-policy default) because + // `=full` is load-bearing *outside* the test build: it is the ratchet's + // validated sensitivity arm — the run that produced the 60 regression rows + // proving `gc-ratchet` can actually fail (`benchmarks/gc_ratchet/README.md`). + // Deleting the mode would delete a gate's only end-to-end proof. This makes + // it verified instead: the suite is green under `=full`. + #[cfg(test)] + if matches!(env_mode, Some(ConservativeStackScanMode::Full)) { + if let Some(pinned) = pinned { + return pinned; + } + } + + if let Some(mode) = env_mode { + return mode; } - if let Some(mode) = CONSERVATIVE_STACK_SCAN_OVERRIDE.with(|c| c.get()) { + if let Some(mode) = pinned { return mode; } // ONE default for every build, tests included. diff --git a/crates/perry-runtime/src/gc/scan_fallback.rs b/crates/perry-runtime/src/gc/scan_fallback.rs new file mode 100644 index 0000000000..cb22142899 --- /dev/null +++ b/crates/perry-runtime/src/gc/scan_fallback.rs @@ -0,0 +1,236 @@ +//! Conservative-scan fallback accounting and the safepoint-deferral counters +//! that make "the fallback never fires" a *measured* statement (#7148). +//! +//! # Why this module exists +//! +//! `ManualGcScanGuard::force_full_scan()` pins this thread's conservative +//! native-stack scan on for the duration of one collection. That is sound — +//! it retains anything reachable from a register or an unspilled native local +//! — but it is *catastrophically* expensive, and the cost is not a slowdown: +//! a conservative scan makes the copying minor ineligible +//! (`CopiedMinorFallbackReason::ConservativeStack`), so a cycle that takes it +//! runs **no copying minor at all**. `benchmarks/gc_ratchet/README.md` +//! measures the end-to-end effect of scanning on the eight ratchet probes: +//! `heap_used_bytes` +364% to +5371%, and `minor_cycles` → **0 on all eight**. +//! +//! So every `force_full_scan()` that fires is a collection that reclaims by +//! sweeping instead of by evacuating, on a heap that has retained everything +//! the native stack happened to look like a pointer to. +//! +//! # The rule +//! +//! Before #7148 there were six `force_full_scan()` sites and no way to tell, +//! for any given program, whether any of them ever ran. "It never fires in +//! practice" was an assumption. Per CLAUDE.md's four-ways-a-gate-cannot-fail +//! rule — *a gate must assert its subject was live* — an unobservable fallback +//! is indistinguishable from a fallback that fires on every cycle. +//! +//! Every site therefore records itself here, and the two *deferral* paths that +//! replace a scan with a precise safepoint collection record themselves too. +//! A test or a benchmark can now assert both halves of the claim: +//! +//! * the conservative fallback did **not** fire (`scan_fallback_count`), and +//! * the precise safepoint collection that replaced it **did** +//! (`safepoint_drain_count`) — not merely that nothing crashed. +//! +//! # Relationship to the explicit-safepoint contract +//! +//! The statepoint experiment (`exp/stackmap-viability`, +//! `docs/statepoint-gc-experiment.md`) states the invariant this instrument +//! measures compliance with: *a collection that skips the conservative stack +//! scan consumes only precise roots, so it may only begin at a declared +//! safepoint (a loop back-edge poll or the outermost microtask-pump +//! boundary); anywhere else it must scan conservatively.* That experiment +//! enforces the invariant with a thread-local declared-safepoint flag plus a +//! check at the root-scan subphase, in two levels (heal / strict panic). +//! +//! This module is the *reachability census* for the same contract on `main`: +//! it counts the sites where the contract's "anywhere else" arm is taken. +//! Driving those counts to zero is what makes the conservative scanner +//! deletable; the contract's enforcement check is what makes zero *provable* +//! rather than observed. The two compose — census first, enforcement second. + +use std::cell::Cell; + +/// A `force_full_scan()` callsite. Ordering matters only for the counter array. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(crate) enum ConservativeScanSite { + /// `gc_check_trigger` old-gen reclaim, at an allocation point, after the + /// safepoint deferral ran out of slack. Automatic. + OldReclaimSlackValve, + /// `gc_check_trigger` nursery-churn direct minor, at an allocation point, + /// after the safepoint deferral ran out of slack. Automatic. + NurseryChurnSlackValve, + /// `gc_try_emergency_reclaim` — a heap allocation already failed and the + /// caller is about to panic. Automatic; cannot defer (see `mod.rs`). + EmergencyReclaim, + /// `js_gc_memory_pressure` with a generated frame live above us. + /// Host-driven. + HostPressure, + /// `manual_gc_collect_now` — explicit `gc()`. Explicit. + ManualCollect, + /// `js_gc_module_minor` — explicit `perry/gc` `minor()`. Explicit. + ManualMinor, +} + +impl ConservativeScanSite { + pub(crate) const COUNT: usize = 6; + + const fn index(self) -> usize { + match self { + Self::OldReclaimSlackValve => 0, + Self::NurseryChurnSlackValve => 1, + Self::EmergencyReclaim => 2, + Self::HostPressure => 3, + Self::ManualCollect => 4, + Self::ManualMinor => 5, + } + } + + pub(crate) const fn as_str(self) -> &'static str { + match self { + Self::OldReclaimSlackValve => "old_reclaim_slack_valve", + Self::NurseryChurnSlackValve => "nursery_churn_slack_valve", + Self::EmergencyReclaim => "emergency_reclaim", + Self::HostPressure => "host_pressure", + Self::ManualCollect => "manual_collect", + Self::ManualMinor => "manual_minor", + } + } + + /// Whether this site is reached without any user code calling `gc()`. + /// The four automatic sites are the ones #7148 is about: they are the + /// collections a program pays for without asking for them. + pub(crate) const fn is_automatic(self) -> bool { + match self { + Self::OldReclaimSlackValve + | Self::NurseryChurnSlackValve + | Self::EmergencyReclaim + | Self::HostPressure => true, + Self::ManualCollect | Self::ManualMinor => false, + } + } + + pub(crate) const ALL: [Self; Self::COUNT] = [ + Self::OldReclaimSlackValve, + Self::NurseryChurnSlackValve, + Self::EmergencyReclaim, + Self::HostPressure, + Self::ManualCollect, + Self::ManualMinor, + ]; +} + +/// A precise-root safepoint collection that ran *instead of* a conservative +/// alloc-point collection. These are the "the deferral actually drained" +/// counters — the live-subject assertion for every deferral gate. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(crate) enum SafepointDrainKind { + /// A deferred nursery-pressure trigger collected at a safepoint. + NurseryMinor, + /// A deferred old-gen reclaim collected at a safepoint (#7148). + OldReclaim, + /// A host memory-pressure request collected at a safepoint (#7148). + HostPressure, +} + +impl SafepointDrainKind { + pub(crate) const COUNT: usize = 3; + + const fn index(self) -> usize { + match self { + Self::NurseryMinor => 0, + Self::OldReclaim => 1, + Self::HostPressure => 2, + } + } + + pub(crate) const fn as_str(self) -> &'static str { + match self { + Self::NurseryMinor => "nursery_minor", + Self::OldReclaim => "old_reclaim", + Self::HostPressure => "host_pressure", + } + } +} + +thread_local! { + static SCAN_FALLBACKS: Cell<[u64; ConservativeScanSite::COUNT]> = + const { Cell::new([0; ConservativeScanSite::COUNT]) }; + static SAFEPOINT_DRAINS: Cell<[u64; SafepointDrainKind::COUNT]> = + const { Cell::new([0; SafepointDrainKind::COUNT]) }; +} + +/// Record that `site` is about to force the conservative native-stack scan. +/// +/// Called from `ManualGcScanGuard::force_full_scan`, i.e. exactly once per +/// engagement, *including* engagements that turn out to be no-ops because an +/// override was already pinned — the census is of the callsite's intent, not +/// of the guard's internal bookkeeping. Under `PERRY_GC_DIAG` each occurrence +/// prints a line so an ops/benchmark run shows which sites a program reaches +/// and how often. +pub(crate) fn record_scan_fallback(site: ConservativeScanSite) { + let count = SCAN_FALLBACKS.with(|c| { + let mut counts = c.get(); + counts[site.index()] = counts[site.index()].saturating_add(1); + c.set(counts); + counts[site.index()] + }); + if std::env::var_os("PERRY_GC_DIAG").is_some() { + eprintln!( + "[gc-scan-fallback] site={} automatic={} count={}", + site.as_str(), + site.is_automatic(), + count + ); + } +} + +/// Record that a deferred collection drained at a precise-root safepoint — +/// the collection that a `force_full_scan()` site would otherwise have run +/// conservatively at an allocation point. +pub(crate) fn record_safepoint_drain(kind: SafepointDrainKind) { + let count = SAFEPOINT_DRAINS.with(|c| { + let mut counts = c.get(); + counts[kind.index()] = counts[kind.index()].saturating_add(1); + c.set(counts); + counts[kind.index()] + }); + if std::env::var_os("PERRY_GC_DIAG").is_some() { + eprintln!( + "[gc-safepoint-drain] kind={} count={}", + kind.as_str(), + count + ); + } +} + +pub(crate) fn scan_fallback_count(site: ConservativeScanSite) -> u64 { + SCAN_FALLBACKS.with(|c| c.get()[site.index()]) +} + +/// Total conservative-scan fallbacks across the four **automatic** sites. This +/// is the number #7148 wants driven to zero: explicit `gc()` is a user request +/// and is not part of the claim. +pub(crate) fn automatic_scan_fallback_total() -> u64 { + SCAN_FALLBACKS.with(|c| { + let counts = c.get(); + ConservativeScanSite::ALL + .iter() + .filter(|site| site.is_automatic()) + .map(|site| counts[site.index()]) + .sum() + }) +} + +pub(crate) fn safepoint_drain_count(kind: SafepointDrainKind) -> u64 { + SAFEPOINT_DRAINS.with(|c| c.get()[kind.index()]) +} + +/// Reset both counter families. Test-only: the counters are process-lifetime +/// on a real run. +#[cfg(test)] +pub(crate) fn reset_scan_fallback_counters() { + SCAN_FALLBACKS.with(|c| c.set([0; ConservativeScanSite::COUNT])); + SAFEPOINT_DRAINS.with(|c| c.set([0; SafepointDrainKind::COUNT])); +} diff --git a/crates/perry-runtime/src/gc/tests/roots.rs b/crates/perry-runtime/src/gc/tests/roots.rs index 82809bea56..ae2febf1e1 100644 --- a/crates/perry-runtime/src/gc/tests/roots.rs +++ b/crates/perry-runtime/src/gc/tests/roots.rs @@ -816,7 +816,7 @@ fn manual_gc_scan_guard_forces_full_scan_only_when_unpinned() { // Unpinned: the guard engages a Full override for its lifetime (#4977 — // explicit gc() must see top-level locals held only on the native stack). { - let _scan = ManualGcScanGuard::force_full_scan(); + let _scan = ManualGcScanGuard::force_full_scan(crate::gc::ConservativeScanSite::ManualCollect); assert_eq!( CONSERVATIVE_STACK_SCAN_OVERRIDE.with(|c| c.get()), Some(ConservativeStackScanMode::Full) @@ -828,7 +828,7 @@ fn manual_gc_scan_guard_forces_full_scan_only_when_unpinned() { // reclaim native-stack locals): the guard must not replace the override. set_conservative_stack_scan_override(Some(ConservativeStackScanMode::Auto)); { - let _scan = ManualGcScanGuard::force_full_scan(); + let _scan = ManualGcScanGuard::force_full_scan(crate::gc::ConservativeScanSite::ManualCollect); assert_eq!( CONSERVATIVE_STACK_SCAN_OVERRIDE.with(|c| c.get()), Some(ConservativeStackScanMode::Auto) From 5f1ae1de397f8f3adf80880e4ff4b9f19825429c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sat, 1 Aug 2026 08:32:11 +0200 Subject: [PATCH 2/8] test(gc): red-then-green coverage for the #7148 safepoint deferrals Every deferral test asserts BOTH that the conservative valve did not fire and that the precise safepoint collection which replaced it DID run (a drain counter). Checking only the first half would pass on a tree where the trigger never armed at all -- the largest possible regression reported as a pass (CLAUDE.md, four ways a gate cannot fail, #4). Adds two test-only scoped overrides next to the existing force_legacy_gc_pacing pattern: pinned moving pacing (so a deferral test cannot silently run under legacy pacing and pass for the wrong reason) and a shrinkable old-gen deferral slack (the shipped slack is 32 MB; crossing it for real would mean committing 32 MB of old-gen inside a unit test). --- crates/perry-runtime/src/gc/policy.rs | 71 ++++- crates/perry-runtime/src/gc/tests/mod.rs | 1 + .../src/gc/tests/scan_fallback.rs | 290 ++++++++++++++++++ 3 files changed, 357 insertions(+), 5 deletions(-) create mode 100644 crates/perry-runtime/src/gc/tests/scan_fallback.rs diff --git a/crates/perry-runtime/src/gc/policy.rs b/crates/perry-runtime/src/gc/policy.rs index 4abdec91b5..61b9947602 100644 --- a/crates/perry-runtime/src/gc/policy.rs +++ b/crates/perry-runtime/src/gc/policy.rs @@ -429,6 +429,22 @@ pub(super) fn force_legacy_gc_pacing() -> LegacyGcPacingGuard { LegacyGcPacingGuard { previous } } +/// Pin moving GC pacing (moving-loop polls ON) for the duration of the returned +/// guard — the shipped default, pinned explicitly so a #7148 deferral test +/// asserts against a *declared* pacing mode instead of inheriting whatever the +/// process-wide `PERRY_GC_MOVING_LOOP_POLLS` OnceLock resolved to. A test that +/// silently ran under legacy pacing would find the deferral branch dead and +/// pass for the wrong reason. +#[cfg(test)] +pub(super) fn force_moving_gc_pacing() -> LegacyGcPacingGuard { + let previous = GC_MOVING_LOOP_POLLS_TEST_OVERRIDE.with(|cell| { + let previous = cell.get(); + cell.set(Some(true)); + previous + }); + LegacyGcPacingGuard { previous } +} + pub(super) fn gc_trace_enabled() -> bool { #[cfg(test)] if GC_TRACE_TEST_FORCE.with(Cell::get) { @@ -668,6 +684,54 @@ pub(super) fn old_reclaim_pressure_bytes() -> usize { crate::arena::old_gen_in_use_bytes().saturating_add(external_side_live_bytes()) } +#[cfg(test)] +thread_local! { + /// Test-only override for the #7148 old-gen deferral slack. The shipped + /// slack is `gc_old_gen_reclaim_growth_dyn_bytes()` — 32 MB on an + /// unbudgeted host — and a test that wanted to watch the safety valve fire + /// would otherwise have to commit 32 MB of old-gen just to cross it. + /// Shrinking the slack lets the valve test drive the *real* branch in + /// `gc_check_trigger` deterministically and in microseconds. + static GC_OLD_RECLAIM_DEFER_SLACK_TEST_OVERRIDE: Cell> = + const { Cell::new(None) }; +} + +/// Growth allowance, in the old-gen trigger's own unit, that a deferred +/// old-gen reclaim may consume past its deferral point before the conservative +/// safety valve fires (#7148). +#[inline] +pub(super) fn old_reclaim_defer_slack_bytes() -> usize { + #[cfg(test)] + if let Some(slack) = GC_OLD_RECLAIM_DEFER_SLACK_TEST_OVERRIDE.with(Cell::get) { + return slack; + } + gc_old_gen_reclaim_growth_dyn_bytes() +} + +/// Scoped test-only override of the old-gen deferral slack. See +/// [`GC_OLD_RECLAIM_DEFER_SLACK_TEST_OVERRIDE`]. +#[cfg(test)] +pub(super) struct OldReclaimDeferSlackGuard { + previous: Option, +} + +#[cfg(test)] +impl Drop for OldReclaimDeferSlackGuard { + fn drop(&mut self) { + GC_OLD_RECLAIM_DEFER_SLACK_TEST_OVERRIDE.with(|cell| cell.set(self.previous)); + } +} + +#[cfg(test)] +pub(super) fn force_old_reclaim_defer_slack(slack: usize) -> OldReclaimDeferSlackGuard { + let previous = GC_OLD_RECLAIM_DEFER_SLACK_TEST_OVERRIDE.with(|cell| { + let previous = cell.get(); + cell.set(Some(slack)); + previous + }); + OldReclaimDeferSlackGuard { previous } +} + /// Committed arena bytes a deferred nursery trigger may allocate **past the /// point at which it was deferred** before the alloc-point non-moving minor /// runs as the safety valve (Phase 2/3). Loop back-edge polls drain the pending @@ -1352,11 +1416,8 @@ pub fn gc_check_trigger() { let already_deferred = GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(Cell::get); let deferred_at = already_deferred.then(|| GC_SAFEPOINT_OLD_RECLAIM_BASE.with(Cell::get)); - if moving_defer_within_slack( - old_pressure, - deferred_at, - gc_old_gen_reclaim_growth_dyn_bytes(), - ) { + if moving_defer_within_slack(old_pressure, deferred_at, old_reclaim_defer_slack_bytes()) + { if !already_deferred { GC_SAFEPOINT_OLD_RECLAIM_BASE.with(|base| base.set(old_pressure)); GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(|p| p.set(true)); diff --git a/crates/perry-runtime/src/gc/tests/mod.rs b/crates/perry-runtime/src/gc/tests/mod.rs index a5940470cb..52026c1819 100644 --- a/crates/perry-runtime/src/gc/tests/mod.rs +++ b/crates/perry-runtime/src/gc/tests/mod.rs @@ -20,6 +20,7 @@ mod os_tag; mod root_words; mod roots; mod runtime_roots; +mod scan_fallback; mod shadow_stack_ops; mod smoke; pub(super) mod support; diff --git a/crates/perry-runtime/src/gc/tests/scan_fallback.rs b/crates/perry-runtime/src/gc/tests/scan_fallback.rs new file mode 100644 index 0000000000..a716abb89c --- /dev/null +++ b/crates/perry-runtime/src/gc/tests/scan_fallback.rs @@ -0,0 +1,290 @@ +//! #7148: the conservative-scan fallback must be *unreachable*, not imprecise. +//! +//! Each test below asserts BOTH halves of a deferral claim, because only the +//! first half is cheap to fake: +//! +//! 1. the conservative valve did **not** fire (`scan_fallback_count == 0`), and +//! 2. the precise safepoint collection that replaced it **did** +//! (`safepoint_drain_count > 0`). +//! +//! CLAUDE.md's fourth way a gate cannot fail is "the gate runs but its subject +//! never did". A test that only checked (1) would pass on a tree where the +//! trigger never armed at all — the strongest possible regression reported as +//! a pass. Every deferral test here therefore ends on a live-subject counter. + +use super::super::*; +use super::support::*; + +/// Make `gc_budgeted_due_trigger()` report `OldReclaim` without allocating +/// 48 MB of old-gen: `GC_OLD_RECLAIM_PENDING` is the sticky "a full old-gen +/// reclaim is owed" flag it reads first (the same lever +/// `budgeted_step_api.rs` uses). +fn arm_old_reclaim() { + GC_OLD_RECLAIM_PENDING.with(|pending| pending.set(true)); +} + +fn clear_old_reclaim_state() { + GC_OLD_RECLAIM_PENDING.with(|pending| pending.set(false)); + GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(|pending| pending.set(false)); + GC_SAFEPOINT_PENDING.with(|pending| pending.set(false)); + let old_in_use = crate::arena::old_gen_in_use_bytes(); + GC_LAST_OLD_RECLAIM_IN_USE_BYTES.with(|bytes| bytes.set(old_in_use)); +} + +#[test] +fn old_reclaim_defers_to_a_precise_safepoint_instead_of_scanning() { + let _isolation = GcTestIsolationGuard::new(); + let _pacing = crate::gc::policy::force_moving_gc_pacing(); + let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + clear_old_reclaim_state(); + reset_scan_fallback_counters(); + + arm_old_reclaim(); + gc_check_trigger(); + + // Before #7148 this call ran a full mark-sweep here, at the allocation + // point, behind `force_full_scan()`. + assert_eq!( + scan_fallback_count(ConservativeScanSite::OldReclaimSlackValve), + 0, + "old-gen reclaim must defer, not scan, while it is within slack" + ); + assert!( + GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(std::cell::Cell::get), + "the deferral must be recorded in the old-gen unit" + ); + assert!( + GC_SAFEPOINT_PENDING.with(std::cell::Cell::get), + "`js_gc_loop_safepoint` polls GC_SAFEPOINT_PENDING, so the old-gen \ + deferral must set it too or nothing drains the request" + ); + assert!( + GC_OLD_RECLAIM_PENDING.with(std::cell::Cell::get), + "deferring must not retire the request — the trigger has to stay due \ + so the safepoint finds it" + ); + + // Drain at the precise safepoint the deferral promised. + js_gc_loop_safepoint(); + + assert_eq!( + safepoint_drain_count(SafepointDrainKind::OldReclaim), + 1, + "LIVE SUBJECT: the deferred full mark-sweep must actually have run at \ + the safepoint — 'nothing scanned' is worthless if nothing collected" + ); + assert_eq!( + scan_fallback_count(ConservativeScanSite::OldReclaimSlackValve), + 0, + "the safepoint path must not force the scan" + ); + assert!( + !GC_OLD_RECLAIM_PENDING.with(std::cell::Cell::get), + "the safepoint collection retires the request" + ); + assert!( + !GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(std::cell::Cell::get), + "the safepoint collection clears its own deferral flag" + ); + + clear_old_reclaim_state(); +} + +#[test] +fn old_reclaim_slack_valve_fires_when_the_deferral_never_drains() { + let _isolation = GcTestIsolationGuard::new(); + let _pacing = crate::gc::policy::force_moving_gc_pacing(); + let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + clear_old_reclaim_state(); + reset_scan_fallback_counters(); + + // The other direction. A deferral whose slack cannot expire is #7024's bug + // wearing new clothes: RSS grows without bound and the branch that was + // supposed to bound it is dead. The shipped slack is 32 MB, so crossing it + // for real would mean committing 32 MB of old-gen inside a unit test; + // shrink it instead so the *real* branch in `gc_check_trigger` runs + // deterministically. Slack 0 with the baseline at the current pressure is + // the exact state "this deferral has used up its whole allowance". + let _slack = crate::gc::policy::force_old_reclaim_defer_slack(0); + arm_old_reclaim(); + let pressure = old_reclaim_pressure_bytes(); + GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(|pending| pending.set(true)); + GC_SAFEPOINT_OLD_RECLAIM_BASE.with(|base| base.set(pressure)); + + gc_check_trigger(); + + assert_eq!( + scan_fallback_count(ConservativeScanSite::OldReclaimSlackValve), + 1, + "past the slack the conservative valve MUST fire — it is the only \ + thing bounding old-gen growth when no safepoint is reachable" + ); + assert!( + !GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(std::cell::Cell::get), + "the valve retires the deferral; a stale exceeded baseline would \ + disable deferral for the rest of the process" + ); + + clear_old_reclaim_state(); +} + +#[test] +fn host_pressure_collects_precisely_when_no_generated_frame_is_live() { + let _isolation = GcTestIsolationGuard::new(); + let _pacing = crate::gc::policy::force_moving_gc_pacing(); + clear_old_reclaim_state(); + reset_shadow_stack(); + reset_scan_fallback_counters(); + + assert!( + !crate::gc::roots::shadow_stack_has_active_frame(), + "precondition: the run-loop-boundary case has an empty shadow stack" + ); + + let result = js_gc_memory_pressure(1); + + assert_eq!(result, 2, "collected synchronously"); + assert_eq!( + scan_fallback_count(ConservativeScanSite::HostPressure), + 0, + "with no generated frame live the precise root set is complete, so \ + the host-pressure collection must not force the scan" + ); + assert_eq!( + safepoint_drain_count(SafepointDrainKind::HostPressure), + 1, + "LIVE SUBJECT: the precise collection ran" + ); + + clear_old_reclaim_state(); +} + +#[test] +fn host_pressure_defers_when_a_generated_frame_is_live() { + let _isolation = GcTestIsolationGuard::new(); + let _pacing = crate::gc::policy::force_moving_gc_pacing(); + let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + clear_old_reclaim_state(); + reset_shadow_stack(); + reset_scan_fallback_counters(); + + let frame = js_shadow_frame_push(2); + assert!(crate::gc::roots::shadow_stack_has_active_frame()); + + // Critical level: the owed collection must be a FULL cycle, so the + // deferral has to arm the sticky old-gen-reclaim request rather than only + // the nursery one. + let result = js_gc_memory_pressure(2); + js_shadow_frame_pop(frame); + + assert_eq!( + result, 1, + "documented return code for 'trigger lowered, collection deferred'" + ); + assert_eq!( + scan_fallback_count(ConservativeScanSite::HostPressure), + 0, + "a live generated frame means a safepoint is reachable — defer to it \ + instead of scanning conservatively" + ); + assert!(GC_SAFEPOINT_PENDING.with(std::cell::Cell::get)); + assert!( + GC_OLD_RECLAIM_PENDING.with(std::cell::Cell::get), + "level >= 2 must owe a FULL cycle, not a minor" + ); + + // The frame is gone; drain the promise. + js_gc_loop_safepoint(); + assert_eq!( + safepoint_drain_count(SafepointDrainKind::OldReclaim), + 1, + "LIVE SUBJECT: the deferred critical-pressure full cycle ran at the \ + safepoint" + ); + + clear_old_reclaim_state(); +} + +#[test] +fn explicit_gc_still_scans_and_the_census_attributes_it() { + let _isolation = GcTestIsolationGuard::new(); + clear_old_reclaim_state(); + reset_scan_fallback_counters(); + + js_gc_collect(); + + // #7148 deliberately does NOT defer explicit `gc()`: it is a user request + // with synchronous semantics. What changes is that its cost is now + // attributable — every `gc_ratchet` probe ends with one of these, so a + // census that lumped it in with the automatic sites would misread. + assert!( + scan_fallback_count(ConservativeScanSite::ManualCollect) >= 1, + "explicit gc() keeps the scan and must be counted as non-automatic" + ); + assert_eq!( + automatic_scan_fallback_total(), + 0, + "an explicit gc() must not be counted against the automatic-site total \ + that #7148 is driving to zero" + ); + + clear_old_reclaim_state(); +} + +#[test] +fn conservative_scan_env_full_does_not_override_a_pinned_test_mode() { + // `PERRY_CONSERVATIVE_STACK_SCAN=full` failed 134 of 1574 runtime tests + // because the env value beat the per-thread override, so a test that had + // declared its roots precise still got the conservative scan and its + // "this should have been collected" assertion broke. The env var may now + // make the scan LESS aggressive than a test declared, never more. + let _env = EnvVarGuard::set("PERRY_CONSERVATIVE_STACK_SCAN", "full"); + let previous = crate::gc::roots::set_conservative_stack_scan_override(None); + + // Unpinned: the ops escape hatch still works. This is the arm the + // `gc_ratchet` sensitivity run depends on — production binaries have no + // pinned override, so `=full` still forces the scan there. + assert_eq!( + crate::gc::roots::conservative_stack_scan_decision(), + ConservativeStackScanDecision::Scan, + "with nothing pinned, =full must still force the scan" + ); + + // Pinned by a test isolation guard: the declared mode wins. + crate::gc::roots::set_conservative_stack_scan_override(Some(ConservativeStackScanMode::Auto)); + assert_eq!( + crate::gc::roots::conservative_stack_scan_decision(), + ConservativeStackScanDecision::SkipDisabled, + "a pinned Auto must beat an ambient =full" + ); + + crate::gc::roots::set_conservative_stack_scan_override(Some( + ConservativeStackScanMode::Disabled, + )); + assert_eq!( + crate::gc::roots::conservative_stack_scan_decision(), + ConservativeStackScanDecision::SkipDisabled, + "a pinned Disabled must beat an ambient =full" + ); + + crate::gc::roots::set_conservative_stack_scan_override(previous); +} + +#[test] +fn conservative_scan_env_off_still_beats_a_forced_scan() { + // The other precedence direction, unchanged by #7148: `=0` is the + // bisection escape hatch, and it must keep winning over + // `ManualGcScanGuard::force_full_scan()` (which pins `Full`). Narrowing + // the #7148 rule to "env asking for Full" is what preserves this. + let _env = EnvVarGuard::set("PERRY_CONSERVATIVE_STACK_SCAN", "0"); + let previous = crate::gc::roots::set_conservative_stack_scan_override(None); + + crate::gc::roots::set_conservative_stack_scan_override(Some(ConservativeStackScanMode::Full)); + assert_eq!( + crate::gc::roots::conservative_stack_scan_decision(), + ConservativeStackScanDecision::SkipDisabled, + "=0 must still disable the scan even when a guard pinned Full" + ); + + crate::gc::roots::set_conservative_stack_scan_override(previous); +} From 4030e55ad8ae32661e3cfa1901c56cadc3fc0601 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sat, 1 Aug 2026 08:39:28 +0200 Subject: [PATCH 3/8] docs(gc-ratchet): record why PERRY_CONSERVATIVE_STACK_SCAN=full is kept and how it was verified --- benchmarks/gc_ratchet/README.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/benchmarks/gc_ratchet/README.md b/benchmarks/gc_ratchet/README.md index 4d9322de36..79ae3404a9 100644 --- a/benchmarks/gc_ratchet/README.md +++ b/benchmarks/gc_ratchet/README.md @@ -122,6 +122,33 @@ error — the "probe ran no collection" rule deliberately lives in that the largest possible regression is not misdiagnosed as "your probe is too small". +### `PERRY_CONSERVATIVE_STACK_SCAN=full` is this gate's sensitivity arm — keep it + +CLAUDE.md's kill-policy says an unexercised mode gets deleted, and `=full` was a +clean candidate: it failed **134 of 1574** `perry-runtime` tests, a shipped +escape hatch nobody had verified (#7148). It was **kept** rather than deleted +for the reason this section documents — it is the only end-to-end proof that +`gc-ratchet` can fail. Deleting the mode would delete a gate's proof, which is +the same failure the kill-policy exists to prevent, one level up. + +It is verified instead. The 134 failures were not soundness failures: a +collector test's central assertion is *"this object should have been +collected"*, and an ambient conservative scan retains whatever the native stack +looks like a pointer to, so `=full` broke exactly the assertions the suite +exists to make. Since #7147 the test build has one declared scan mode and the +isolation guards are its authority, so as of #7148 a pinned per-thread override +beats an env request for `Full` **in the test build only** — the env var may +make the scan less aggressive than a test declared, never more. Production +binaries pin no override, so the arm above is unchanged and still forces the +scan. + +The `heap_used_bytes` column above is also the quantitative case behind #7148: +every `force_full_scan()` that fires at runtime is a collection that runs no +copying minor, so the four *automatic* fallback sites were each worth this much +RSS whenever they were reached. They are now counted +(`[gc-scan-fallback] site=… automatic=…` under `PERRY_GC_DIAG`), so a probe run +shows directly whether any of them fired. + ## Cross-host evidence (why the shared-CI profile gates what it gates) The baseline is captured on an 8-core M1 with 8 GB at load ~1.2. The first From 951874719cf4af3b4796c581bc9ca7c7c6b7da15 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sat, 1 Aug 2026 08:41:05 +0200 Subject: [PATCH 4/8] changelog: fragment for PR 7166 --- .../7166-gc-safepoint-deferral-scan-census.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 changelog.d/7166-gc-safepoint-deferral-scan-census.md diff --git a/changelog.d/7166-gc-safepoint-deferral-scan-census.md b/changelog.d/7166-gc-safepoint-deferral-scan-census.md new file mode 100644 index 0000000000..2ce057a646 --- /dev/null +++ b/changelog.d/7166-gc-safepoint-deferral-scan-census.md @@ -0,0 +1,15 @@ +fix(gc): defer old-gen reclaim and OS memory-pressure collections to precise safepoints, and put a census on every conservative-scan fallback (#7148). + +`main` reached a forced conservative native-stack scan from **six** sites, **four of them without any user calling `gc()`**. That is not merely an imprecision: a forced scan makes the copying minor ineligible (`CopiedMinorFallbackReason::ConservativeStack`), so the cycle it covers runs **no copying minor at all**. `benchmarks/gc_ratchet/README.md` measures the end state — `heap_used_bytes` **+364% to +5371%** and `minor_cycles` → **0 on all eight probes**. Every automatic fallback that fired was worth that much RSS. + +The right end state (#7148) is that the fallback becomes *unreachable* — collections always defer to a precise safepoint — not *imprecise*, which would trade a cost problem for a soundness problem. Per-site: + +- **Old-gen reclaim (`gc/policy.rs`) — deferred.** `gc_safepoint_moving_minor` used to bail on `OldReclaim`; it now runs the same full mark-sweep at the safepoint with `SkipDisabled` roots and no forced scan. The allocation-point arm keeps only a bounded safety valve, and the valve's slack is a **delta in the trigger's own unit** (old-gen in-use + external side-table bytes; one 32 MB growth quantum). That unit choice is load-bearing: #5476's workload is >16 KB temporaries born straight into the old arena, which barely move `arena_total_bytes()`, so an arena-unit slack would never expire and the valve would never fire — #7024's dead-branch bug inverted. +- **OS memory pressure (`gc/pressure.rs`) — deferred, conditionally.** The old rationale ("a host may deliver the callback with unspilled locals on the native frames above us") is now *tested* rather than assumed. With no active shadow frame the precise root set is complete, so the collection runs here with no scan and the copying minor stays eligible — the run-loop-boundary case the module's own docs call typical, and the case where deferring would be worst, since an idle process reaches no safepoint and would shed nothing before the jetsam. With a generated frame live a safepoint *is* reachable, so the request is deferred and the handler returns `1`, the code this API already documents as "trigger lowered, collection deferred". The lowered arena trigger is untouched either way, so no backstop is removed. +- **Nursery-churn valve, emergency reclaim, `gc()`, `perry/gc` `minor()` — kept, now counted.** Emergency reclaim provably cannot defer: it runs after an allocation has already failed and the caller's next act is to panic, so there is no next safepoint to defer to. The two explicit sites are user requests with synchronous semantics. + +New `gc/scan_fallback.rs` records every fallback (`[gc-scan-fallback] site=… automatic=… count=…` under `PERRY_GC_DIAG`) and every precise-safepoint drain that replaced one (`[gc-safepoint-drain] kind=…`), separating automatic from explicit sites — all eight ratchet probes end with an explicit full `gc()`, so that site fires once per probe by construction and an undifferentiated census would misread. This is the reachability census for the invariant the statepoint experiment states and enforces (`docs/statepoint-gc-experiment.md`, `exp/stackmap-viability`): *a collection that skips the conservative scan consumes only precise roots, so it may only begin at a declared safepoint.* Census first, enforcement second. + +Also fixes `PERRY_CONSERVATIVE_STACK_SCAN=full`, which failed **134 of 1574** runtime tests on `main` — a shipped escape hatch nobody had verified. Fixed rather than deleted, against the kill-policy's default, because `=full` is `gc-ratchet`'s validated sensitivity arm: the run that produced the 60 regression rows proving that gate can fail. Deleting it would delete a gate's only end-to-end proof. The failures were never soundness failures — a collector test's central assertion is *"this object should have been collected"*, and an ambient conservative scan retains whatever the native stack looks like a pointer to. Since #7147 the test build has one declared scan mode and the isolation guards are its authority, so **in the test build only** a pinned override now beats an env request for `Full`; the env var may make the scan less aggressive than a test declared, never more. `=0` still beats a pinned `Full`, so the bisection hatch is intact. Production binaries pin no override and are unchanged. + +Every deferral test asserts both that the conservative valve did not fire and that the precise collection which replaced it actually ran (a drain counter) — checking only the first half would pass on a tree where the trigger never armed at all, the largest possible regression reported as a pass. Verified red-then-green with five one-hunk sabotages (deferral removed, safepoint drain removed, host pressure reverted, slack valve removed, `=full` precedence reverted), each reddening exactly the tests that name the property it broke. From 6abb9e6dfafd10527e045577e8a6563e47ad7ffa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sat, 1 Aug 2026 08:53:44 +0200 Subject: [PATCH 5/8] refactor(gc): split roots/scan_mode.rs out; delete the host-pressure conservative arm outright Two follow-ups the compiler found. 1. gc/roots.rs crossed the 2000-line file-size gate (1992 -> 2035). The conservative-scan mode block is topically self-contained -- the mode enum, its env/override precedence, ManualGcScanGuard, and the decision the mark phase consults -- so it moves to roots/scan_mode.rs. roots.rs is now 1851. 2. ConservativeScanSite::HostPressure was never constructed, because after the deferral js_gc_memory_pressure has NO conservative arm left: it either collects with precise roots or defers. So the variant is deleted rather than kept 'for completeness'. An enum arm nothing can produce is a claim no test can check -- the host-pressure tests now assert automatic_scan_fallback_total == 0 instead, which catches reintroduction at any automatic site. The counter read APIs are #[cfg(test)]; ops observability is the PERRY_GC_DIAG line, which is unconditional. --- crates/perry-runtime/src/gc/mod.rs | 6 +- crates/perry-runtime/src/gc/policy.rs | 3 +- crates/perry-runtime/src/gc/roots.rs | 205 ++--------------- .../perry-runtime/src/gc/roots/scan_mode.rs | 210 ++++++++++++++++++ crates/perry-runtime/src/gc/scan_fallback.rs | 33 +-- crates/perry-runtime/src/gc/tests/roots.rs | 6 +- .../src/gc/tests/scan_fallback.rs | 7 +- 7 files changed, 255 insertions(+), 215 deletions(-) create mode 100644 crates/perry-runtime/src/gc/roots/scan_mode.rs diff --git a/crates/perry-runtime/src/gc/mod.rs b/crates/perry-runtime/src/gc/mod.rs index 41990bb365..d639a8a18e 100644 --- a/crates/perry-runtime/src/gc/mod.rs +++ b/crates/perry-runtime/src/gc/mod.rs @@ -340,7 +340,8 @@ fn gc_collect_emergency_full() -> GcCollectOutcome { /// It is instead made *measurable* (`ConservativeScanSite::EmergencyReclaim` /// + a `PERRY_GC_DIAG` line), so "emergency reclaim never fires in practice" /// stops being an assumption. The long-term plan for this site is the -/// statepoint work (`docs/statepoint-gc-experiment.md`): with native stack +/// statepoint work (`docs/statepoint-gc-experiment.md` on branch +/// `exp/stackmap-viability`, not on `main`): with native stack /// maps a precise root set exists at *any* mapped PC, so an OOM-time /// collection would not need the scan at all. That is the only mechanism that /// removes this site, and it is not a deferral. @@ -355,8 +356,7 @@ pub(crate) fn gc_try_emergency_reclaim() -> bool { return false; } IN_EMERGENCY.with(|c| c.set(true)); - let _scan = - roots::ManualGcScanGuard::force_full_scan(ConservativeScanSite::EmergencyReclaim); + let _scan = roots::ManualGcScanGuard::force_full_scan(ConservativeScanSite::EmergencyReclaim); let _ = gc_collect_emergency_full(); IN_EMERGENCY.with(|c| c.set(false)); true diff --git a/crates/perry-runtime/src/gc/policy.rs b/crates/perry-runtime/src/gc/policy.rs index 61b9947602..7ba914bcd2 100644 --- a/crates/perry-runtime/src/gc/policy.rs +++ b/crates/perry-runtime/src/gc/policy.rs @@ -2365,7 +2365,8 @@ pub extern "C" fn js_gc_collect() { /// The known cost is #6942/#6946: forcing the scan makes this path non-moving, /// which is why `PERRY_GC_FORCE_EVACUATE` was inert for every `gc()`-driven /// test. Removing the scan here needs precise roots at the `gc()` callsite PC -/// — the safepoint contract in `docs/statepoint-gc-experiment.md` — not a +/// — the safepoint contract in `docs/statepoint-gc-experiment.md` on branch +/// `exp/stackmap-viability` (not on `main`) — not a /// deferral. fn manual_gc_collect_now() { let _scan = super::roots::ManualGcScanGuard::force_full_scan( diff --git a/crates/perry-runtime/src/gc/roots.rs b/crates/perry-runtime/src/gc/roots.rs index 575418afe9..91390de30e 100644 --- a/crates/perry-runtime/src/gc/roots.rs +++ b/crates/perry-runtime/src/gc/roots.rs @@ -2,6 +2,7 @@ use super::*; use std::any::Any; mod runtime_handles; +mod scan_mode; mod scanner_shims; mod shadow_stack; mod temp_roots; @@ -22,6 +23,18 @@ pub use scanner_shims::{ small_int_cache_mutable_root_scanner, small_int_cache_root_scanner, timer_mutable_root_scanner, timer_root_scanner, transition_cache_mutable_root_scanner, transition_cache_root_scanner, }; +// The conservative-scan mode lives in `roots/scan_mode.rs` (split out in #7148 +// when this file crossed the 2,000-line gate) but every consumer names it +// `gc::roots::…` or reaches it through `mod.rs`'s `pub use roots::*`, so the +// whole surface is re-exported here — same shape, and same `allow`, as the +// `shadow_stack` re-exports below. +#[allow(unused_imports)] +pub(crate) use scan_mode::{ + conservative_stack_scan_decision, conservative_stack_scan_decision_for, + conservative_stack_scan_mode, conservative_stack_scan_mode_from_value, + set_conservative_stack_scan_override, ConservativeStackScanDecision, ConservativeStackScanMode, + ManualGcScanGuard, CONSERVATIVE_STACK_SCAN_OVERRIDE, +}; pub(crate) use shadow_stack::shadow_stack_has_active_frame; pub(crate) use shadow_stack::SHADOW; #[allow(unused_imports)] @@ -147,198 +160,6 @@ pub(super) fn exit_gc_root_lock() { } } -#[derive(Clone, Copy, Debug, Eq, PartialEq)] -pub(crate) enum ConservativeStackScanMode { - Auto, - Disabled, - Full, -} - -#[derive(Clone, Copy, Debug, Eq, PartialEq, Default)] -pub(super) enum ConservativeStackScanDecision { - Scan, - #[default] - SkipDisabled, -} - -impl ConservativeStackScanDecision { - #[cfg(feature = "diagnostics")] - #[inline] - pub(super) const fn as_str(self) -> &'static str { - match self { - Self::Scan => "scan", - Self::SkipDisabled => "skip_disabled", - } - } -} - -pub(super) fn conservative_stack_scan_mode_from_value( - value: Option<&str>, -) -> ConservativeStackScanMode { - let Some(value) = value else { - return ConservativeStackScanMode::Auto; - }; - match value.trim().to_ascii_lowercase().as_str() { - "" | "auto" => ConservativeStackScanMode::Auto, - "0" | "off" | "false" => ConservativeStackScanMode::Disabled, - "1" | "on" | "true" | "full" | "debug" => ConservativeStackScanMode::Full, - _ => ConservativeStackScanMode::Auto, - } -} - -thread_local! { - /// Per-thread override for the conservative-scan mode, taking precedence over - /// the `#[cfg(test)]` default but not an explicit env var. The GC unit tests - /// set this to `Auto` (skip) inside their controlled-root scopes so a forced - /// collection still reclaims the objects they hold only as native-stack - /// locals; see `ScopedRootScannerRegistryGuard`. - pub(super) static CONSERVATIVE_STACK_SCAN_OVERRIDE: std::cell::Cell> = - const { std::cell::Cell::new(None) }; -} - -/// Set (or clear) this thread's conservative-scan mode override, returning the -/// previous value. -#[allow(dead_code)] // test-only scaffolding: GC unit tests (gc/tests) pin the scan mode via this override -pub(crate) fn set_conservative_stack_scan_override( - mode: Option, -) -> Option { - CONSERVATIVE_STACK_SCAN_OVERRIDE.with(|c| c.replace(mode)) -} - -/// Scoped guard forcing the conservative native-stack scan for an explicit -/// `gc()` collection (#4977). In the default `Auto` mode a full collection -/// skips the native scan, but at a `gc()` callsite live module-init/top-level -/// locals may be held only on the native stack — neither the precise -/// shadow-stack roots nor the module-var scanners cover them — so the -/// collector reclaimed live object graphs and later field reads returned -/// dangling-pointer garbage. An already-pinned per-thread override wins (the -/// GC unit tests pin `Auto` so a forced collection still reclaims objects they -/// hold only as native-stack locals), and an explicit -/// `PERRY_CONSERVATIVE_STACK_SCAN` env value beats any override either way, -/// so the bisection escape hatch keeps working. -/// -/// ★ #7148: engaging this guard is *expensive*, not merely imprecise — the -/// conservative scan makes the copying minor ineligible -/// (`CopiedMinorFallbackReason::ConservativeStack`), so the cycle it covers -/// runs no copying minor at all (`benchmarks/gc_ratchet/README.md`: -/// `heap_used_bytes` +364%..+5371%, `minor_cycles` → 0 on all eight probes). -/// Every callsite therefore declares which of the six sites it is and is -/// counted in `super::scan_fallback`, so "this never fires in practice" is a -/// measurement instead of an assumption. -pub(super) struct ManualGcScanGuard { - engaged: bool, -} - -impl ManualGcScanGuard { - pub(super) fn force_full_scan(site: super::ConservativeScanSite) -> Self { - super::record_scan_fallback(site); - let engaged = CONSERVATIVE_STACK_SCAN_OVERRIDE.with(|c| { - if c.get().is_some() { - return false; - } - c.set(Some(ConservativeStackScanMode::Full)); - true - }); - Self { engaged } - } -} - -impl Drop for ManualGcScanGuard { - fn drop(&mut self) { - if self.engaged { - CONSERVATIVE_STACK_SCAN_OVERRIDE.with(|c| c.set(None)); - } - } -} - -pub(super) fn conservative_stack_scan_mode() -> ConservativeStackScanMode { - // Explicit env var wins (so the dedicated mode tests and ops bisection keep - // working), then a per-thread override, then the build-time default. - let env_mode = std::env::var("PERRY_CONSERVATIVE_STACK_SCAN") - .ok() - .map(|value| conservative_stack_scan_mode_from_value(Some(&value))); - let pinned = CONSERVATIVE_STACK_SCAN_OVERRIDE.with(|c| c.get()); - - // ★ #7148: in the TEST build a pinned override beats an env request for - // `Full`. The env var may make the scan *less* aggressive than the mode a - // test declared, never more. - // - // Why: `PERRY_CONSERVATIVE_STACK_SCAN=full` failed **134 of 1574** - // perry-runtime tests — a shipped escape hatch in a state nobody had - // verified. The failures were not soundness failures. A collector test's - // central assertion is *"this object should have been collected"*, and a - // conservative scan retains whatever the native stack happens to look like - // a pointer to, so an ambient `=full` broke exactly the assertions the - // suite exists to make. Since #7147 the test build has ONE declared scan - // mode (`Auto`) and the isolation guards are its authority; letting an - // ambient env var silently re-impose `Full` on a guarded scope reintroduces - // the test-build-is-a-different-GC-configuration problem #7147 removed. - // - // Kept rather than deleted (CLAUDE.md's kill-policy default) because - // `=full` is load-bearing *outside* the test build: it is the ratchet's - // validated sensitivity arm — the run that produced the 60 regression rows - // proving `gc-ratchet` can actually fail (`benchmarks/gc_ratchet/README.md`). - // Deleting the mode would delete a gate's only end-to-end proof. This makes - // it verified instead: the suite is green under `=full`. - #[cfg(test)] - if matches!(env_mode, Some(ConservativeStackScanMode::Full)) { - if let Some(pinned) = pinned { - return pinned; - } - } - - if let Some(mode) = env_mode { - return mode; - } - if let Some(mode) = pinned { - return mode; - } - // ONE default for every build, tests included. - // - // The test build used to default to `Full` on the grounds that unit tests - // root GC-managed pointers as raw locals on the *native* (Rust) stack - // rather than the shadow stack, so the conservative scan was what kept them - // alive. That is a description of how some tests were written, not a - // requirement of testing, and it had a cost that outweighed the - // convenience: it made the test build a DIFFERENT GC configuration from - // production. A missing precise root — the #7055 shape — was rescued by the - // native scan under test and was a live wrong answer in production, so the - // suite filtered out precisely the failure class it exists to catch. It - // also made the copying minor ineligible under test - // (`CopiedMinorFallbackReason::ConservativeStack`), so relocation was - // largely unexercised. - // - // The correct mechanism already existed and the tests already used it: - // `RuntimeHandleScope` / `gc_temp_root_*`, plus the isolation guards, which - // pin `Auto` themselves — 351 of the 531 gc tests were already running this - // way. Flipping the default cost exactly one test conversion - // (`test_minor_preserves_old_to_young_edge_across_minors`, which read a - // relocated child through a stale pre-collection local). - // - // A test that needs the scan *provably* off — not merely resolved off by - // today's `Auto` policy — should pin `ConservativeScanDisabledGuard`. - ConservativeStackScanMode::Auto -} - -#[inline] -pub(super) fn conservative_stack_scan_decision_for( - mode: ConservativeStackScanMode, - _shadow_frame_active: bool, -) -> ConservativeStackScanDecision { - match mode { - ConservativeStackScanMode::Disabled => ConservativeStackScanDecision::SkipDisabled, - ConservativeStackScanMode::Full => ConservativeStackScanDecision::Scan, - ConservativeStackScanMode::Auto => ConservativeStackScanDecision::SkipDisabled, - } -} - -pub(super) fn conservative_stack_scan_decision() -> ConservativeStackScanDecision { - conservative_stack_scan_decision_for( - conservative_stack_scan_mode(), - shadow_stack_has_active_frame(), - ) -} - /// Register a root scanner function. /// Each scanner is called during the mark phase to discover roots. /// This legacy API exposes copied values only. It remains supported for diff --git a/crates/perry-runtime/src/gc/roots/scan_mode.rs b/crates/perry-runtime/src/gc/roots/scan_mode.rs new file mode 100644 index 0000000000..12122b0fcb --- /dev/null +++ b/crates/perry-runtime/src/gc/roots/scan_mode.rs @@ -0,0 +1,210 @@ +//! Conservative native-stack scan mode: how a collection decides whether to +//! walk this thread's native (Rust) stack looking for anything that might be a +//! heap pointer. +//! +//! Split out of `roots.rs` in #7148 — that file crossed the 2,000-line gate, +//! and this is the topically self-contained piece: the mode enum, its env/ +//! override precedence, the `ManualGcScanGuard` that pins `Full` for one +//! collection, and the decision the mark phase consults. Nothing else in +//! `roots.rs` reads its internals. +//! +//! ★ The decision this module returns is the single most expensive bit in the +//! collector. `Scan` makes the copying minor ineligible +//! (`CopiedMinorFallbackReason::ConservativeStack`), so a cycle that takes it +//! runs no copying minor at all — measured at `heap_used_bytes` +364%..+5371% +//! and `minor_cycles` → 0 on all eight `gc_ratchet` probes. See +//! `super::super::scan_fallback` for the census of who asks for it. + +use super::super::*; + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(crate) enum ConservativeStackScanMode { + Auto, + Disabled, + Full, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, Default)] +pub(crate) enum ConservativeStackScanDecision { + Scan, + #[default] + SkipDisabled, +} + +impl ConservativeStackScanDecision { + #[cfg(feature = "diagnostics")] + #[inline] + pub(crate) const fn as_str(self) -> &'static str { + match self { + Self::Scan => "scan", + Self::SkipDisabled => "skip_disabled", + } + } +} + +pub(crate) fn conservative_stack_scan_mode_from_value( + value: Option<&str>, +) -> ConservativeStackScanMode { + let Some(value) = value else { + return ConservativeStackScanMode::Auto; + }; + match value.trim().to_ascii_lowercase().as_str() { + "" | "auto" => ConservativeStackScanMode::Auto, + "0" | "off" | "false" => ConservativeStackScanMode::Disabled, + "1" | "on" | "true" | "full" | "debug" => ConservativeStackScanMode::Full, + _ => ConservativeStackScanMode::Auto, + } +} + +thread_local! { + /// Per-thread override for the conservative-scan mode, taking precedence over + /// the `#[cfg(test)]` default but not an explicit env var. The GC unit tests + /// set this to `Auto` (skip) inside their controlled-root scopes so a forced + /// collection still reclaims the objects they hold only as native-stack + /// locals; see `ScopedRootScannerRegistryGuard`. + pub(crate) static CONSERVATIVE_STACK_SCAN_OVERRIDE: std::cell::Cell> = + const { std::cell::Cell::new(None) }; +} + +/// Set (or clear) this thread's conservative-scan mode override, returning the +/// previous value. +#[allow(dead_code)] // test-only scaffolding: GC unit tests (gc/tests) pin the scan mode via this override +pub(crate) fn set_conservative_stack_scan_override( + mode: Option, +) -> Option { + CONSERVATIVE_STACK_SCAN_OVERRIDE.with(|c| c.replace(mode)) +} + +/// Scoped guard forcing the conservative native-stack scan for an explicit +/// `gc()` collection (#4977). In the default `Auto` mode a full collection +/// skips the native scan, but at a `gc()` callsite live module-init/top-level +/// locals may be held only on the native stack — neither the precise +/// shadow-stack roots nor the module-var scanners cover them — so the +/// collector reclaimed live object graphs and later field reads returned +/// dangling-pointer garbage. An already-pinned per-thread override wins (the +/// GC unit tests pin `Auto` so a forced collection still reclaims objects they +/// hold only as native-stack locals), and an explicit +/// `PERRY_CONSERVATIVE_STACK_SCAN` env value beats any override either way, +/// so the bisection escape hatch keeps working. +/// +/// ★ #7148: engaging this guard is *expensive*, not merely imprecise — the +/// conservative scan makes the copying minor ineligible +/// (`CopiedMinorFallbackReason::ConservativeStack`), so the cycle it covers +/// runs no copying minor at all (`benchmarks/gc_ratchet/README.md`: +/// `heap_used_bytes` +364%..+5371%, `minor_cycles` → 0 on all eight probes). +/// Every callsite therefore declares which of the six sites it is and is +/// counted in `super::scan_fallback`, so "this never fires in practice" is a +/// measurement instead of an assumption. +pub(crate) struct ManualGcScanGuard { + engaged: bool, +} + +impl ManualGcScanGuard { + pub(crate) fn force_full_scan(site: super::ConservativeScanSite) -> Self { + super::record_scan_fallback(site); + let engaged = CONSERVATIVE_STACK_SCAN_OVERRIDE.with(|c| { + if c.get().is_some() { + return false; + } + c.set(Some(ConservativeStackScanMode::Full)); + true + }); + Self { engaged } + } +} + +impl Drop for ManualGcScanGuard { + fn drop(&mut self) { + if self.engaged { + CONSERVATIVE_STACK_SCAN_OVERRIDE.with(|c| c.set(None)); + } + } +} + +pub(crate) fn conservative_stack_scan_mode() -> ConservativeStackScanMode { + // Explicit env var wins (so the dedicated mode tests and ops bisection keep + // working), then a per-thread override, then the build-time default. + let env_mode = std::env::var("PERRY_CONSERVATIVE_STACK_SCAN") + .ok() + .map(|value| conservative_stack_scan_mode_from_value(Some(&value))); + let pinned = CONSERVATIVE_STACK_SCAN_OVERRIDE.with(|c| c.get()); + + // ★ #7148: in the TEST build a pinned override beats an env request for + // `Full`. The env var may make the scan *less* aggressive than the mode a + // test declared, never more. + // + // Why: `PERRY_CONSERVATIVE_STACK_SCAN=full` failed **134 of 1574** + // perry-runtime tests — a shipped escape hatch in a state nobody had + // verified. The failures were not soundness failures. A collector test's + // central assertion is *"this object should have been collected"*, and a + // conservative scan retains whatever the native stack happens to look like + // a pointer to, so an ambient `=full` broke exactly the assertions the + // suite exists to make. Since #7147 the test build has ONE declared scan + // mode (`Auto`) and the isolation guards are its authority; letting an + // ambient env var silently re-impose `Full` on a guarded scope reintroduces + // the test-build-is-a-different-GC-configuration problem #7147 removed. + // + // Kept rather than deleted (CLAUDE.md's kill-policy default) because + // `=full` is load-bearing *outside* the test build: it is the ratchet's + // validated sensitivity arm — the run that produced the 60 regression rows + // proving `gc-ratchet` can actually fail (`benchmarks/gc_ratchet/README.md`). + // Deleting the mode would delete a gate's only end-to-end proof. This makes + // it verified instead: the suite is green under `=full`. + #[cfg(test)] + if matches!(env_mode, Some(ConservativeStackScanMode::Full)) { + if let Some(pinned) = pinned { + return pinned; + } + } + + if let Some(mode) = env_mode { + return mode; + } + if let Some(mode) = pinned { + return mode; + } + // ONE default for every build, tests included. + // + // The test build used to default to `Full` on the grounds that unit tests + // root GC-managed pointers as raw locals on the *native* (Rust) stack + // rather than the shadow stack, so the conservative scan was what kept them + // alive. That is a description of how some tests were written, not a + // requirement of testing, and it had a cost that outweighed the + // convenience: it made the test build a DIFFERENT GC configuration from + // production. A missing precise root — the #7055 shape — was rescued by the + // native scan under test and was a live wrong answer in production, so the + // suite filtered out precisely the failure class it exists to catch. It + // also made the copying minor ineligible under test + // (`CopiedMinorFallbackReason::ConservativeStack`), so relocation was + // largely unexercised. + // + // The correct mechanism already existed and the tests already used it: + // `RuntimeHandleScope` / `gc_temp_root_*`, plus the isolation guards, which + // pin `Auto` themselves — 351 of the 531 gc tests were already running this + // way. Flipping the default cost exactly one test conversion + // (`test_minor_preserves_old_to_young_edge_across_minors`, which read a + // relocated child through a stale pre-collection local). + // + // A test that needs the scan *provably* off — not merely resolved off by + // today's `Auto` policy — should pin `ConservativeScanDisabledGuard`. + ConservativeStackScanMode::Auto +} + +#[inline] +pub(crate) fn conservative_stack_scan_decision_for( + mode: ConservativeStackScanMode, + _shadow_frame_active: bool, +) -> ConservativeStackScanDecision { + match mode { + ConservativeStackScanMode::Disabled => ConservativeStackScanDecision::SkipDisabled, + ConservativeStackScanMode::Full => ConservativeStackScanDecision::Scan, + ConservativeStackScanMode::Auto => ConservativeStackScanDecision::SkipDisabled, + } +} + +pub(crate) fn conservative_stack_scan_decision() -> ConservativeStackScanDecision { + conservative_stack_scan_decision_for( + conservative_stack_scan_mode(), + shadow_stack_has_active_frame(), + ) +} diff --git a/crates/perry-runtime/src/gc/scan_fallback.rs b/crates/perry-runtime/src/gc/scan_fallback.rs index cb22142899..b29d9b69b8 100644 --- a/crates/perry-runtime/src/gc/scan_fallback.rs +++ b/crates/perry-runtime/src/gc/scan_fallback.rs @@ -64,26 +64,30 @@ pub(crate) enum ConservativeScanSite { /// `gc_try_emergency_reclaim` — a heap allocation already failed and the /// caller is about to panic. Automatic; cannot defer (see `mod.rs`). EmergencyReclaim, - /// `js_gc_memory_pressure` with a generated frame live above us. - /// Host-driven. - HostPressure, /// `manual_gc_collect_now` — explicit `gc()`. Explicit. ManualCollect, /// `js_gc_module_minor` — explicit `perry/gc` `minor()`. Explicit. ManualMinor, + // ★ There is deliberately no `HostPressure` variant. `js_gc_memory_pressure` + // used to force the scan unconditionally; after #7148 it either collects + // with precise roots (empty shadow stack) or defers to a safepoint (a + // generated frame is live), so it has no conservative arm left to count. + // The site is not "unreached", it is **deleted** — which is the strongest + // form of what #7148 asks for, and why an unconstructed variant kept here + // "for completeness" would be the wrong kind of tidy: an enum arm nothing + // can produce is a claim no test can check. } impl ConservativeScanSite { - pub(crate) const COUNT: usize = 6; + pub(crate) const COUNT: usize = 5; const fn index(self) -> usize { match self { Self::OldReclaimSlackValve => 0, Self::NurseryChurnSlackValve => 1, Self::EmergencyReclaim => 2, - Self::HostPressure => 3, - Self::ManualCollect => 4, - Self::ManualMinor => 5, + Self::ManualCollect => 3, + Self::ManualMinor => 4, } } @@ -92,30 +96,28 @@ impl ConservativeScanSite { Self::OldReclaimSlackValve => "old_reclaim_slack_valve", Self::NurseryChurnSlackValve => "nursery_churn_slack_valve", Self::EmergencyReclaim => "emergency_reclaim", - Self::HostPressure => "host_pressure", Self::ManualCollect => "manual_collect", Self::ManualMinor => "manual_minor", } } /// Whether this site is reached without any user code calling `gc()`. - /// The four automatic sites are the ones #7148 is about: they are the + /// The automatic sites are the ones #7148 is about: they are the /// collections a program pays for without asking for them. pub(crate) const fn is_automatic(self) -> bool { match self { - Self::OldReclaimSlackValve - | Self::NurseryChurnSlackValve - | Self::EmergencyReclaim - | Self::HostPressure => true, + Self::OldReclaimSlackValve | Self::NurseryChurnSlackValve | Self::EmergencyReclaim => { + true + } Self::ManualCollect | Self::ManualMinor => false, } } + #[cfg(test)] pub(crate) const ALL: [Self; Self::COUNT] = [ Self::OldReclaimSlackValve, Self::NurseryChurnSlackValve, Self::EmergencyReclaim, - Self::HostPressure, Self::ManualCollect, Self::ManualMinor, ]; @@ -205,6 +207,7 @@ pub(crate) fn record_safepoint_drain(kind: SafepointDrainKind) { } } +#[cfg(test)] pub(crate) fn scan_fallback_count(site: ConservativeScanSite) -> u64 { SCAN_FALLBACKS.with(|c| c.get()[site.index()]) } @@ -212,6 +215,7 @@ pub(crate) fn scan_fallback_count(site: ConservativeScanSite) -> u64 { /// Total conservative-scan fallbacks across the four **automatic** sites. This /// is the number #7148 wants driven to zero: explicit `gc()` is a user request /// and is not part of the claim. +#[cfg(test)] pub(crate) fn automatic_scan_fallback_total() -> u64 { SCAN_FALLBACKS.with(|c| { let counts = c.get(); @@ -223,6 +227,7 @@ pub(crate) fn automatic_scan_fallback_total() -> u64 { }) } +#[cfg(test)] pub(crate) fn safepoint_drain_count(kind: SafepointDrainKind) -> u64 { SAFEPOINT_DRAINS.with(|c| c.get()[kind.index()]) } diff --git a/crates/perry-runtime/src/gc/tests/roots.rs b/crates/perry-runtime/src/gc/tests/roots.rs index ae2febf1e1..58a74a1583 100644 --- a/crates/perry-runtime/src/gc/tests/roots.rs +++ b/crates/perry-runtime/src/gc/tests/roots.rs @@ -816,7 +816,8 @@ fn manual_gc_scan_guard_forces_full_scan_only_when_unpinned() { // Unpinned: the guard engages a Full override for its lifetime (#4977 — // explicit gc() must see top-level locals held only on the native stack). { - let _scan = ManualGcScanGuard::force_full_scan(crate::gc::ConservativeScanSite::ManualCollect); + let _scan = + ManualGcScanGuard::force_full_scan(crate::gc::ConservativeScanSite::ManualCollect); assert_eq!( CONSERVATIVE_STACK_SCAN_OVERRIDE.with(|c| c.get()), Some(ConservativeStackScanMode::Full) @@ -828,7 +829,8 @@ fn manual_gc_scan_guard_forces_full_scan_only_when_unpinned() { // reclaim native-stack locals): the guard must not replace the override. set_conservative_stack_scan_override(Some(ConservativeStackScanMode::Auto)); { - let _scan = ManualGcScanGuard::force_full_scan(crate::gc::ConservativeScanSite::ManualCollect); + let _scan = + ManualGcScanGuard::force_full_scan(crate::gc::ConservativeScanSite::ManualCollect); assert_eq!( CONSERVATIVE_STACK_SCAN_OVERRIDE.with(|c| c.get()), Some(ConservativeStackScanMode::Auto) diff --git a/crates/perry-runtime/src/gc/tests/scan_fallback.rs b/crates/perry-runtime/src/gc/tests/scan_fallback.rs index a716abb89c..102e98cfd8 100644 --- a/crates/perry-runtime/src/gc/tests/scan_fallback.rs +++ b/crates/perry-runtime/src/gc/tests/scan_fallback.rs @@ -145,10 +145,11 @@ fn host_pressure_collects_precisely_when_no_generated_frame_is_live() { assert_eq!(result, 2, "collected synchronously"); assert_eq!( - scan_fallback_count(ConservativeScanSite::HostPressure), + automatic_scan_fallback_total(), 0, "with no generated frame live the precise root set is complete, so \ - the host-pressure collection must not force the scan" + the host-pressure collection must not force the scan (the site has no \ + conservative arm left at all — this catches its reintroduction)" ); assert_eq!( safepoint_drain_count(SafepointDrainKind::HostPressure), @@ -182,7 +183,7 @@ fn host_pressure_defers_when_a_generated_frame_is_live() { "documented return code for 'trigger lowered, collection deferred'" ); assert_eq!( - scan_fallback_count(ConservativeScanSite::HostPressure), + automatic_scan_fallback_total(), 0, "a live generated frame means a safepoint is reachable — defer to it \ instead of scanning conservatively" From 3bf3e5457f800ae47a0641f11b5d83368f7787ce Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sat, 1 Aug 2026 08:56:03 +0200 Subject: [PATCH 6/8] =?UTF-8?q?docs(gc):=20correct=20SafepointDrainKind::H?= =?UTF-8?q?ostPressure=20=E2=80=94=20it=20is=20a=20synchronous=20inline=20?= =?UTF-8?q?collection,=20not=20a=20deferred=20drain=20(CodeRabbit)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- crates/perry-runtime/src/gc/scan_fallback.rs | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/crates/perry-runtime/src/gc/scan_fallback.rs b/crates/perry-runtime/src/gc/scan_fallback.rs index b29d9b69b8..34c21cddbc 100644 --- a/crates/perry-runtime/src/gc/scan_fallback.rs +++ b/crates/perry-runtime/src/gc/scan_fallback.rs @@ -123,16 +123,25 @@ impl ConservativeScanSite { ]; } -/// A precise-root safepoint collection that ran *instead of* a conservative -/// alloc-point collection. These are the "the deferral actually drained" -/// counters — the live-subject assertion for every deferral gate. +/// A precise-root collection that ran *instead of* a conservative one. These +/// are the "the precise path actually ran" counters — the live-subject +/// assertion for every gate in this family, without which "nothing scanned" +/// would also be satisfied by "nothing collected". #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub(crate) enum SafepointDrainKind { /// A deferred nursery-pressure trigger collected at a safepoint. NurseryMinor, /// A deferred old-gen reclaim collected at a safepoint (#7148). OldReclaim, - /// A host memory-pressure request collected at a safepoint (#7148). + /// A host memory-pressure request collected **synchronously, inline in + /// `js_gc_memory_pressure`**, with precise roots because no generated frame + /// was live (#7148). Unlike the other two kinds this is not a deferred + /// drain through `js_gc_loop_safepoint` → `gc_safepoint_moving_minor`: the + /// handler is already at a precise point, and deferring there would shed + /// nothing, since a process idle enough to get an OS memory warning reaches + /// no loop back-edge and pumps no microtasks. (When a frame *is* live the + /// handler defers, and the drain is counted as `OldReclaim` or + /// `NurseryMinor` by whichever trigger it armed.) HostPressure, } From 684449a82a1966013ea65bb828b389e6a3bb3a4e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sat, 1 Aug 2026 09:06:43 +0200 Subject: [PATCH 7/8] fix(gc): site 1 is 'keep + add a competing precise path', not 'defer' The full release suite caught the design error: deferring the allocation-point old-gen reclaim breaks #5476's regression test, which asserts that a SINGLE gc_check_trigger call completes the reclaim. That guarantee exists because the workload it was filed for is a compute-only loop reaching no host step, and the bug title was 'RSS climbs unbounded'. Two reasons the deferral was wrong here: 1. The headline RSS argument does not apply. The +364%..+5371% figure is the cost of the copying minor becoming ineligible; this arm runs a FULL mark-sweep, non-moving either way. The scan costs conservative retention for one cycle, not evacuation -- much smaller, and unmeasured. 2. It trades a measured RSS guarantee for an unmeasured one. So the alloc-point arm is unchanged and counted, and gc_safepoint_moving_minor keeps the precise full mark-sweep it gained -- programs reaching a safepoint get the reclaim precisely and NO LATER than before. The fallback is attacked by adding a competing earlier precise path, not by postponing the collection. Removes the old-gen deferral state and the test-only slack override with it. Default-robustness for #7161 (proposed PERRY_GC_MOVING_LOOP_POLLS flip to OFF), now tested in both directions: - old-gen alloc-point arm: does not read the gate at all -> identical behaviour and identical census under either default. - host pressure precise inline collection: does not read the gate -> unaffected. - host pressure deferral: js_gc_loop_safepoint goes no-op, but the lowered arena trigger still makes the next allocation-point check collect the owed cycle; test asserts that backstop directly. - nursery-churn valve: pre-existing code already skips the deferral when polls are off, so every nursery collection scans conservatively -- inert and sound, but the census now makes that cost visible. --- crates/perry-runtime/src/gc/policy.rs | 161 ++++------------ crates/perry-runtime/src/gc/scan_fallback.rs | 18 +- .../src/gc/tests/scan_fallback.rs | 179 ++++++++++++------ 3 files changed, 170 insertions(+), 188 deletions(-) diff --git a/crates/perry-runtime/src/gc/policy.rs b/crates/perry-runtime/src/gc/policy.rs index 7ba914bcd2..f0af8ce5de 100644 --- a/crates/perry-runtime/src/gc/policy.rs +++ b/crates/perry-runtime/src/gc/policy.rs @@ -661,75 +661,6 @@ thread_local! { /// last set — the baseline the deferral slack is measured from (#7024). /// Meaningless while `GC_SAFEPOINT_PENDING` is false. pub(super) static GC_SAFEPOINT_DEFER_ARENA_BASE: Cell = const { Cell::new(0) }; - /// #7148: an old-gen reclaim is owed at the next precise-root safepoint. - /// Distinct from `GC_SAFEPOINT_PENDING` (which the nursery arm also uses) - /// because the two are bounded in different units — nursery pressure by - /// committed arena bytes, old-gen reclaim by old-gen in-use bytes — and a - /// slack measured in the wrong unit is a slack that never expires. (The - /// old-gen arm also sets `GC_SAFEPOINT_PENDING`, because that is the flag - /// `js_gc_loop_safepoint` polls.) - pub(super) static GC_SAFEPOINT_OLD_RECLAIM_PENDING: Cell = const { Cell::new(false) }; - /// Old-gen in-use bytes (plus external side-table bytes — the same quantity - /// `gc_budgeted_due_trigger` measures) sampled when the old-gen reclaim was - /// deferred. Meaningless while `GC_SAFEPOINT_OLD_RECLAIM_PENDING` is false. - pub(super) static GC_SAFEPOINT_OLD_RECLAIM_BASE: Cell = const { Cell::new(0) }; -} - -/// The quantity `gc_budgeted_due_trigger` compares against the old-gen reclaim -/// thresholds. Factored out so the #7148 deferral slack is measured in exactly -/// the same unit as the trigger that armed it — a slack in the wrong unit is a -/// slack that never expires (the #7024 shape, inverted). -#[inline] -pub(super) fn old_reclaim_pressure_bytes() -> usize { - crate::arena::old_gen_in_use_bytes().saturating_add(external_side_live_bytes()) -} - -#[cfg(test)] -thread_local! { - /// Test-only override for the #7148 old-gen deferral slack. The shipped - /// slack is `gc_old_gen_reclaim_growth_dyn_bytes()` — 32 MB on an - /// unbudgeted host — and a test that wanted to watch the safety valve fire - /// would otherwise have to commit 32 MB of old-gen just to cross it. - /// Shrinking the slack lets the valve test drive the *real* branch in - /// `gc_check_trigger` deterministically and in microseconds. - static GC_OLD_RECLAIM_DEFER_SLACK_TEST_OVERRIDE: Cell> = - const { Cell::new(None) }; -} - -/// Growth allowance, in the old-gen trigger's own unit, that a deferred -/// old-gen reclaim may consume past its deferral point before the conservative -/// safety valve fires (#7148). -#[inline] -pub(super) fn old_reclaim_defer_slack_bytes() -> usize { - #[cfg(test)] - if let Some(slack) = GC_OLD_RECLAIM_DEFER_SLACK_TEST_OVERRIDE.with(Cell::get) { - return slack; - } - gc_old_gen_reclaim_growth_dyn_bytes() -} - -/// Scoped test-only override of the old-gen deferral slack. See -/// [`GC_OLD_RECLAIM_DEFER_SLACK_TEST_OVERRIDE`]. -#[cfg(test)] -pub(super) struct OldReclaimDeferSlackGuard { - previous: Option, -} - -#[cfg(test)] -impl Drop for OldReclaimDeferSlackGuard { - fn drop(&mut self) { - GC_OLD_RECLAIM_DEFER_SLACK_TEST_OVERRIDE.with(|cell| cell.set(self.previous)); - } -} - -#[cfg(test)] -pub(super) fn force_old_reclaim_defer_slack(slack: usize) -> OldReclaimDeferSlackGuard { - let previous = GC_OLD_RECLAIM_DEFER_SLACK_TEST_OVERRIDE.with(|cell| { - let previous = cell.get(); - cell.set(Some(slack)); - previous - }); - OldReclaimDeferSlackGuard { previous } } /// Committed arena bytes a deferred nursery trigger may allocate **past the @@ -1374,36 +1305,42 @@ pub fn gc_check_trigger() { // allocation point (e.g. the temporary currently being built) is retained; // only genuinely unreachable old blocks are returned. // - // ★ #7148 disposition: **defer.** The scan is what makes this arm sound, - // and it is also what makes it ruinous — a scanning cycle retains whatever - // the native stack looks like a pointer to and makes the copying minor - // ineligible for the rest of the cycle. So do here what the nursery arm - // already does: arm a deferral and let `gc_safepoint_moving_minor` run the - // SAME full mark-sweep at the next precise-root safepoint (loop back-edge - // poll or the outermost microtask-pump boundary), where no forced scan is - // needed at all. The conservative arm below survives only as the bounded - // safety valve for the case the deferral does not drain. + // ★ #7148 disposition: **keep, justified, observable — and add a precise + // path that beats it to the punch.** The first attempt at this site was a + // deferral like the nursery arm's. It was wrong, and the reasoning is kept + // because the shape recurs. + // + // 1. **The headline RSS argument does not apply here.** A conservative scan + // costs +364%..+5371% `heap_used_bytes` on the ratchet probes *because + // it makes the copying minor ineligible* — `minor_cycles` → 0. This arm + // runs a FULL mark-sweep, which is non-moving with or without the scan. + // What the scan costs here is conservative *retention* for one cycle, + // not the loss of evacuation. Much smaller, and unmeasured. + // 2. **Deferring it breaks a tested RSS guarantee.** #5476's regression test + // (`check_trigger_drives_old_reclaim_to_completion_without_host_stepping`) + // asserts that *a single* `gc_check_trigger` call — what every allocation + // does — drives the reclaim to completion, because the workload that + // motivated it is a compute-only loop that never reaches a host step. A + // deferral makes that "within one 32 MB growth quantum" instead, on the + // exact workload whose bug report was titled *RSS climbs unbounded*. + // Trading conservative retention for bounded-but-real extra old-gen + // residency is not obviously a win, and nothing here measured that it is. // - // **What if pressure spikes before a safepoint is reached?** That is the - // question this site exists to answer, so it is answered in the trigger's - // own unit rather than hand-waved. The deferral is allowed only while - // old-gen pressure stays within `gc_old_gen_reclaim_growth_dyn_bytes()` - // (32 MB by default — one growth quantum, the same delta that armed the - // trigger) of where it stood when the collection was deferred. Past that, - // this arm runs exactly as it did before #7148, counted as - // `ConservativeScanSite::OldReclaimSlackValve`. So the worst case is a - // bounded one growth-quantum of extra old-gen residency before the valve - // fires, and #5476's "RSS climbs unbounded" property is preserved: there - // is no path on which pressure grows without a collection. + // So this arm is unchanged, and `gc_safepoint_moving_minor` instead gained + // the SAME full mark-sweep with precise roots (#7148). Programs that reach a + // safepoint — every event-loop program — now get their old-gen reclaim + // precisely and *no later* than before; nothing is delayed for anyone. The + // fallback is attacked by adding a competing earlier precise path, not by + // postponing the collection. `ConservativeScanSite::OldReclaimAllocPoint` + // counts how often the alloc point still gets there first. // - // ★ The slack is a DELTA in the trigger's own unit, deliberately. #7024 - // burned this project once: an absolute cap that shared a formula with the - // trigger ceiling made "a trigger is due" and "the deferral is allowed" - // exact complements, and the branch became dead. Measuring old-gen slack in - // *arena* bytes would be the same bug wearing different clothes — an - // old-gen-only workload (>16 KB temporaries, born straight into the old - // arena; #5476's exact shape) barely moves `arena_total_bytes()`, so an - // arena-unit slack would never expire and the valve would never fire. + // Default-robustness (#7161 proposes flipping `PERRY_GC_MOVING_LOOP_POLLS` + // OFF): this arm does not consult that gate at all, so it behaves + // identically either way. The precise safepoint path is reached from the + // microtask pump under `gc_moving_safepoint_enabled` (a different knob, + // untouched by #7161) and from `js_gc_loop_safepoint` only while polls are + // on. Polls off ⇒ the precise path is reached less often ⇒ this arm fires + // more often ⇒ the census counter rises. Inert, not unsound. if !gc_budgeted_cycle_active() && matches!( gc_budgeted_due_trigger(), @@ -1411,37 +1348,10 @@ pub fn gc_check_trigger() { ) && !GC_OLD_RECLAIM_IN_PROGRESS.with(Cell::get) { - if gc_moving_loop_polls_enabled() || gc_moving_safepoint_enabled() { - let old_pressure = old_reclaim_pressure_bytes(); - let already_deferred = GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(Cell::get); - let deferred_at = - already_deferred.then(|| GC_SAFEPOINT_OLD_RECLAIM_BASE.with(Cell::get)); - if moving_defer_within_slack(old_pressure, deferred_at, old_reclaim_defer_slack_bytes()) - { - if !already_deferred { - GC_SAFEPOINT_OLD_RECLAIM_BASE.with(|base| base.set(old_pressure)); - GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(|p| p.set(true)); - } - // `GC_SAFEPOINT_PENDING` is the flag `js_gc_loop_safepoint` - // polls, so the loop back-edge drains this too. Keep its arena - // baseline coherent for the nursery arm's own slack test. - if !GC_SAFEPOINT_PENDING.with(Cell::get) { - GC_SAFEPOINT_DEFER_ARENA_BASE - .with(|base| base.set(crate::arena::arena_total_bytes())); - GC_SAFEPOINT_PENDING.with(|p| p.set(true)); - } - return; - } - // The deferral never drained; the collection below IS the one that - // was owed. Retire the request so a stale, already-exceeded - // baseline cannot disable deferral for the rest of the process - // (#7024's shape). - GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(|p| p.set(false)); - } let _reentry = OldReclaimReentryGuard::enter(); GC_OLD_RECLAIM_PENDING.with(|pending| pending.set(false)); let _scan = super::roots::ManualGcScanGuard::force_full_scan( - super::ConservativeScanSite::OldReclaimSlackValve, + super::ConservativeScanSite::OldReclaimAllocPoint, ); gc_collect_full_mark_sweep_with_trigger(GcTriggerSnapshot::capture( GcTriggerKind::OldGenBytes, @@ -1783,7 +1693,6 @@ pub(crate) fn gc_safepoint_moving_minor() { } let _reentry = OldReclaimReentryGuard::enter(); GC_OLD_RECLAIM_PENDING.with(|pending| pending.set(false)); - GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(|p| p.set(false)); // No `force_full_scan`: roots are precise at this safepoint. gc_collect_full_mark_sweep_with_trigger(GcTriggerSnapshot::capture( GcTriggerKind::OldGenBytes, diff --git a/crates/perry-runtime/src/gc/scan_fallback.rs b/crates/perry-runtime/src/gc/scan_fallback.rs index 34c21cddbc..d83347be7a 100644 --- a/crates/perry-runtime/src/gc/scan_fallback.rs +++ b/crates/perry-runtime/src/gc/scan_fallback.rs @@ -55,9 +55,13 @@ use std::cell::Cell; /// A `force_full_scan()` callsite. Ordering matters only for the counter array. #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub(crate) enum ConservativeScanSite { - /// `gc_check_trigger` old-gen reclaim, at an allocation point, after the - /// safepoint deferral ran out of slack. Automatic. - OldReclaimSlackValve, + /// `gc_check_trigger` old-gen reclaim, at an allocation point. Automatic. + /// Deliberately NOT deferred — #5476 requires a single `gc_check_trigger` + /// call to complete the reclaim, and delaying it would regress the RSS bug + /// that arm exists for. What #7148 added is a competing PRECISE path at + /// safepoints (`SafepointDrainKind::OldReclaim`); this counter is how often + /// the allocation point still got there first. + OldReclaimAllocPoint, /// `gc_check_trigger` nursery-churn direct minor, at an allocation point, /// after the safepoint deferral ran out of slack. Automatic. NurseryChurnSlackValve, @@ -83,7 +87,7 @@ impl ConservativeScanSite { const fn index(self) -> usize { match self { - Self::OldReclaimSlackValve => 0, + Self::OldReclaimAllocPoint => 0, Self::NurseryChurnSlackValve => 1, Self::EmergencyReclaim => 2, Self::ManualCollect => 3, @@ -93,7 +97,7 @@ impl ConservativeScanSite { pub(crate) const fn as_str(self) -> &'static str { match self { - Self::OldReclaimSlackValve => "old_reclaim_slack_valve", + Self::OldReclaimAllocPoint => "old_reclaim_alloc_point", Self::NurseryChurnSlackValve => "nursery_churn_slack_valve", Self::EmergencyReclaim => "emergency_reclaim", Self::ManualCollect => "manual_collect", @@ -106,7 +110,7 @@ impl ConservativeScanSite { /// collections a program pays for without asking for them. pub(crate) const fn is_automatic(self) -> bool { match self { - Self::OldReclaimSlackValve | Self::NurseryChurnSlackValve | Self::EmergencyReclaim => { + Self::OldReclaimAllocPoint | Self::NurseryChurnSlackValve | Self::EmergencyReclaim => { true } Self::ManualCollect | Self::ManualMinor => false, @@ -115,7 +119,7 @@ impl ConservativeScanSite { #[cfg(test)] pub(crate) const ALL: [Self; Self::COUNT] = [ - Self::OldReclaimSlackValve, + Self::OldReclaimAllocPoint, Self::NurseryChurnSlackValve, Self::EmergencyReclaim, Self::ManualCollect, diff --git a/crates/perry-runtime/src/gc/tests/scan_fallback.rs b/crates/perry-runtime/src/gc/tests/scan_fallback.rs index 102e98cfd8..e0269aff3f 100644 --- a/crates/perry-runtime/src/gc/tests/scan_fallback.rs +++ b/crates/perry-runtime/src/gc/tests/scan_fallback.rs @@ -25,104 +25,102 @@ fn arm_old_reclaim() { fn clear_old_reclaim_state() { GC_OLD_RECLAIM_PENDING.with(|pending| pending.set(false)); - GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(|pending| pending.set(false)); GC_SAFEPOINT_PENDING.with(|pending| pending.set(false)); let old_in_use = crate::arena::old_gen_in_use_bytes(); GC_LAST_OLD_RECLAIM_IN_USE_BYTES.with(|bytes| bytes.set(old_in_use)); } #[test] -fn old_reclaim_defers_to_a_precise_safepoint_instead_of_scanning() { +fn old_reclaim_runs_precisely_at_a_safepoint() { let _isolation = GcTestIsolationGuard::new(); let _pacing = crate::gc::policy::force_moving_gc_pacing(); let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); clear_old_reclaim_state(); reset_scan_fallback_counters(); + // `gc_safepoint_moving_minor` used to bail on OldReclaim ("stays on its + // existing full mark-sweep path"), so the ONLY place an old-gen reclaim + // could happen was the allocation point, behind `force_full_scan()`. arm_old_reclaim(); - gc_check_trigger(); - - // Before #7148 this call ran a full mark-sweep here, at the allocation - // point, behind `force_full_scan()`. - assert_eq!( - scan_fallback_count(ConservativeScanSite::OldReclaimSlackValve), - 0, - "old-gen reclaim must defer, not scan, while it is within slack" - ); - assert!( - GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(std::cell::Cell::get), - "the deferral must be recorded in the old-gen unit" - ); - assert!( - GC_SAFEPOINT_PENDING.with(std::cell::Cell::get), - "`js_gc_loop_safepoint` polls GC_SAFEPOINT_PENDING, so the old-gen \ - deferral must set it too or nothing drains the request" - ); - assert!( - GC_OLD_RECLAIM_PENDING.with(std::cell::Cell::get), - "deferring must not retire the request — the trigger has to stay due \ - so the safepoint finds it" - ); - - // Drain at the precise safepoint the deferral promised. + GC_SAFEPOINT_PENDING.with(|p| p.set(true)); js_gc_loop_safepoint(); assert_eq!( safepoint_drain_count(SafepointDrainKind::OldReclaim), 1, - "LIVE SUBJECT: the deferred full mark-sweep must actually have run at \ - the safepoint — 'nothing scanned' is worthless if nothing collected" + "LIVE SUBJECT: the full mark-sweep must actually have run at the \ + safepoint — 'nothing scanned' is worthless if nothing collected" ); assert_eq!( - scan_fallback_count(ConservativeScanSite::OldReclaimSlackValve), + automatic_scan_fallback_total(), 0, - "the safepoint path must not force the scan" + "the safepoint path has precise roots, so it must not force the scan" ); assert!( !GC_OLD_RECLAIM_PENDING.with(std::cell::Cell::get), - "the safepoint collection retires the request" - ); - assert!( - !GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(std::cell::Cell::get), - "the safepoint collection clears its own deferral flag" + "the safepoint collection retires the request, so the allocation-point \ + arm finds nothing due and never runs a second, conservative cycle" ); clear_old_reclaim_state(); } #[test] -fn old_reclaim_slack_valve_fires_when_the_deferral_never_drains() { +fn old_reclaim_alloc_point_still_completes_immediately_and_is_counted() { let _isolation = GcTestIsolationGuard::new(); let _pacing = crate::gc::policy::force_moving_gc_pacing(); let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); clear_old_reclaim_state(); reset_scan_fallback_counters(); - // The other direction. A deferral whose slack cannot expire is #7024's bug - // wearing new clothes: RSS grows without bound and the branch that was - // supposed to bound it is dead. The shipped slack is 32 MB, so crossing it - // for real would mean committing 32 MB of old-gen inside a unit test; - // shrink it instead so the *real* branch in `gc_check_trigger` runs - // deterministically. Slack 0 with the baseline at the current pressure is - // the exact state "this deferral has used up its whole allowance". - let _slack = crate::gc::policy::force_old_reclaim_defer_slack(0); + // The other direction, and the reason this site is NOT deferred. #5476's + // own regression test asserts that a single `gc_check_trigger` call — what + // every allocation does — drives the reclaim to completion, because the + // workload it was filed for is a compute-only loop that reaches no host + // step. Deferring here would turn "RSS climbs unbounded" back on for one + // growth quantum. What #7148 changes is only that the cost is now visible. + let collections_before = gc_collection_count(); arm_old_reclaim(); - let pressure = old_reclaim_pressure_bytes(); - GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(|pending| pending.set(true)); - GC_SAFEPOINT_OLD_RECLAIM_BASE.with(|base| base.set(pressure)); - gc_check_trigger(); + assert!( + gc_collection_count() > collections_before, + "a single gc_check_trigger call must still complete the reclaim (#5476)" + ); assert_eq!( - scan_fallback_count(ConservativeScanSite::OldReclaimSlackValve), + scan_fallback_count(ConservativeScanSite::OldReclaimAllocPoint), 1, - "past the slack the conservative valve MUST fire — it is the only \ - thing bounding old-gen growth when no safepoint is reachable" + "and it is still a conservative-scan collection — counted, not hidden" ); + + clear_old_reclaim_state(); +} + +#[test] +fn old_reclaim_is_unchanged_when_moving_loop_polls_are_off() { + // #7161 proposes flipping `PERRY_GC_MOVING_LOOP_POLLS` OFF as a stopgap for + // #7154. The allocation-point arm must not consult that gate at all, so + // that the flip can never make this site behave differently — inert, not + // unsound. (The precise safepoint path is simply reached less often, which + // shows up as a higher census count, not as a correctness change.) + let _isolation = GcTestIsolationGuard::new(); + let _pacing = crate::gc::policy::force_legacy_gc_pacing(); + let _triggers = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + clear_old_reclaim_state(); + reset_scan_fallback_counters(); + + let collections_before = gc_collection_count(); + arm_old_reclaim(); + gc_check_trigger(); + assert!( - !GC_SAFEPOINT_OLD_RECLAIM_PENDING.with(std::cell::Cell::get), - "the valve retires the deferral; a stale exceeded baseline would \ - disable deferral for the rest of the process" + gc_collection_count() > collections_before, + "polls off must not stop the allocation-point reclaim completing" + ); + assert_eq!( + scan_fallback_count(ConservativeScanSite::OldReclaimAllocPoint), + 1, + "identical behaviour and identical census to the polls-on arm" ); clear_old_reclaim_state(); @@ -289,3 +287,74 @@ fn conservative_scan_env_off_still_beats_a_forced_scan() { crate::gc::roots::set_conservative_stack_scan_override(previous); } + +#[test] +fn host_pressure_precise_collection_does_not_depend_on_moving_loop_polls() { + // The host-pressure win that matters — collecting with precise roots when + // no generated frame is live — is reached without consulting + // `PERRY_GC_MOVING_LOOP_POLLS` at all, so #7161's proposed flip cannot make + // this site scan conservatively again. + let _isolation = GcTestIsolationGuard::new(); + let _pacing = crate::gc::policy::force_legacy_gc_pacing(); + clear_old_reclaim_state(); + reset_shadow_stack(); + reset_scan_fallback_counters(); + + let result = js_gc_memory_pressure(1); + + assert_eq!(result, 2, "collected synchronously with polls off"); + assert_eq!( + automatic_scan_fallback_total(), + 0, + "still no conservative scan with polls off" + ); + assert_eq!( + safepoint_drain_count(SafepointDrainKind::HostPressure), + 1, + "LIVE SUBJECT: the precise collection ran" + ); + + clear_old_reclaim_state(); +} + +#[test] +fn host_pressure_deferral_still_has_a_drain_when_moving_loop_polls_are_off() { + // With polls off `js_gc_loop_safepoint` is a no-op, so the frame-live + // deferral must not depend on it. Two backstops survive #7161: the + // outermost microtask-pump boundary (gated by `PERRY_GC_MOVING_SAFEPOINT`, + // a different knob) and the lowered arena trigger, which makes the ordinary + // allocation-point arm collect at the next check. This test pins the second + // one, because it is the one that holds even for a program that never + // yields to the event loop. + let _isolation = GcTestIsolationGuard::new(); + let _pacing = crate::gc::policy::force_legacy_gc_pacing(); + clear_old_reclaim_state(); + reset_shadow_stack(); + reset_scan_fallback_counters(); + + let frame = js_shadow_frame_push(2); + let result = js_gc_memory_pressure(2); + js_shadow_frame_pop(frame); + + assert_eq!(result, 1, "deferred, as with polls on"); + assert_eq!( + automatic_scan_fallback_total(), + 0, + "deferring never scans, whatever the pacing mode" + ); + assert!( + GC_OLD_RECLAIM_PENDING.with(std::cell::Cell::get), + "the owed FULL cycle is still armed, so the next allocation-point \ + trigger check collects it even with no safepoint machinery running" + ); + + // The polls-off backstop: an ordinary trigger check completes it. + let collections_before = gc_collection_count(); + gc_check_trigger(); + assert!( + gc_collection_count() > collections_before, + "LIVE SUBJECT: the deferred critical-pressure cycle actually ran" + ); + + clear_old_reclaim_state(); +} From 55a6f6fa5caa567b084be3d60197e08bffbca59d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Sat, 1 Aug 2026 09:11:15 +0200 Subject: [PATCH 8/8] changelog: update fragment for the reworked site-1 disposition and the #7161 interaction --- .../7166-gc-safepoint-deferral-scan-census.md | 20 +++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/changelog.d/7166-gc-safepoint-deferral-scan-census.md b/changelog.d/7166-gc-safepoint-deferral-scan-census.md index 2ce057a646..00c0e751b2 100644 --- a/changelog.d/7166-gc-safepoint-deferral-scan-census.md +++ b/changelog.d/7166-gc-safepoint-deferral-scan-census.md @@ -1,15 +1,19 @@ -fix(gc): defer old-gen reclaim and OS memory-pressure collections to precise safepoints, and put a census on every conservative-scan fallback (#7148). +fix(gc): run old-gen reclaim precisely at safepoints, stop scanning conservatively on OS memory pressure, and put a census on every conservative-scan fallback (#7148). -`main` reached a forced conservative native-stack scan from **six** sites, **four of them without any user calling `gc()`**. That is not merely an imprecision: a forced scan makes the copying minor ineligible (`CopiedMinorFallbackReason::ConservativeStack`), so the cycle it covers runs **no copying minor at all**. `benchmarks/gc_ratchet/README.md` measures the end state — `heap_used_bytes` **+364% to +5371%** and `minor_cycles` → **0 on all eight probes**. Every automatic fallback that fired was worth that much RSS. +`main` reached a forced conservative native-stack scan from **six** sites, **four of them without any user calling `gc()`**, and nothing measured whether any of them ever ran. A forced scan is not merely an imprecision: it makes the copying minor ineligible (`CopiedMinorFallbackReason::ConservativeStack`), so the cycle it covers runs **no copying minor at all**. `benchmarks/gc_ratchet/README.md` quantifies the end state — `heap_used_bytes` **+364% to +5371%** and `minor_cycles` → **0 on all eight probes**. -The right end state (#7148) is that the fallback becomes *unreachable* — collections always defer to a precise safepoint — not *imprecise*, which would trade a cost problem for a soundness problem. Per-site: +#7148 asks for the fallback to become *unreachable*, not *imprecise*. Per site: -- **Old-gen reclaim (`gc/policy.rs`) — deferred.** `gc_safepoint_moving_minor` used to bail on `OldReclaim`; it now runs the same full mark-sweep at the safepoint with `SkipDisabled` roots and no forced scan. The allocation-point arm keeps only a bounded safety valve, and the valve's slack is a **delta in the trigger's own unit** (old-gen in-use + external side-table bytes; one 32 MB growth quantum). That unit choice is load-bearing: #5476's workload is >16 KB temporaries born straight into the old arena, which barely move `arena_total_bytes()`, so an arena-unit slack would never expire and the valve would never fire — #7024's dead-branch bug inverted. -- **OS memory pressure (`gc/pressure.rs`) — deferred, conditionally.** The old rationale ("a host may deliver the callback with unspilled locals on the native frames above us") is now *tested* rather than assumed. With no active shadow frame the precise root set is complete, so the collection runs here with no scan and the copying minor stays eligible — the run-loop-boundary case the module's own docs call typical, and the case where deferring would be worst, since an idle process reaches no safepoint and would shed nothing before the jetsam. With a generated frame live a safepoint *is* reachable, so the request is deferred and the handler returns `1`, the code this API already documents as "trigger lowered, collection deferred". The lowered arena trigger is untouched either way, so no backstop is removed. -- **Nursery-churn valve, emergency reclaim, `gc()`, `perry/gc` `minor()` — kept, now counted.** Emergency reclaim provably cannot defer: it runs after an allocation has already failed and the caller's next act is to panic, so there is no next safepoint to defer to. The two explicit sites are user requests with synchronous semantics. +- **Old-gen reclaim (`gc/policy.rs`) — kept, and given a competing precise path.** `gc_safepoint_moving_minor` used to bail on `OldReclaim` ("stays on its existing full mark-sweep path"); it now runs the same full mark-sweep there with `SkipDisabled` roots and no forced scan, so every program that reaches a safepoint gets its old-gen reclaim precisely and **no later than before**. The allocation-point arm is deliberately **not** deferred. An earlier revision of this change did defer it and the release suite caught it: #5476's regression test asserts that *a single* `gc_check_trigger` call — what every allocation does — completes the reclaim, because the workload it was filed for is a compute-only loop that reaches no host step. Two reasons that verdict is right rather than merely awkward: the headline RSS figure is the cost of the *copying minor* becoming ineligible, and this arm runs a full mark-sweep that is non-moving with or without the scan (the scan costs one cycle of conservative retention, which is much smaller and was not measured); and deferring trades a measured RSS guarantee — #5476's title is *RSS climbs unbounded* — for an unmeasured one. So the fallback is attacked by adding an earlier precise path, not by postponing the collection. +- **OS memory pressure (`gc/pressure.rs`) — the conservative arm is deleted outright.** The old rationale ("a host may deliver the callback with unspilled locals on the native frames above us") is now *tested* rather than assumed. With no active shadow frame the precise root set is complete, so the collection runs inline with no scan and the copying minor stays eligible — the run-loop-boundary case the module's own docs call typical, and the case where deferring would be worst, since a process idle enough to get an OS memory warning reaches no loop back-edge and pumps no microtasks. With a generated frame live a safepoint *is* reachable, so the request is deferred and the handler returns `1`, the code this API already documents as "trigger lowered, collection deferred". The lowered arena trigger is untouched either way, so no backstop is removed. `ConservativeScanSite` has no `HostPressure` variant as a result: an enum arm nothing can produce is a claim no test can check. +- **Nursery-churn valve, emergency reclaim, `gc()`, `perry/gc` `minor()` — kept, now counted.** Emergency reclaim provably cannot defer: it runs after an allocation has already failed and the caller's next act is to panic, so there is no next safepoint. The two explicit sites are user requests with synchronous semantics. -New `gc/scan_fallback.rs` records every fallback (`[gc-scan-fallback] site=… automatic=… count=…` under `PERRY_GC_DIAG`) and every precise-safepoint drain that replaced one (`[gc-safepoint-drain] kind=…`), separating automatic from explicit sites — all eight ratchet probes end with an explicit full `gc()`, so that site fires once per probe by construction and an undifferentiated census would misread. This is the reachability census for the invariant the statepoint experiment states and enforces (`docs/statepoint-gc-experiment.md`, `exp/stackmap-viability`): *a collection that skips the conservative scan consumes only precise roots, so it may only begin at a declared safepoint.* Census first, enforcement second. +New `gc/scan_fallback.rs` records every fallback (`[gc-scan-fallback] site=… automatic=… count=…` under `PERRY_GC_DIAG`) and every precise collection that replaced one (`[gc-safepoint-drain] kind=…`), separating automatic from explicit sites — all eight ratchet probes end with an explicit full `gc()`, so that site fires once per probe by construction and an undifferentiated census would misread. This is the reachability census for the invariant the statepoint experiment states and enforces (`docs/statepoint-gc-experiment.md` on `exp/stackmap-viability`): *a collection that skips the conservative scan consumes only precise roots, so it may only begin at a declared safepoint.* Census first, enforcement second. + +Every claim is tested under **both** `PERRY_GC_MOVING_LOOP_POLLS` defaults, since #7161 proposes flipping it OFF: the old-gen arm and the host-pressure precise collection never read that gate, and the host-pressure deferral keeps two gate-independent drains (the microtask-pump boundary, and the lowered arena trigger — the latter pinned by its own test because it is the one that holds for a program that never yields to the event loop). Site 2's dependence on the gate is pre-existing and unchanged; with polls off every nursery collection scans conservatively, which the census now makes visible rather than assumed. Also fixes `PERRY_CONSERVATIVE_STACK_SCAN=full`, which failed **134 of 1574** runtime tests on `main` — a shipped escape hatch nobody had verified. Fixed rather than deleted, against the kill-policy's default, because `=full` is `gc-ratchet`'s validated sensitivity arm: the run that produced the 60 regression rows proving that gate can fail. Deleting it would delete a gate's only end-to-end proof. The failures were never soundness failures — a collector test's central assertion is *"this object should have been collected"*, and an ambient conservative scan retains whatever the native stack looks like a pointer to. Since #7147 the test build has one declared scan mode and the isolation guards are its authority, so **in the test build only** a pinned override now beats an env request for `Full`; the env var may make the scan less aggressive than a test declared, never more. `=0` still beats a pinned `Full`, so the bisection hatch is intact. Production binaries pin no override and are unchanged. -Every deferral test asserts both that the conservative valve did not fire and that the precise collection which replaced it actually ran (a drain counter) — checking only the first half would pass on a tree where the trigger never armed at all, the largest possible regression reported as a pass. Verified red-then-green with five one-hunk sabotages (deferral removed, safepoint drain removed, host pressure reverted, slack valve removed, `=full` precedence reverted), each reddening exactly the tests that name the property it broke. +Every test asserts both that no conservative scan fired and that the precise collection which replaced it actually ran (a drain counter) — checking only the first half would pass on a tree where the trigger never armed at all, the largest possible regression reported as a pass. Verified with one-hunk sabotages in both directions; the sharpest is reintroducing the rejected old-gen deferral, which reddens #5476's own pre-existing test as well as the new one. + +`gc/roots.rs` crossed the 2,000-line file-size gate as a side effect, so the conservative-scan mode block moved to `gc/roots/scan_mode.rs` (1,992 → 1,851 lines).