fix(gc): measure the moving-defer allowance from the deferral point so the copying minor is reachable under a heap budget

changelog.d/7057-gc-moving-defer-slack.md

@@ -0,0 +1,54 @@
+Fixed the moving-GC deferral so the copying minor is reachable under an explicit
+heap budget, and made the per-PR GC matrix arms assert that it actually ran.
+
+`gc_check_trigger`'s deferral arm — the one that hands an allocation-point
+nursery trigger to the next precise-root safepoint, and therefore to the copying
+minor #7019 shipped — was guarded by an absolute committed-arena cap derived from
+`budget_scaled(128 MB, 1, 4, 2 MB)`. That is byte-for-byte the formula behind
+`gc_trigger_absolute_ceiling_bytes()`, so under any heap budget small enough for
+the ceiling to reach the 16 MB nursery cap (every `PERRY_GC_HEAP_LIMIT` ≤ 64, and
+every device budget a small container or watch-class device derives) the two
+collapse to one number. A nursery trigger is due exactly when
+`arena_total_bytes() >= trigger` while the deferral required
+`arena_total_bytes() < cap`: same number, so the two predicates were exact
+complements and the deferral was unreachable. Control fell through to the
+alloc-point minor under `ManualGcScanGuard::force_full_scan()`, the collector
+reported `[gc-copy-minor] eligible=false fallback=conservative_stack`, and a
+heap-limited deployment silently ran the pre-#7019 non-moving collector. Measured
+on the representation corpus at `--pressure 8`: the `default` arm collected on 13
+of 22 rows and ran **zero** copying minors on all 22.
+
+The allowance is now a slack measured **from the deferral point**
+(`GC_MOVING_DEFER_SLACK_BYTES`, `gc_moving_defer_slack_dyn_bytes()`): the first
+deferral of a cycle is unconditional, and the safety valve fires once the arena
+has grown one slack past it, retiring the pending request so the baseline cannot
+go stale and pin the deferral off for the rest of the process. A delta cannot
+collapse into an absolute trigger at any heap budget. No env knob was added.
+
+The `default` arm now runs a real copying minor — `[gc-copy-minor] ran
+copied_objects=3576 … eligible=true fallback=none` on
+`test_gap_repsel_canonical_i32`, 8 604 215 objects over 154 copying minors on
+`test_gap_repsel_gc_stress` — 12 of 22 corpus rows, up from 0. Full matrix
+(`--arms all --pressure 8`, 440 cells) moves `PASS=324 UNVER=91 XFAIL=1 FAIL=24`
+to `PASS=325 UNVER=100 XFAIL=1 FAIL=14`; all 14 residual failures are the single
+pre-existing #6981 cell `test_gap_repsel_p4a3_numarray_barriers`, and no corpus
+row broke that was not already red.
+
+That also closes the per-PR gate hole in #6993: the relocating-minor defect class
+was previously invisible to `--arms pr`, and the proof it no longer is, is a cell
+that changed colour — `default × test_gap_repsel_p4a3_numarray_barriers` went
+from PASS (`cycles=1 scavenged=0`) to FAIL (`exit=139 scavenged=3594`), the same
+SIGSEGV only the push-only evacuating arms could produce before.
+
+Gate mechanics hardened alongside: a `scavenge` liveness requirement reads the
+copying minor's own `[gc-copy-minor] ran copied_objects=` counter rather than the
+sum that also counts the C4b mark-sweep evacuation (#7025), and is given to the
+four arms whose subject is the relocating young-gen minor. A compiled-program
+regression test (`crates/perry/tests/gc_copy_minor_under_heap_limit.rs`) pins the
+observable that distinguishes the two worlds under `PERRY_GC_HEAP_LIMIT=8`:
+`copied_objects > 0`. Exit 0 does not — the broken build exited 0 and collected.
+
+One measured consequence is recorded rather than smoothed over:
+`test_gap_specabi_reassign`, a 20-line program with no loop and therefore no
+back-edge poll to drain the deferral, now exits before collecting under the
+pressure knob and is reported UNVER instead of PASS.

crates/perry-runtime/src/gc/heap_budget.rs

@@ -6,7 +6,7 @@
 use std::sync::OnceLock;
 
 use super::policy::{
-    GC_COPY_PROMOTION_HANDOFF_MIN_BYTES, GC_MOVING_DEFER_HARD_CAP_BYTES,
+    GC_COPY_PROMOTION_HANDOFF_MIN_BYTES, GC_MOVING_DEFER_SLACK_BYTES,
     GC_OLD_GEN_RECLAIM_GROWTH_BYTES, GC_OLD_GEN_RECLAIM_THRESHOLD_BYTES,
     GC_SUPPRESSED_TINY_PARSE_FULL_GC_IN_USE_TRIGGER_BYTES,
     GC_SUPPRESSED_TINY_PARSE_IN_USE_TRIGGER_BYTES, GC_TRIGGER_ABSOLUTE_CEILING,
@@ -152,11 +152,24 @@ budget_scaled_accessor!(
     2 * 1024 * 1024
 );
 budget_scaled_accessor!(
-    gc_moving_defer_hard_cap_dyn_bytes,
-    GC_MOVING_DEFER_HARD_CAP_BYTES,
+    /// Growth allowance for a nursery trigger that has been deferred to a
+    /// precise-root safepoint, measured **from the deferral point** (#7024).
+    ///
+    /// ★ Do not give this the `1, 4, 2 MB` shape of
+    /// `gc_trigger_absolute_ceiling_bytes` again, and do not turn it back into
+    /// an absolute arena cap. While it was both, the two collapsed to the same
+    /// number under every explicit `PERRY_GC_HEAP_LIMIT`, which made "a trigger
+    /// is due" (`arena_total >= trigger`) and "the deferral is allowed"
+    /// (`arena_total < cap`) exact complements: the copying minor became
+    /// unreachable at precisely the pressure settings that provoke it. A delta
+    /// is immune to that by construction; the denominator is a third rather
+    /// than a quarter so the deferral point (≈ a quarter of the budget) plus
+    /// its slack still leaves the budget headroom.
+    gc_moving_defer_slack_dyn_bytes,
+    GC_MOVING_DEFER_SLACK_BYTES,
     1,
-    4,
-    2 * 1024 * 1024
+    3,
+    1024 * 1024
 );
 budget_scaled_accessor!(
     gc_tiny_parse_in_use_trigger_dyn_bytes,

crates/perry-runtime/src/gc/policy.rs

@@ -641,17 +641,56 @@ thread_local! {
     /// back-edge poll) so the copying minor can MOVE survivors instead of the
     /// conservative non-moving minor running mid-expression.
     pub(super) static GC_SAFEPOINT_PENDING: Cell<bool> = const { Cell::new(false) };
+    /// `arena_total_bytes()` sampled at the moment `GC_SAFEPOINT_PENDING` was
+    /// last set — the baseline the deferral slack is measured from (#7024).
+    /// Meaningless while `GC_SAFEPOINT_PENDING` is false.
+    pub(super) static GC_SAFEPOINT_DEFER_ARENA_BASE: Cell<usize> = const { Cell::new(0) };
 }
 
-/// Hard cap on committed arena bytes before which a nursery trigger may be
-/// deferred to a safepoint (Phase 2/3). Loop back-edge polls drain the pending
+/// Committed arena bytes a deferred nursery trigger may allocate **past the
+/// point at which it was deferred** before the alloc-point non-moving minor
+/// runs as the safety valve (Phase 2/3). Loop back-edge polls drain the pending
 /// flag every iteration, so the arena never grows near this in normal code; the
-/// cap bounds RSS for code that reaches no safepoint before the next trigger —
+/// slack bounds RSS for code that reaches no safepoint before the next trigger —
 /// a synchronous loop on a specialized lowering path that doesn't yet emit the
-/// poll, or a single mega-expression — where the alloc-point non-moving minor
-/// runs as the safety valve. Kept modest so those cases don't balloon under the
-/// default-on moving GC (raise once poll coverage is complete).
-pub(super) const GC_MOVING_DEFER_HARD_CAP_BYTES: usize = 128 * 1024 * 1024;
+/// poll, or a single mega-expression.
+///
+/// ★ #7024: this is a SLACK (a delta from the deferral point), not an absolute
+/// arena size, and that is the whole point. It was an absolute cap derived by
+/// `budget_scaled(_, 1, 4, 2 MB)` — **the same formula as
+/// `gc_trigger_absolute_ceiling_bytes()`**. `gc_budgeted_due_trigger()` reports
+/// `ArenaBytes` due exactly when `arena_total_bytes() >= trigger`, and the
+/// deferral required `arena_total_bytes() < cap`; under any explicit
+/// `PERRY_GC_HEAP_LIMIT` the two collapsed to the same number, so the two
+/// predicates became exact complements and the deferral was *unreachable* — the
+/// copying minor could never run under the very pressure setting the stress
+/// matrix used to provoke it (`default` arm: 0 copying minors on all 22 corpus
+/// rows). A delta cannot collapse into the trigger, at any heap budget: the
+/// first deferral of a cycle is always taken and the arena is allowed a bounded
+/// amount of growth to reach a poll.
+pub(super) const GC_MOVING_DEFER_SLACK_BYTES: usize = 64 * 1024 * 1024;
+
+/// Whether an alloc-point nursery trigger may (still) be deferred to the next
+/// precise-root safepoint.
+///
+/// `deferred_at` is `Some(arena_total_at_the_first_deferral)` while a deferral
+/// is outstanding, `None` when none is. The first deferral of a cycle is
+/// unconditional — deferring is the *sound* path (it collects with precise,
+/// rewritable roots at a real safepoint) and the alloc-point fallback exists
+/// only to bound growth when nothing drains the deferral. See
+/// `GC_MOVING_DEFER_SLACK_BYTES` for why this is a delta and not an absolute
+/// cap (#7024).
+#[inline]
+pub(super) fn moving_defer_within_slack(
+    arena_total: usize,
+    deferred_at: Option<usize>,
+    slack: usize,
+) -> bool {
+    match deferred_at {
+        None => true,
+        Some(base) => arena_total < base.saturating_add(slack),
+    }
+}
 
 /// RAII guard that marks a #5476 direct old-gen reclaim in progress so a nested
 /// `gc_check_trigger` can't re-enter it. See `GC_OLD_RECLAIM_IN_PROGRESS`.
@@ -1317,14 +1356,39 @@ pub fn gc_check_trigger() {
             // to the next precise-root safepoint (event-loop boundary or a
             // codegen loop back-edge poll) so the copying minor MOVES survivors
             // instead of the conservative non-moving minor running here at a
-            // register-imprecise point. Safety valve: once committed arena bytes
-            // pass the hard cap (a mega-expression that reached no poll), fall
-            // through and collect non-moving here so growth stays bounded.
-            if gc_moving_loop_polls_enabled()
-                && crate::arena::arena_total_bytes() < gc_moving_defer_hard_cap_dyn_bytes()
-            {
-                GC_SAFEPOINT_PENDING.with(|p| p.set(true));
-                return;
+            // register-imprecise point. Safety valve: once the arena has grown
+            // `gc_moving_defer_slack_dyn_bytes()` PAST the point at which the
+            // collection was deferred (a mega-expression that reached no poll),
+            // fall through and collect non-moving here so growth stays bounded.
+            //
+            // #7024: the allowance is measured from the deferral point, not
+            // against an absolute arena size. The absolute cap shared
+            // `budget_scaled(_, 1, 4, 2 MB)` with the trigger ceiling, so under
+            // an explicit PERRY_GC_HEAP_LIMIT "a trigger is due" and "the
+            // deferral is allowed" became exact complements and this branch was
+            // dead — see `GC_MOVING_DEFER_SLACK_BYTES`.
+            if gc_moving_loop_polls_enabled() {
+                let arena_total = crate::arena::arena_total_bytes();
+                let already_deferred = GC_SAFEPOINT_PENDING.with(Cell::get);
+                let deferred_at =
+                    already_deferred.then(|| GC_SAFEPOINT_DEFER_ARENA_BASE.with(Cell::get));
+                if moving_defer_within_slack(
+                    arena_total,
+                    deferred_at,
+                    gc_moving_defer_slack_dyn_bytes(),
+                ) {
+                    if !already_deferred {
+                        GC_SAFEPOINT_DEFER_ARENA_BASE.with(|base| base.set(arena_total));
+                        GC_SAFEPOINT_PENDING.with(|p| p.set(true));
+                    }
+                    return;
+                }
+                // The deferral never drained. The direct minor below IS the
+                // collection that was owed, so retire the request — leaving it
+                // pending would pin `GC_SAFEPOINT_DEFER_ARENA_BASE` at a stale,
+                // already-exceeded baseline and disable deferral for the rest of
+                // the process (the same "the branch is dead" shape as #7024).
+                GC_SAFEPOINT_PENDING.with(|p| p.set(false));
             }
             let pre_in_use = crate::arena::arena_in_use_bytes();
             let pre_malloc_count = malloc_object_count();

crates/perry-runtime/src/gc/tests/triggers.rs

@@ -250,6 +250,129 @@ fn test_budget_scaled_clamps_only_under_budget() {
     assert_eq!(budget_scaled_with(Some(MB), 128 * MB, 1, 4, 2 * MB), 2 * MB);
 }
 
+// ───────────────────────────────────────────────────────────────────────────
+// #7024: the alloc-point deferral must be REACHABLE at the moment a nursery
+// trigger becomes due, at every heap budget.
+//
+// `gc_budgeted_due_trigger()` reports `ArenaBytes` due exactly when
+// `arena_total_bytes() >= effective_next_arena_trigger()`. The deferral that
+// hands the collection to the precise-root safepoint (and therefore to the
+// COPYING minor) used to be guarded by `arena_total_bytes() < <absolute cap>`,
+// where the cap came from `budget_scaled(128 MB, 1, 4, 2 MB)` — byte-for-byte
+// the trigger-ceiling formula. Under any budget small enough for the ceiling to
+// sit at or below the nursery cap the two are the same number, so the two
+// predicates are exact complements: the deferral is refused at precisely the
+// arena size that makes the trigger due, and the copying minor never runs.
+// Measured consequence: the stress matrix's `default` arm ran zero copying
+// minors on all 22 corpus rows.
+//
+// These tests fail against the pre-#7024 predicate (`arena_total < cap`) and
+// pass against the slack-from-the-deferral-point predicate.
+// ───────────────────────────────────────────────────────────────────────────
+#[test]
+fn test_moving_defer_reachable_when_the_arena_trigger_is_due() {
+    use super::super::heap_budget::budget_scaled_with;
+    use super::super::policy::{
+        moving_defer_within_slack, GC_MOVING_DEFER_SLACK_BYTES, GC_TRIGGER_ABSOLUTE_CEILING,
+    };
+    const MB: usize = 1024 * 1024;
+    // `gc_scavenge_nursery_cap_bytes()`'s default; moving mode clamps the
+    // effective trigger to it (`effective_next_arena_trigger`).
+    const NURSERY_CAP: usize = 16 * MB;
+
+    for budget in [
+        None,
+        Some(2 * MB),
+        Some(8 * MB), // the stress matrix's `--pressure 8`
+        Some(16 * MB),
+        Some(32 * MB),
+        Some(64 * MB),
+        Some(128 * MB),
+        Some(512 * MB),
+    ] {
+        let ceiling = budget_scaled_with(budget, GC_TRIGGER_ABSOLUTE_CEILING, 1, 4, 2 * MB);
+        let slack = budget_scaled_with(budget, GC_MOVING_DEFER_SLACK_BYTES, 1, 3, MB);
+        // The smallest `arena_total` at which `gc_budgeted_due_trigger()`
+        // reports ArenaBytes, in moving mode.
+        let due_at = ceiling.min(NURSERY_CAP);
+
+        // The collapse premise, asserted rather than assumed: whenever the
+        // budget pulls the ceiling to or below the nursery cap — every device
+        // budget ≤ 64 MB, and every `--pressure` setting the matrix uses — the
+        // pre-#7024 absolute cap is already reached at `due_at`, so the old
+        // guard `arena_total < cap` was FALSE exactly when the trigger fired.
+        let legacy_cap = budget_scaled_with(budget, 128 * MB, 1, 4, 2 * MB);
+        if ceiling <= NURSERY_CAP {
+            assert!(
+                due_at >= legacy_cap,
+                "budget {budget:?}: expected the pre-#7024 absolute cap ({legacy_cap}) to be \
+                 unreachable at the due point ({due_at})"
+            );
+        }
+
+        // The fix: the first deferral of a cycle is unconditional, so the
+        // copying minor is reachable at every budget.
+        assert!(
+            moving_defer_within_slack(due_at, None, slack),
+            "budget {budget:?}: a nursery trigger due at {due_at} bytes must be deferrable"
+        );
+        // …and it stays deferrable for a whole slack of further growth, so a
+        // loop back-edge poll has room to drain it.
+        assert!(
+            moving_defer_within_slack(due_at + slack - 1, Some(due_at), slack),
+            "budget {budget:?}: deferral must survive until the slack is spent"
+        );
+    }
+}
+
+#[test]
+fn test_moving_defer_slack_still_has_a_safety_valve() {
+    use super::super::policy::moving_defer_within_slack;
+    const MB: usize = 1024 * 1024;
+    let slack = 4 * MB;
+    let base = 2 * MB;
+
+    // No deferral outstanding: always allowed, however large the arena. This is
+    // the sound path (precise, rewritable roots at a real safepoint); the
+    // alloc-point fallback exists only to bound growth when nothing drains it.
+    assert!(moving_defer_within_slack(0, None, slack));
+    assert!(moving_defer_within_slack(4 * 1024 * MB, None, slack));
+
+    // Deferral outstanding: bounded overshoot, measured from the deferral point.
+    assert!(moving_defer_within_slack(base, Some(base), slack));
+    assert!(moving_defer_within_slack(
+        base + slack - 1,
+        Some(base),
+        slack
+    ));
+    assert!(!moving_defer_within_slack(base + slack, Some(base), slack));
+    assert!(!moving_defer_within_slack(
+        base + slack + MB,
+        Some(base),
+        slack
+    ));
+
+    // The valve is relative, not absolute: a program whose live set already
+    // sits far above any fixed cap still gets its slack (and therefore still
+    // gets copying minors) instead of being pinned on the non-moving path.
+    let big = 900 * MB;
+    assert!(moving_defer_within_slack(big, None, slack));
+    assert!(moving_defer_within_slack(big + slack - 1, Some(big), slack));
+    assert!(!moving_defer_within_slack(big + slack, Some(big), slack));
+
+    // Overflow-safe.
+    assert!(moving_defer_within_slack(
+        usize::MAX - 1,
+        Some(usize::MAX),
+        slack
+    ));
+    assert!(!moving_defer_within_slack(
+        usize::MAX,
+        Some(usize::MAX),
+        slack
+    ));
+}
+
 // The un-armed trigger cell (desktop-default const initializer) reads as
 // the device ceiling; an armed trigger above the ceiling is legitimate
 // (headroom floor over a big live set) and must NOT be clamped.