perf(gc): emit the shadow-slot root store inline

[proggeramlug]

A ceiling measurement (#7079) put the shadow stack at +63.7 % instructions
on application-shaped code (w6_records, Pi 5) versus not emitting it at all,
and concluded — labelled as a reading, not a measurement — that what remained
was per-store extern "C" calls that only a codegen change can touch. There
are two codegen changes that touch it: replacing the shadow stack with stack
maps, and inlining the store. This PR is the second one.

It removes 56 % of that ceiling. w6_records goes from +60.8 % to
+26.7 % against an identical toolchain with only the store form changed.

Why a call, not the work, was the cost

An extern "C" call costs twice: the call itself, and the fact that it is
opaque to LLVM, which forces a spill of every live value around it and
blocks hoisting across it. Read out of the shipped aarch64 archive,
js_shadow_slot_bind's fast path was ~35 instructions, of which about six did
any work:

	before	after
call/return	bl + 4-instruction prologue/epilogue pair	—
reach the thread-local	TLSDESC adrp/ldr/add + indirect blr into the resolver, mrs tpidr_el0	— (state pointer already in a register)
lazy TLS destructor check	ldrb/cmp/b.eq + a panic_access_error edge	— (the TLS type is now drop-free)
the work	2 guards, address computation, stp, gated barrier	unchanged

How the thread-local block is reached

Not by re-deriving the TLS address in generated code. That would mean
modelling Rust's TLS model per platform (TLSDESC on this Linux build, tlv on
macOS) and would be a second, unverified path to the same memory.

Instead the address is obtained from the runtime and cached for the
activation. js_shadow_frame_enter is js_shadow_frame_push returning the
address of this thread's ShadowStackState instead of the frame handle, so
codegen pays exactly the one thread-local lookup per activation that the push
already paid. The handle the matching js_shadow_frame_pop needs is recovered
as frame_top - SHADOW_STACK_HEADER_SLOTS, so the pop side is untouched.

Caching it is sound because it is the address of a const-initialized,
drop-free thread_local!: fixed for the thread's lifetime, never reallocated.
The buffer it points at does move when a deeper frame grows it — which is
exactly why no frame base is cached and ptr / len / frame_top are
re-loaded from the state at every store. One activation of a compiled function
runs entirely on one thread (perry/thread hands a whole call to a worker; a
resumed async state machine re-enters through the function entry and calls
js_shadow_frame_enter again), so a cached pointer never escapes its thread.

Because the pointer comes from the runtime's own SHADOW.with, the inline
store addresses the same memory js_shadow_slot_set writes by construction.
js_shadow_state_addr makes that testable from Rust: a test writes an entry
through the published offsets and reads it back with js_shadow_slot_get, and
vice versa.

ShadowStackState is now #[repr(C)] with an explicit ptr/len/cap
buffer, because Vec's layout is not a contract — in the archive read at the
time of writing it placed cap at 0, ptr at 8 and len at 16, and a silent
reorder would have codegen writing live GC roots through the wrong word.
Dropping the Vec also drops the type's drop glue, which is what forced the
per-op lazy destructor-registration check; the buffer is freed at thread exit by
a separate guard thread-local armed only from the cold growth path.

Soundness, per root property

Liveness — the inline store writes the same ShadowEntry.value and sets the
same SLOT_ACTIVE bit of meta, at the same index, so
visit_shadow_stack_root_slots marks it identically.

Rewritability — meta still carries the bound compiled-local address, with
bound_slot_meta's alignment fallback intact (a tag-colliding address is
recorded active-but-unbound, never truncated), so an evacuating collection
rewrites the alloca the mutator reads after the safepoint, not just the mirror.

The value the mutator stored — the value is read from the local slot at the
store site, in the position the call occupied, and written immediately. Nothing
re-reads a slot at a later safepoint.

The incremental-mark root shading barrier is emitted inline behind the same
PERRY_INCREMENTAL_MARK_BARRIER_ACTIVE_COUNT gate the runtime and
emit_persistent_shadow_root_barrier already use.

Guards. Both of the runtime function's guards are emitted: the
frame_top == usize::MAX sentinel and the slot < len bounds check. Honest
scope: the sentinel is unreachable in a balanced program (that state implies
len == 0, so the bounds check alone would skip). It is kept because if the two
ever diverge the failure mode is silent corruption, not a skip —
usize::MAX + idx wraps to idx - 1, an in-bounds index into the frame
header, unlinking every outer frame from the root scan. Same wrap-around class
#7079 fixed in frame_pop.

Bounds and capacity are not dropped. Growth stays in the runtime's cold
path; the inline sequence never grows the buffer, it re-reads len and skips
when out of range, exactly as js_shadow_slot_set does.

Each store also emits a null-state fallback arm calling the original runtime
function. js_shadow_frame_enter is declared nonnull, so LLVM folds that arm
away wherever the push dominates — verified in the linked binary, where
js_shadow_slot_set no longer appears in the symbol table at all.

Measured

Pi 5 (Cortex-A76, governor performance), instructions retired under perf stat, taskset -c 3, 12 interleaved reps per arm, runs dropped if the arm
clock moved >1 % or any throttle bit set. 0 of 300 runs dropped, 0 oracle
mismatches, clock 2 400 017–2 400 037 kHz (within 0.001 %), throttle word
0x0 throughout, temp ≤ 67.5 °C, load average 0.93/2.54/3.34 at launch.
Every arm's stdout diffed against pinned Node 26.5.0. Instruction-count cv is
0.00 % on every workload arm, which is also why the decaying 5/15-minute
load does not threaten these figures — instruction counts are load-insensitive,
unlike the cycles and wall columns.

Single-variable isolation — same compiler, same runtime archives, only
PERRY_INLINE_SHADOW_SLOT differs (it is in both cache keys, so the arms cannot
share a cached .o; binary distinctness asserted per workload):

workload	ceiling, store as a call	ceiling, store inline	ceiling removed	instructions saved
w6_records (application-shaped)	+60.78 %	+26.65 %	56 %	−21.22 %
w1_calldepth	+23.91 %	+12.13 %	49 %	−9.50 %
w3_treewalk	+35.90 %	+28.08 %	22 %	−5.75 %
w5_purity (floor)	−0.00 %	−0.00 %	—	0.00 %

w5_purity is exactly inert, as it must be: a pure numeric loop reserves no
shadow slots, so there is nothing to inline.

The ceiling, measured directly rather than inferred. PERRY_SHADOW_STACK=1
vs =0 re-run with the change in. The #7079 reference arms were re-run
interleaved in the same session and reproduce their published numbers, which
is what makes the two columns comparable:

workload	#7079 ceiling (re-measured here)	#7079 published	new ceiling
w6_records	+63.69 %	63.7 %	+26.65 %
w3_treewalk	+41.34 %	41.3 %	+28.08 %
w1_calldepth	+28.02 %	28.0 %	+12.13 %

Not inlined, deliberately

The frame push/pop pair (per activation, not per store), and the parameter /
closure-prologue / persistent entry-setup binds. Those are emitted while block 0
is being built directly, before the lowering context that can create the guard
and barrier basic blocks exists.

What that leaves is checkable rather than asserted. In w6_records' emitted IR:

symbol	callsites	real	folded ss.slow arms
js_shadow_slot_set	16	0	16
js_shadow_slot_bind	22	11	11
js_shadow_frame_enter	6	6	—
js_shadow_frame_pop	7	7	—

Every opaque shadow call left in w6 is per-activation; none is per-store.
js_shadow_slot_set is gone from the linked binary's symbol table entirely, and
the 11 surviving js_shadow_slot_bind sites are all parameter and prologue
binds. Since alwaysinline leaf functions turn per-activation into
per-iteration, those are the natural next target — and they are why
w3_treewalk (deep recursion, few stores per activation) improves least here.

Gates

scripts/gc_repsel_matrix.sh --arms all --pressure 8, pinned Node 26.5.0:
PASS=339 UNVER=100 XFAIL=1 FAIL=0 over 440 cells — the required numbers
exactly, byte-exact against the oracle on 439/440 (the one non-match is the
registered XFAIL, repsel_ptr_shape_locals under rep_ptr_shape_off). The
arms were live, not inert: every requires=move arm ran 21/22 rows with
moved-objects and copy-minor both confirmed, and the requires=scavenge
arms 12/22 — so the evacuating configurations actually evacuated with the
inline stores in place.

gc-ratchet: all 72 gated retention/evacuation counters bit-identical to
pristine origin/main. Rather than trust the pinned baseline artifact —
which was captured on darwin-arm64 and which the harness itself refuses to
gate against a linux-aarch64 run — origin/main @ 404c1c948 was built in
the same worktree, with the same package set, and measured on the same host.
heap_used_bytes, heap_total_bytes, minor_cycles, step_cycles,
copied_objects, copied_bytes, promoted_objects, promoted_bytes and
freed_bytes are equal across all 8 probes: 0 of 72 differ. Nothing is kept
alive or dropped that was not before.

The comparison is on the whole metric object — median, min, max, spread, stdev
and every sample — not just medians.

And the A/B was not vacuous: on the same two runs the ungated metrics
differ on 20 of 24 rows, wall_ms and rss_bytes on all 8 probes (branch
faster on 6/8, median ≈ −1.7 % wall). Identical counters, demonstrably
different binaries. The four identical peak_rss_bytes values are
page-quantised.

(Against the pinned artifact, 71 of 72 gated rows match to the byte and one —
05_closure_capture heap_used_bytes, +6.47 % — does not. That is the
platform, not this PR: the pre-#7088 pinned toolchain reproduces the branch's
exact figure, 1,107,552, on this host, and the branch reproduces main's
exactly. The harness flags the platform mismatch itself and refuses to gate
across it.)

Tests — 6 new runtime, 5 new codegen, 3 new cross-crate contract. Every one
was verified to fail under a targeted sabotage of the thing it covers:

sabotage	test that caught it
codegen offset drifts from the runtime's	shadow_layout_contract_matches_the_runtime
inline meta write drops SLOT_ACTIVE	inline_bound_slot_survives_and_is_rewritten_by_a_copying_minor, inline_write_and_runtime_accessor_address_the_same_entry
sentinel guard removed	inline_write_with_no_frame_installed_is_skipped_not_wrapped
inline emitter disabled	4 of the 5 codegen shape tests
root shading barrier dropped	inline_bind_keeps_the_gated_root_shading_barrier
clear zeroes meta instead of masking	dead_local_clear_is_inline_and_preserves_the_binding

The sentinel-guard test needed a second pass: the first version passed with the
guard deleted, because after a balanced pop len is 0 and the bounds check
alone skips the write. It now forces frame_top = usize::MAX while the frame's
entries are still live, which is the state the guard actually exists for, and
fails when the guard is removed.

PERRY_INLINE_SHADOW_SLOT=0/off/false reverts to the calls for bisection.

[coderabbitai]

📝 Walkthrough

Walkthrough

The PR replaces vector-backed shadow-stack state with a fixed raw-buffer layout, adds js_shadow_frame_enter, emits guarded inline shadow-slot binds and clears, preserves runtime fallbacks, updates cache keys, and adds codegen/runtime layout and behavior tests.

Changes

Inline shadow-slot storage and code generation

Layer / File(s)	Summary
Raw shadow-stack state and runtime operations crates/perry-runtime/src/gc/roots/..., crates/perry-runtime/src/gc/tests/support.rs	ShadowStackState now stores raw buffer pointers, lengths, capacity, and frame position; frame, slot, root-scanning, savepoint, restore, and reset paths use the new representation.
State-pointer frame entry and layout contract crates/perry-codegen/src/function.rs, crates/perry-codegen/src/runtime_decls/arrays.rs, crates/perry-codegen/src/module.rs, crates/perry-codegen/src/lib.rs, crates/perry-codegen/src/expr/mod.rs	Codegen declares and emits js_shadow_frame_enter, caches its state pointer, derives frame handles from layout offsets, and exposes layout constants for cross-crate tests.
Inline bind, clear, and barrier emission crates/perry-codegen/src/expr/shadow_inline.rs, crates/perry-codegen/src/expr/shadow_slot.rs, crates/perry-codegen/src/codegen/helpers.rs	Guarded inline writes update shadow entries directly, preserve metadata and liveness behavior, emit the incremental-mark barrier when enabled, and fall back to runtime helpers when needed.
Emission validation, runtime contracts, and cache mode selection crates/perry-codegen/tests/shadow_slot_hygiene.rs, crates/perry-runtime/src/gc/tests/..., crates/perry/src/shadow_layout_contract.rs, crates/perry/src/commands/compile/..., changelog.d/7088-inline-shadow-slot-store.md	Tests cover IR shape, address symmetry, clearing, GC rewriting, sentinel and bounds guards, frame recovery, layout consistency, and cache separation for the inline toggle.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant GeneratedCode
  participant js_shadow_frame_enter
  participant ShadowStackState
  participant ShadowEntry
  participant GC
  GeneratedCode->>js_shadow_frame_enter: enter frame
  js_shadow_frame_enter-->>GeneratedCode: return state pointer
  GeneratedCode->>ShadowStackState: check frame and slot bounds
  GeneratedCode->>ShadowEntry: write inline bind or clear
  GC->>ShadowStackState: scan and rewrite active roots

Loading

Possibly related PRs

• PerryTS/perry#7079: Introduces overlapping shadow-stack runtime internals and incremental-mark barrier behavior used by this PR.
• PerryTS/perry#6811: Adds the incremental-mark barrier counter consulted by the inline bind path.
• PerryTS/perry#6846: Also changes shadow-slot clear emission in shadow_slot.rs.

Suggested labels: performance

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description is detailed, but it does not follow the required template and omits explicit Summary, Changes, Related issue, Test plan, and Checklist sections.	Rewrite the PR description using the repository template and add clear Summary, Changes, Related issue, Test plan, Screenshots/output, and Checklist sections.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and accurately summarizes the main change: inlining the shadow-slot root store for GC performance.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch perf/inline-shadow-slot-store

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (4)

crates/perry-codegen/src/function.rs (1)

319-323: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

ShadowFramePush::handle_reg now holds the state register, not a handle.

shadow_frame_push_line renders %reg = call ptr @js_shadow_frame_enter(...), so this field is the state pointer register; the name will mislead the next reader of reserve_shadow_slot. Consider renaming to state_reg (field doc at Line 190 too).

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/perry-codegen/src/function.rs` around lines 319 - 323, Rename the
ShadowFramePush field handle_reg to state_reg and update all references,
including reserve_shadow_slot and shadow_frame_push_line, so the field
accurately describes the state pointer register. Update the field documentation
near the ShadowFramePush definition to use the new name and terminology.

crates/perry-runtime/src/gc/roots/shadow_stack.rs (1)

283-288: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Stale SAFETY comment contradicts the code.

The comment says the length is passed as 0, then immediately says the real length is re-declared; the code passes s.len. Trim it to the accurate rationale.

📝 Suggested comment fix

-        // SAFETY: `ptr`/`cap` came from a `Vec<ShadowEntry>` built here. Length
-        // is passed as 0 because `ShadowEntry: Copy` has no drop glue, and the
-        // live prefix is copied by `reserve` from the raw allocation anyway —
-        // so re-declare the real length to keep the data.
+        // SAFETY: `ptr`/`cap` came from a `Vec<ShadowEntry>` built here. The
+        // real `len` is re-declared so `reserve` copies the live prefix into
+        // the new allocation.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/perry-runtime/src/gc/roots/shadow_stack.rs` around lines 283 - 288,
Update the SAFETY comment immediately above Vec::from_raw_parts in the
shadow-stack reconstruction to accurately describe that s.len is passed as the
vector length; remove the contradictory claim that length is passed as 0 while
retaining only the valid rationale about the Vec allocation, Copy element type,
and preserved live prefix.

crates/perry-codegen/src/lib.rs (1)

11-11: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Revert expr to crate-local visibility.

perry only needs perry_codegen::expr_shadow_layout; pub use crate::expr::shadow_inline::{…} works while expr remains pub(crate), so widening it makes the entire expression lowering namespace part of perry-codegen’s public API unnecessarily.

♻️ Proposed change

-pub mod expr;
+pub(crate) mod expr;

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/perry-codegen/src/lib.rs` at line 11, Change the expr module
declaration back to crate-local visibility by replacing the public export in the
module declarations. Preserve the existing public re-export of
expr_shadow_layout through the established shadow_inline path so perry can
access only that API without exposing the full expression-lowering namespace.

crates/perry-codegen/src/expr/shadow_inline.rs (1)

428-547: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Brittle hardcoded SSA register numbers in three unit tests.

frame_push_uses_frame_enter_and_derives_the_handle (Line 440, 447), inline_store_keeps_the_sentinel_and_bounds_guards (Lines 507, 512, 518), and inline_bind_keeps_the_gated_root_shading_barrier (Line 543) assert against exact register names (%r3, %r5, %r10, %r13, %r15, %r17, %r23). Any unrelated instruction added/removed earlier in emit_inline_slot_write (or in shared codegen helpers) will shift numbering and break these tests without the guarded property actually regressing. The file already demonstrates a more robust technique for other assertions — content-based block extraction (store_blocks/bind_block/clear_block/entry_reg) and a wildcard-register fallback for the frame_top offset check at Line 442 — that could be applied consistently here too (e.g., match "icmp eq i64 %", ", -1" and "sub i64 %", &SHADOW_STACK_HEADER_SLOTS.to_string() without pinning the exact register).

♻️ Example direction for one of the assertions

-        assert!(
-            body.contains(&format!("sub i64 %r5, {}", SHADOW_STACK_HEADER_SLOTS)),
-            "pop handle must be frame_top - {SHADOW_STACK_HEADER_SLOTS}; body:\n{body}"
-        );
+        assert!(
+            body.contains("sub i64 %") && body.contains(&format!(", {}\n", SHADOW_STACK_HEADER_SLOTS)),
+            "pop handle must be frame_top - {SHADOW_STACK_HEADER_SLOTS}; body:\n{body}"
+        );

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/perry-codegen/src/expr/shadow_inline.rs` around lines 428 - 547,
Replace hardcoded SSA register names in
frame_push_uses_frame_enter_and_derives_the_handle,
inline_store_keeps_the_sentinel_and_bounds_guards, and
inline_bind_keeps_the_gated_root_shading_barrier with content-based or
wildcard-register assertions. Preserve each test’s validation of the relevant
operation and constants, using existing helpers such as bind_block and entry_reg
where appropriate, so unrelated codegen changes do not break the tests.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/perry-codegen/src/expr/shadow_inline.rs`:
- Around line 428-547: Replace hardcoded SSA register names in
frame_push_uses_frame_enter_and_derives_the_handle,
inline_store_keeps_the_sentinel_and_bounds_guards, and
inline_bind_keeps_the_gated_root_shading_barrier with content-based or
wildcard-register assertions. Preserve each test’s validation of the relevant
operation and constants, using existing helpers such as bind_block and entry_reg
where appropriate, so unrelated codegen changes do not break the tests.

In `@crates/perry-codegen/src/function.rs`:
- Around line 319-323: Rename the ShadowFramePush field handle_reg to state_reg
and update all references, including reserve_shadow_slot and
shadow_frame_push_line, so the field accurately describes the state pointer
register. Update the field documentation near the ShadowFramePush definition to
use the new name and terminology.

In `@crates/perry-codegen/src/lib.rs`:
- Line 11: Change the expr module declaration back to crate-local visibility by
replacing the public export in the module declarations. Preserve the existing
public re-export of expr_shadow_layout through the established shadow_inline
path so perry can access only that API without exposing the full
expression-lowering namespace.

In `@crates/perry-runtime/src/gc/roots/shadow_stack.rs`:
- Around line 283-288: Update the SAFETY comment immediately above
Vec::from_raw_parts in the shadow-stack reconstruction to accurately describe
that s.len is passed as the vector length; remove the contradictory claim that
length is passed as 0 while retaining only the valid rationale about the Vec
allocation, Copy element type, and preserved live prefix.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: b9028ae2-20d2-4f88-8724-e73741398b84

📥 Commits

Reviewing files that changed from the base of the PR and between c86835f and 33152a4.

📒 Files selected for processing (20)

• changelog.d/7086-inline-shadow-slot-store.md
• crates/perry-codegen/src/codegen/closure.rs
• crates/perry-codegen/src/codegen/helpers.rs
• crates/perry-codegen/src/expr/mod.rs
• crates/perry-codegen/src/expr/shadow_inline.rs
• crates/perry-codegen/src/expr/shadow_slot.rs
• crates/perry-codegen/src/function.rs
• crates/perry-codegen/src/lib.rs
• crates/perry-codegen/src/module.rs
• crates/perry-codegen/src/runtime_decls/arrays.rs
• crates/perry-codegen/tests/shadow_slot_hygiene.rs
• crates/perry-runtime/src/gc/roots.rs
• crates/perry-runtime/src/gc/roots/shadow_stack.rs
• crates/perry-runtime/src/gc/tests/debt_pacer.rs
• crates/perry-runtime/src/gc/tests/shadow_stack_ops.rs
• crates/perry-runtime/src/gc/tests/support.rs
• crates/perry/src/commands/compile/build_cache.rs
• crates/perry/src/commands/compile/object_cache.rs
• crates/perry/src/main.rs
• crates/perry/src/shadow_layout_contract.rs