You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The same .ts file, byte-identical, compiled by the same perry binary, linking the same runtime archive, produces two different object files — and one of them runs 09_method_calls44× slower than the other. The discriminator is where the source file lives on disk.
source at <perry-checkout>/**/09_method_calls.ts → object 6efab0f508029e70.o → 78–110 ms
source anywhere else → object b0047dfba1cb3e98.o → 3417–3503 ms
benchmarks/results/public-node-bun-v1.json reports 79 ms for 09_method_calls, because benchmarks/compare.sh does cd benchmarks/suite and compiles there. A user compiling the identical file in their own project gets 3450 ms. The published baseline is measuring an arm that only exists inside the Perry checkout.
Reproduction (deterministic, 12/12 and 6/6)
P=target/release/perry
SRC=benchmarks/suite/09_method_calls.ts
# arm A — inside the checkout, any subdirectory, any filename
mkdir -p ./zzp && cp $SRC ./zzp/ && (cd ./zzp && PERRY_CACHE_DIR=/tmp/c1 ../../$P 09_method_calls.ts -o /tmp/A)
/tmp/A # method_calls:104# arm B — outside the checkout, same file
mkdir -p /tmp/zzp && cp $SRC /tmp/zzp/ && (cd /tmp/zzp && PERRY_CACHE_DIR=/tmp/c2 $PWD/$P 09_method_calls.ts -o /tmp/B)
/tmp/B # method_calls:3435
Controls, all with a freshPERRY_CACHE_DIR each time (so no stale cached object is involved):
condition
result
<checkout>/benchmarks/suite/
108 ms
<checkout>/zzp11/deep/er/ (arbitrary subdir)
104 ms
<checkout>/zzp3/renamed_bench.ts (renamed)
110 ms
/private/tmp/zzprobe/
3426 ms
/private/tmp/a/b/c/d/e/
3436 ms
/private/tmp/perryfake/benchmarks/suite/ (same dir names)
3501 ms
/private/tmp/perryfake2/perry/benchmarks/suite/
3503 ms
<scratchpad>/mc_clean/
3435 ms
/Users/amlug/projects/perry/ (checkout's parent)
3433 ms
Ruled out, each by direct test:
not the cwd — arm B stays slow when run from inside the checkout with an absolute path to an outside source, and arm A stays fast when run from /private/tmp with an absolute path to an in-checkout source
not the filename — a renamed copy inside the checkout is fast
not the object cache — every arm above used a freshly created PERRY_CACHE_DIR; the stored object hashes differ (6efab0f508029e70.o vs b0047dfba1cb3e98.o), so codegen genuinely differs
not the runtime archive — both link target/perry-auto-2f56c86280e7e60e/release/libperry_runtime.a (identical ld warning line in -vv output)
not auto-optimize — PERRY_NO_AUTO_OPTIMIZE=1 gives 3450 ms in arm B, unchanged
not PERRY_WORKSPACE_ROOT — pointing it at the checkout from arm B does not help (and the workspace root is resolved from the exe path anyway, so it is already the checkout in both arms)
not PERRY_ALLOW_PERRY_FEATURES — no effect
not the installed package set — symlinking the checkout's node_modules into /private/tmp makes the wildcard expand to the same 18 packages, and the build is still slow (3451 ms)
not a package.json / tsconfig.json — the checkout has no perry key and no tsconfig; adding a plain package.json outside changes nothing
One further datum that is probably the thread to pull: inside the checkout, adding a package.json with "perry": {"compilePackages": []}or"perry": {"compilePackages": ["*"]} flips the build to the slow arm (3435 / 3417 ms), while no package.json at all (falling through to the checkout root's, which has no perry key, i.e. the implicit auto-default) stays fast (106 ms). So the switch is somewhere in compile/host_config.rs host-config resolution — compile_packages_explicit and friends — feeding something that reaches codegen. But it is not the expanded package set, per the symlink control above.
Full -vv diff between the two arms
The entire observable difference in the compile log is:
< Compile package wildcard: expanded to 18 installed package(s)
> Compile package wildcard: expanded to 2 installed package(s)
< perry-codegen: emitted 188906 bytes of LLVM IR for '09_method_calls.ts'
> perry-codegen: emitted 197288 bytes of LLVM IR for '09_method_calls.ts'
Same module count (Found 1 module(s): 1 native, 0 JavaScript), same clang line, same target, same archives.
What the two objects actually differ by
--trace llvm from both locations, 1147 lines of diff. The fast arm lowers this.value to the typed class-field path:
The slow arm never emits class_field_inline.* at all and instead routes every access through generic polymorphic inline caches (@perry_ic_1/2/5/7, put.dynic.*, __ic_decl_5), i.e. the js_typed_feedback_class_field_{get,set}_guard → TLS-hashmap path described in #5094. #5094 quotes method_calls at 3300 ms; that is exactly the arm reproduced here, still live in 0.5.1279.
Blast radius
Only class-field access in a hot loop diverges. Checked side by side:
benchmark
in-checkout
outside
09_method_calls
78
3435
16_matrix_multiply
631
642
11_prime_sieve
107
108
15_mandelbrot
22
22
bench_object_property
149
141
07_object_create
3
3
12_binary_trees
4
4
bench_gc_pressure
37
30
So the other published numbers are trustworthy; 09_method_calls is not.
Why this is more than a benchmarking embarrassment
A user-visible 44× cliff with no diagnosable cause. Nothing in the compile output tells the user which arm they got. Moving a file, or adding a perry key to package.json, silently changes generated code by 44×.
It makes every measurement in the repo conditional. Any A/B run from a scratch directory — the normal way ad-hoc .ts probes are measured — silently lands on a different codegen path than CI and than the published baseline. That is the "gate runs but its subject never did" failure mode from CLAUDE.md, applied to benchmarking.
Bisect compile/host_config.rs host-config resolution against the codegen predicate that selects class_field_inline — the compilePackages-shaped controls above narrow it a long way.
Make the published-baseline harness assert its arm (e.g. nm/IR check that class_field_inline is present), so a regression to the slow path shows up as a failure rather than as a number.
Measured on Apple M1 Max, macOS 26.5, perry 0.5.1279 @ defa4d601, release, auto-optimize on. Numbers are min of 5 self-reported ms; the spread within each arm is under 3%, and the in-checkout arm reproduces public-node-bun-v1.json (78 vs 79).
Summary
The same
.tsfile, byte-identical, compiled by the sameperrybinary, linking the same runtime archive, produces two different object files — and one of them runs09_method_calls44× slower than the other. The discriminator is where the source file lives on disk.benchmarks/results/public-node-bun-v1.jsonreports 79 ms for09_method_calls, becausebenchmarks/compare.shdoescd benchmarks/suiteand compiles there. A user compiling the identical file in their own project gets 3450 ms. The published baseline is measuring an arm that only exists inside the Perry checkout.Reproduction (deterministic, 12/12 and 6/6)
Controls, all with a fresh
PERRY_CACHE_DIReach time (so no stale cached object is involved):<checkout>/benchmarks/suite/<checkout>/zzp11/deep/er/(arbitrary subdir)<checkout>/zzp3/renamed_bench.ts(renamed)/private/tmp/zzprobe//private/tmp/a/b/c/d/e//private/tmp/perryfake/benchmarks/suite/(same dir names)/private/tmp/perryfake2/perry/benchmarks/suite/<scratchpad>/mc_clean//Users/amlug/projects/perry/(checkout's parent)Ruled out, each by direct test:
/private/tmpwith an absolute path to an in-checkout sourcePERRY_CACHE_DIR; the stored object hashes differ (6efab0f508029e70.ovsb0047dfba1cb3e98.o), so codegen genuinely differstarget/perry-auto-2f56c86280e7e60e/release/libperry_runtime.a(identicalldwarning line in-vvoutput)PERRY_NO_AUTO_OPTIMIZE=1gives 3450 ms in arm B, unchangedPERRY_WORKSPACE_ROOT— pointing it at the checkout from arm B does not help (and the workspace root is resolved from the exe path anyway, so it is already the checkout in both arms)PERRY_ALLOW_PERRY_FEATURES— no effectnode_modulesinto/private/tmpmakes the wildcard expand to the same 18 packages, and the build is still slow (3451 ms)package.json/tsconfig.json— the checkout has noperrykey and no tsconfig; adding a plainpackage.jsonoutside changes nothingOne further datum that is probably the thread to pull: inside the checkout, adding a
package.jsonwith"perry": {"compilePackages": []}or"perry": {"compilePackages": ["*"]}flips the build to the slow arm (3435 / 3417 ms), while nopackage.jsonat all (falling through to the checkout root's, which has noperrykey, i.e. the implicit auto-default) stays fast (106 ms). So the switch is somewhere incompile/host_config.rshost-config resolution —compile_packages_explicitand friends — feeding something that reaches codegen. But it is not the expanded package set, per the symlink control above.Full
-vvdiff between the two armsThe entire observable difference in the compile log is:
Same module count (
Found 1 module(s): 1 native, 0 JavaScript), same clang line, same target, same archives.What the two objects actually differ by
--trace llvmfrom both locations, 1147 lines of diff. The fast arm lowersthis.valueto the typed class-field path:The slow arm never emits
class_field_inline.*at all and instead routes every access through generic polymorphic inline caches (@perry_ic_1/2/5/7,put.dynic.*,__ic_decl_5), i.e. thejs_typed_feedback_class_field_{get,set}_guard→ TLS-hashmap path described in #5094. #5094 quotesmethod_callsat 3300 ms; that is exactly the arm reproduced here, still live in 0.5.1279.Blast radius
Only class-field access in a hot loop diverges. Checked side by side:
09_method_calls16_matrix_multiply11_prime_sieve15_mandelbrotbench_object_property07_object_create12_binary_treesbench_gc_pressureSo the other published numbers are trustworthy;
09_method_callsis not.Why this is more than a benchmarking embarrassment
perrykey topackage.json, silently changes generated code by 44×..tsprobes are measured — silently lands on a different codegen path than CI and than the published baseline. That is the "gate runs but its subject never did" failure mode from CLAUDE.md, applied to benchmarking.Suggested next steps
--opt-report, cf. --opt-report: candidate-generation pre-filters for Ptr<Shape> / Ptr<NumArray> record nothing, so "0 candidates" cannot be read #7112 / feat(cli): --opt-report — surface which values could not be statically typed, why, and whether the developer can fix it #6952), so the two arms are distinguishable from the compile output.compile/host_config.rshost-config resolution against the codegen predicate that selectsclass_field_inline— thecompilePackages-shaped controls above narrow it a long way.nm/IR check thatclass_field_inlineis present), so a regression to the slow path shows up as a failure rather than as a number.Measured on Apple M1 Max, macOS 26.5,
perry 0.5.1279@defa4d601, release, auto-optimize on. Numbers are min of 5 self-reported ms; the spread within each arm is under 3%, and the in-checkout arm reproducespublic-node-bun-v1.json(78 vs 79).