compat: async lowering blocks self-fetch (express/fastify/nestjs/hono)

[proggeramlug]

Summary

Perry's TypeScript→native compiler lowers user async () => ... arrows into a busy-wait loop rather than a real Future state machine. This makes any in-process await fetch(...) against a server running in the same process deadlock — the busy-wait holds the V8 thread, the server's accept future on deno's executor never settles its continuation, and the program times out (rc=124).

This blocks four compat sweep fixtures simultaneously, all with the same shape:

Package	Sweep status	Symptom
express	rc=124 timeout	server listens, then await fetch(self) hangs
fastify	rc=124 timeout	"Server listening on http://0.0.0.0:18932" then hang
nestjs	empty stdout	similar — listen→fetch loop never completes
hono	rc=139 SIGSEGV under redirected stdout	async app.fetch(req) chain doesn't pump cleanly

Diagnosis (from PR #997 agent)

"Perry's TS→native compilation lowers user async () => ... arrows into a busy-wait loop (crates/perry-codegen/src/expr.rs:10687), so a native callback invoked via native_callback_trampoline blocks the V8 thread for the full callback duration. Inside the busy-wait, js_run_jsruntime_pump re-enters poll_event_loop — but op_perry_http_accept's rx.recv().await future on deno's executor never settles its V8 continuation while the trampoline frame holds an active scope. Fixing it cleanly needs real async lowering across the codegen, not a V8-side patch."

Workarounds already shipped (do NOT regress)

• External clients work: Perry-compiled servers respond correctly to external curl/fetch. PR fix(jsruntime/http): V8 listen keepalive + express handler smoke #997 added JsRuntimeState::last_poll_was_pending to keep the codegen outer event loop alive while op_perry_http_listen's bind future is in flight, and made op_perry_fetch async via spawn_blocking ureq. Server lifecycle (listen → handle → close) is functional.
• fastify via perry-stdlib works end-to-end because it doesn't route through the V8-fallback hyper bridge.
• PR fix(runtime): native await of V8 Promise handle in jose JWT roundtrip #1010 fixed js_async_step_chain / js_async_step_done to await V8 Promise handles instead of passing them through as primitives, which lifted jose and any package doing await <V8-imported-function>(...).

What's needed

Lower async function and await to a real state machine (resume points, suspend frames) rather than busy-wait. The current busy-wait lives in crates/perry-codegen/src/expr.rs (~L10687 per #997 agent). The trampoline-frame issue is at the V8↔Perry boundary, specifically native_callback_trampoline holding a scope while pump re-entry tries to settle V8 promises.

A starting plan:

1. Identify every site that emits the busy-wait loop { pump(); if done break } pattern in codegen.
2. Replace with proper continuation-passing or state-machine lowering — store the async frame's locals + resume index, return control to the runtime, resume from the index when the awaited future settles.
3. Ensure native_callback_trampoline releases the V8 scope before pumping.
4. Verify each of the four blocked packages flips to PASS in /tmp/perry-compat-sweep/.

Reproducer

mkdir -p /tmp/perry-self-fetch && cd /tmp/perry-self-fetch
cat > test.ts <<'TS'
import express from "express";
const app = express();
app.get("/ping", (_req, res) => { res.send("pong"); });
const server = app.listen(19100, async () => {
  console.log("listening");
  const r = await fetch("http://127.0.0.1:19100/ping");
  console.log("body=" + await r.text());
  server.close();
});
TS
cat > package.json <<'JSON'
{ "name": "test", "type": "module", "dependencies": { "express": "4.21.0" } }
JSON
npm install --silent --no-audit --no-fund
perry test.ts -o out --enable-js-runtime && ./out

Expected: listening then body=pong. Actual: hangs at listening until rc=124.

Same shape works through an external curl while the server is up, confirming the deadlock is internal to the busy-wait+V8 scope interaction, not the server.

Reference files

• crates/perry-codegen/src/expr.rs ~L10687 — busy-wait async lowering
• crates/perry-jsruntime/src/ops.rs — hyper accept loop + op_perry_fetch
• crates/perry-jsruntime/src/lib.rs — TOKIO_RUNTIME.enter() + with_runtime
• crates/perry-jsruntime/src/interop.rs — native_callback_trampoline
• PR fix(jsruntime/http): V8 listen keepalive + express handler smoke #997 (V8 listen keepalive + express handler smoke) — landed the partial workarounds
• PR fix(runtime): native await of V8 Promise handle in jose JWT roundtrip #1010 (js_async_step_chain awaits V8 Promise handles) — fixed a sibling marshalling bug

Impact

This single architectural change unblocks 4 of Perry's 16 sweep fixtures. Roughly: any npm package that imports node:http, runs a server, and exercises an in-process client against it. Without it, Perry can't run the canonical "self-test" idiom that 50%+ of compat smokes use.

[proggeramlug]

Progress at v0.5.1007 — #1041 landed the state machine

Sweep results after PR #1041 (fix(async): #1021 async-closure state machine + #1029 re-invocation idempotency):

Package	Before	After (v0.5.1007)
express	rc=124 timeout	✅ PASS
hono	rc=139 SIGSEGV on redirect	✅ PASS (also closed #1023)
fastify	rc=124 timeout	still rc=124 timeout
nestjs	empty stdout	still empty stdout

Two of four flipped. Remaining two:

fastify

Server listening on http://0.0.0.0:18932 then hangs. Different from express — fastify uses its own server adapter rather than http.createServer. Likely a separate pump-progress gap inside perry-stdlib/src/fastify/the bundled-fastify path, not the user-async-arrow lowering #1041 fixed.

nestjs

Silent empty stdout (rc=0). Reflect.metadata / decorator pipeline likely no-oping. May be a separate concern from the async lowering — worth bisecting.

Suggested next step: split this issue. The state-machine fix is in; remaining work is per-package and probably belongs in narrower follow-ups. I'll leave this issue open for the fastify/nestjs remainders; let me know if you'd rather close + reopen separately.