Merge remote-tracking branch 'origin/main' into codex/hook-retention-local

# Conflicts:
#	packages/world-local/src/index.ts
#	packages/world-local/src/storage/events-storage.ts

Author: NathanColosimo

.changeset/bench-histogram-run-trace-links.md

@@ -0,0 +1,4 @@
+---
+---
+
+Log the run id of each sequential-steps benchmark iteration alongside two Datadog APM links — the trigger request's trace and a span search for the run — so a suspicious STSO distribution can be opened in APM directly instead of hunted down by deployment id and time window.

.changeset/hardened-serialization-simplify.md

@@ -0,0 +1,6 @@
+---
+'@workflow/core': patch
+'workflow': patch
+---
+
+Simplify hardened serialization: intrinsic captures assert at import instead of degrading per use, `URLSearchParams` now serializes natively on Node 18, and bound-function getters are reported as guest code (previously a getter defined via `fn.bind()` executed with an empty report).

.changeset/lazy-hook-resumption.md

@@ -0,0 +1,11 @@
+---
+"workflow": minor
+"@workflow/core": minor
+"@workflow/world": minor
+"@workflow/world-vercel": minor
+"@workflow/world-local": minor
+"@workflow/web-shared": patch
+"@workflow/world-postgres": patch
+---
+
+Lazy hook resumption: on a fast path, `resumeHook()` writes the `hook_received` event and dispatches the workflow queue message concurrently instead of sequentially, cutting a round trip off resume latency. A `(runId, resumeId)` dedup constraint keeps the two writers converging on exactly one event; the runtime falls back to the sequential path when dedup support is unavailable or when `WORKFLOW_DISABLE_LAZY_HOOK_RESUME=1`. `resumeHook()` resolves to a `ResumedHook` (a `Hook` plus an optional `resilientResume: true` flag, set only when the direct write failed transiently and the resume was recovered via the queue consumer's re-ensure) — preserving the contract introduced alongside the resilient-resume work.

.changeset/sidebar-request-id.md

@@ -0,0 +1,5 @@
+---
+'@workflow/web-shared': minor
+---
+
+Surface an analytics event's `vercelId` in the run sidebar's Metadata list as a copyable "Request ID", and stop rendering the literal text `null` for a provenance id the analytics contract left empty. Also order Metadata rows missing from the panel's explicit order list after the listed ones instead of ahead of them, so a failed step's Error Code no longer renders above the step's own name.

docs/content/docs/v5/api-reference/workflow-api/resume-hook.mdx

@@ -50,7 +50,7 @@ showSections={["parameters"]}
 
 ### Returns
 
-Returns a `Promise<ResumedHook>`. `ResumedHook` extends `Hook` with an optional `resilientResume` flag, which is `true` when the `hook_received` event write failed with a transient error (429/5xx) and the payload will instead be delivered through the workflow queue (see the [resilient hook resume changelog](/docs/changelog/resilient-resume)). The `Hook` it extends resolves to:
+Returns a `Promise<ResumedHook>` — a `Hook` extended with an optional `resilientResume` flag. Resolving means the resume was accepted and the workflow will continue, whether the `hook_received` event was written directly or, on the parallel fast path, delivered through the workflow queue for the runtime to materialize (see the [lazy hook resume changelog](/docs/changelog/resilient-resume)). `resilientResume` is `true` only when the direct event write failed transiently and the resume was recovered through the queue; on the happy path it is absent. The resolved hook:
 
 <TSDoc
 definition={`

docs/content/docs/v5/changelog/resilient-resume.mdx

@@ -7,16 +7,16 @@ description: resumeHook() now tolerates transient event storage failures, as lon
 
 ## Motivation
 
-When event storage is degraded but the queue is up, `resumeHook()` previously failed entirely because the `hook_received` event write happens before the queue dispatch. This change brings `resumeHook()` to feature parity with [resilient `start()`](/docs/changelog/resilient-start): a transient event-write failure (429/5xx) no longer fails the resume when the payload can be delivered through the queue instead.
+`resumeHook()` used to write the `hook_received` event and dispatch the workflow queue message strictly one after the other, so every resume paid two sequential round trips and a transient event-storage failure failed the whole resume even when the queue was healthy. This change runs both writes **concurrently** — cutting a round trip off resume latency — and, on the same path, brings `resumeHook()` to parity with [resilient `start()`](/docs/changelog/resilient-start): a transient event-write failure no longer fails the resume when the payload can still be delivered through the queue.
 
 ## Design
 
-- `resumeHook()` writes the `hook_received` event first, then dispatches to the workflow queue. The order is deliberately sequential: dispatching in parallel could let the workflow runtime process the message and materialize a duplicate `hook_received` before the direct write commits.
-- If the event write fails with a retryable error (429/5xx), the queue dispatch carries a `hookInput` payload — the dehydrated hook payload plus a client-minted `resumeId` idempotency key and the hook token. The workflow runtime materializes the missing `hook_received` event from `hookInput` during replay, deduplicating by `resumeId` in case the direct write actually committed or the queue message is redelivered.
-- Replay itself also deduplicates: `hook_received` events sharing a `resumeId` belong to the same resume attempt, and only the first in the event log is delivered to workflow code. Even if concurrent queue redeliveries materialize the event twice, the payload reaches the workflow exactly once.
-- If the event write fails with any other error, or the queue dispatch fails, `resumeHook()` throws as before.
-- `resumeHook()` now returns `ResumedHook` (exported from `workflow/api`), which extends `Hook` with an optional `resilientResume` flag. The flag is `true` when the fallback path was taken — the resume is accepted and will be delivered through the queue.
+- On the fast path, `resumeHook()` writes the `hook_received` event and dispatches the workflow queue message **concurrently** (`Promise.allSettled`). The queue message carries a `hookInput` payload — the dehydrated hook payload plus a client-minted `resumeId` idempotency key, the hook token, and a payload digest.
+- A `(runId, resumeId)` dedup constraint keeps the two writers converging on **exactly one** `hook_received` event: whichever lands first wins, and the other is resolved server-side as success rather than a duplicate. The queue consumer idempotently re-ensures the event from `hookInput` before replay, so the resume is guaranteed even if the direct write never commits.
+- Replay also deduplicates: `hook_received` events sharing a `resumeId` belong to the same resume attempt, and only the first in the event log is delivered to workflow code. Even if a redelivery materializes the event twice, the payload reaches the workflow exactly once.
+- **Queue dispatch failure is fatal** — the run was not re-triggered, so no consumer will re-ensure the event, and `resumeHook()` throws. A transient event-write failure (429/5xx, a transport error, or an expected `(runId, resumeId)` conflict with the consumer's own re-ensure) is swallowed because the queue delivery still guarantees the resume; a terminal run surfaces as `HookNotFoundError`, and any other event-write error is rethrown.
+- `resumeHook()` returns `ResumedHook` (exported from `workflow/api`), which extends `Hook` with an optional `resilientResume` flag. The flag is `true` only when the direct write failed transiently and the resume was recovered through the queue; on the happy path and the sequential fallback it is absent.
 
 ## Compatibility
 
-The resilient path is only taken when the target workflow run's deployment is recent enough to understand `hookInput` on the queue message. Because runs keep executing on the deployment they were created on, a resume that targets a run from an older deployment fails fast with the original error instead — the previous behavior, allowing the caller to retry.
+The parallel fast path is gated per resume: it activates only when both the target run's queue consumer and the live backend independently attest dedup support (re-checked on every resume, so rollout and rollback both degrade safely). Otherwise — for oversized payloads, legacy runs, or with `WORKFLOW_DISABLE_LAZY_HOOK_RESUME=1` — `resumeHook()` falls back to the original sequential write-then-dispatch path. Because runs keep executing on the deployment they were created on, a resume targeting a run from an older deployment simply uses the sequential path.

docs/content/docs/v5/configuration/runtime-tuning.mdx

@@ -38,6 +38,13 @@ For example, a workflow can run a 10-minute inline step even with `WORKFLOW_REPL
 - Default: `3`
 - Recovery replays before replay divergence is recorded as corruption.
 
+### `WORKFLOW_DISABLE_LAZY_HOOK_RESUME`
+
+- Default: enabled (lazy hook resume on)
+- Resuming a hook persists the `hook_received` event and publishes the workflow invocation concurrently, cutting a round trip off resume latency. On this parallel path the queue message also carries the payload, so a transient event-write failure still resumes the run — the queue consumer re-ensures the `hook_received` event before replay. A backend `(runId, resumeId)` constraint keeps the two writers converging on exactly one event.
+- The runtime falls back to the sequential path automatically when the consumer or backend does not attest dedup support (or the payload is too large to inline on the queue message). On the sequential path the event is written *before* dispatch and its failure fails the resume — the fallback trades that resilience away to stay safe when dedup is not enforced, it does not preserve it.
+- Set `1` to force the sequential path as a kill switch. The chosen strategy is reported on the resume span as `workflow.hook.resume_strategy`.
+
 ### `WORKFLOW_PRECONDITION_GUARD`
 
 - Default: enabled

packages/core/e2e/benchmark.test.ts

@@ -193,6 +193,9 @@ interface StreamIterationResult {
 
 interface SequentialIterationResult {
   runId: string;
+  /** Datadog trace id for the `/api/bench` request that started this run, when
+   * the deployment's route reports one (older deployments won't). */
+  traceId?: string;
   /** STSO gaps preceding an 'inline' step (same warm process as the step
    * before it) — the framework's pure step-to-step overhead. */
   stsoInlineMs: number[];
@@ -221,6 +224,8 @@ interface BenchTriggerResponse {
   runId: string;
   /** Date.now() stamped in the route immediately before start(). */
   clientStart: number;
+  /** Datadog trace id for this request, when the route reports one. */
+  traceId?: string;
 }
 
 function withTimeout<T>(
@@ -272,7 +277,13 @@ async function triggerBenchRun(
       `bench trigger for ${workflowFn} returned malformed body: ${JSON.stringify(data)?.slice(0, 200)}`
     );
   }
-  return { runId: data.runId, clientStart: data.clientStart };
+  return {
+    runId: data.runId,
+    clientStart: data.clientStart,
+    // Optional: a deployment built before the route reported it simply logs
+    // the run id without a trace link.
+    traceId: typeof data.traceId === 'string' ? data.traceId : undefined,
+  };
 }
 
 /** Poll a run's return value to completion (the handle polls internally). */
@@ -340,7 +351,7 @@ async function runStreamIteration(
 async function runSequentialIteration(
   stepCount: number
 ): Promise<SequentialIterationResult> {
-  const { runId, clientStart } = await triggerBenchRun(
+  const { runId, clientStart, traceId } = await triggerBenchRun(
     'benchSequentialStepsWorkflow',
     [stepCount]
   );
@@ -366,6 +377,7 @@ async function runSequentialIteration(
 
     return {
       runId,
+      traceId,
       stsoInlineMs,
       stsoQueueHopMs,
       woMs: workflowOverheadMs(clientStart, steps),
@@ -636,6 +648,29 @@ const SCENARIO_DESCRIPTIONS = [
   },
 ];
 
+// Datadog APM permalink for a trace id. The benchmark deployment exports its
+// OTel spans to Datadog, and `/api/bench` returns the trace id of the request
+// that started each run.
+const DATADOG_TRACE_URL = 'https://app.datadoghq.com/apm/trace/';
+
+/**
+ * Datadog APM search for the spans tagged with a given `workflow.run.id`.
+ *
+ * The permalink above opens the *trigger's* trace, which under the default
+ * `WORKFLOW_TRACE_MODE=linked` holds only `workflow.start` plus span links out
+ * to the per-invocation trace roots — an entry point to the run rather than the
+ * run itself. This search lands straight on the execution spans, which is where
+ * an STSO investigation actually goes, so both are logged.
+ *
+ * Depends on `workflow.run.id` being an indexed span tag in the Datadog org. If
+ * it isn't, this returns an empty search and the trace permalink stays the way
+ * in; neither link is load-bearing for the benchmark itself.
+ */
+function datadogRunSearchUrl(runId: string): string {
+  const query = encodeURIComponent(`@workflow.run.id:${runId}`);
+  return `https://app.datadoghq.com/apm/traces?query=${query}`;
+}
+
 describe('workflow benchmarks', () => {
   // Preflight: prove the deployment executes workflows (and the trigger route
   // works) before any scenario spends its attempt budget. Without this, a
@@ -777,6 +812,19 @@ describe('workflow benchmarks', () => {
         extraAttempts: Math.max(2, Math.ceil(SEQUENTIAL_ITERATIONS * 0.5)),
       }
     );
+    // Name the runs behind the STSO histograms in this job's own log, right
+    // where they were produced. When a bucket looks wrong the investigation
+    // starts in APM, and this saves the usual hunt by deployment id + time
+    // window. Logged rather than rendered into the PR comment so it is also
+    // there for a local `pnpm bench` and for a run whose comment step never
+    // gets to execute.
+    for (const { runId, traceId } of results) {
+      console.log(
+        `[bench] ${SCENARIO_SEQUENTIAL} run ${runId}` +
+          (traceId ? ` — trace ${DATADOG_TRACE_URL}${traceId}` : '') +
+          ` — spans ${datadogRunSearchUrl(runId)}`
+      );
+    }
     // Report STSO split by whether the step that ends the gap was 'inline'
     // (same warm process as the step before it — pure framework overhead) or
     // a 'queue-hop' (first step of a fresh process — dispatch + reinit cost).

packages/core/e2e/e2e.test.ts

@@ -22,7 +22,6 @@ import {
   test,
 } from 'vitest';
 import { getTrustedSourcesHeaders } from '../../../scripts/trusted-sources-headers.mjs';
-import { QUEUE_HOOK_INPUT_MIN_VERSION } from '../src/capabilities';
 import type { Run } from '../src/runtime';
 import {
   getHookByToken,
@@ -31,7 +30,6 @@ import {
   healthCheck,
   start as rawStart,
   resumeHook,
-  setWorld,
 } from '../src/runtime';
 import {
   cliCancel,
@@ -4044,107 +4042,6 @@ describe('e2e', () => {
     }
   );
 
-  // ============================================================
-  // Resilient resume: hook payload delivered even when hook_received fails
-  // ============================================================
-  test(
-    'resilient resume: hookWorkflow receives payload when hook_received returns 500',
-    { timeout: 60_000 },
-    async () => {
-      const token = `resilient-resume-${Math.random().toString(36).slice(2)}`;
-      const customData = Math.random().toString(36).slice(2);
-
-      // Start the hook-awaiting workflow normally
-      const run = await start(await e2e('hookWorkflow'), [token, customData]);
-
-      // Wait for the hook to be registered
-      await sleep(5_000);
-
-      // The resilient path is gated on the target run's recorded
-      // `@workflow/core` version understanding `hookInput` on the queue
-      // payload (see QUEUE_HOOK_INPUT_MIN_VERSION in capabilities.ts).
-      // This build's own not-yet-bumped version sits below that published
-      // cutoff — deliberately, there is no own-version escape hatch — so
-      // simulate a run recorded by a capable release. The target deployment
-      // here genuinely understands `hookInput` (it is built from this same
-      // source), so the simulation is honest.
-      const capableVersion = QUEUE_HOOK_INPUT_MIN_VERSION;
-
-      // Build a stubbed world whose events.create throws a 500 on the
-      // hook_received write, but passes all other events through. The queue
-      // dispatch should still succeed, and the workflow runtime should
-      // materialize the missing hook_received event from `hookInput` on the
-      // queue message (resilient resume).
-      const realWorld = await getWorld();
-      const stubbedWorld: World = {
-        ...realWorld,
-        events: {
-          ...realWorld.events,
-          create: (async (...args: Parameters<World['events']['create']>) => {
-            const [, event] = args;
-            if (event.eventType === 'hook_received') {
-              throw new WorkflowWorldError('Simulated storage outage', {
-                status: 500,
-              });
-            }
-            return realWorld.events.create(...args);
-          }) as World['events']['create'],
-        },
-        runs: {
-          ...realWorld.runs,
-          // Fallback path (hooks without a stored resumeContext): rewrite
-          // the run's recorded core version to the capable release.
-          get: (async (...args: Parameters<World['runs']['get']>) => {
-            const run = await realWorld.runs.get(...args);
-            return {
-              ...run,
-              executionContext: {
-                ...run.executionContext,
-                workflowCoreVersion: capableVersion,
-              },
-            };
-          }) as World['runs']['get'],
-        },
-      };
-
-      const hook = await getHookByToken(token);
-      expect(hook.runId).toBe(run.runId);
-      // Fast path (hooks with a stored resumeContext): rewrite the recorded
-      // core version on the hook object we pass to resumeHook().
-      if (hook.resumeContext) {
-        hook.resumeContext.workflowCoreVersion = capableVersion;
-      }
-
-      // Swap in the stubbed world for the duration of the resumeHook() call.
-      // `resumeHook` uses `getWorld()` internally (no `world` option), so we
-      // use `setWorld()` to replace the cached instance and restore the real
-      // one afterwards.
-      setWorld(stubbedWorld);
-      let resumedHook: Awaited<ReturnType<typeof resumeHook>>;
-      try {
-        resumedHook = await resumeHook(hook, {
-          message: 'via-resilient-resume',
-          customData: (hook.metadata as any)?.customData,
-          done: true,
-        });
-      } finally {
-        setWorld(realWorld);
-      }
-
-      // The direct hook_received write failed with 500, so resumeHook should
-      // have taken the resilient path and flagged the returned hook.
-      expect(resumedHook.resilientResume).toBe(true);
-
-      // Despite hook_received failing, the workflow should still receive
-      // the payload via the runtime's queue-payload fallback.
-      const returnValue = await run.returnValue;
-      expect(returnValue).toHaveLength(1);
-      expect(returnValue[0].message).toBe('via-resilient-resume');
-      expect(returnValue[0].customData).toBe(customData);
-      expect(returnValue[0].done).toBe(true);
-    }
-  );
-
   test(
     'getterStepWorkflow - getter functions with "use step" directive',
     { timeout: 60_000 },