fix(core): re-route runs delivered to the wrong deployment (#2960)

## Summary & Motivation

A queue callback that reaches a deployment other than the one its run is pinned to derives the per-run encryption key from the wrong master key, so the delivery fails before user code runs and the run dies as a blank "exceeded max retries". The delivery is re-enqueued explicitly addressed to the run's own deployment — strictly better-targeted than the send that misrouted — and the run is failed with the new `DEPLOYMENT_MISMATCH` error code only once `WORKFLOW_DEPLOYMENT_MISMATCH_MAX_RETRIES` (default 3) is spent. Gated on the new World capability `deploymentAffinity`, so worlds with synthetic or version-tagged deployment ids are unaffected.

## Test Plan

Unit tests added for the guard and both runtime paths; local vitest and typechecks pass.

Author: alangenfeld

.changeset/fail-cross-deployment-runs.md

@@ -0,0 +1,8 @@
+---
+"@workflow/core": patch
+"@workflow/errors": minor
+"@workflow/world": minor
+"@workflow/world-vercel": patch
+---
+
+Re-route a queue delivery that reaches a deployment other than the one its run is pinned to, retry transient publishing failures through normal queue redelivery, and fail the run with the new `DEPLOYMENT_MISMATCH` error code once the recovery budget is spent.

docs/content/docs/v4/errors/deployment-mismatch.mdx

@@ -0,0 +1,71 @@
+---
+title: deployment-mismatch
+description: A workflow run was delivered to a deployment other than the one it is pinned to.
+type: troubleshooting
+summary: Understand how Workflow recovers from a misrouted delivery, and why a run eventually fails with DEPLOYMENT_MISMATCH.
+prerequisites:
+  - /docs/foundations/workflows-and-steps
+related:
+  - /docs/foundations/versioning
+  - /docs/errors/runtime-decryption-failed
+  - /docs/foundations/errors-and-retries
+---
+
+Every run is pinned to a single deployment when it starts. When a queued workflow or step callback is delivered to a **different** deployment, Workflow does not execute it there. Instead it re-routes the message to the deployment the run is pinned to, and only if the run keeps arriving elsewhere does it fail with the `DEPLOYMENT_MISMATCH` classification.
+
+This is an SDK/runtime signal, not an error thrown by your workflow code, and it is not catchable inside a workflow function.
+
+## Error Message
+
+```
+Workflow run "wrun_..." is pinned to deployment "dpl_A", but was received by deployment "dpl_B". The runtime re-routed the message to "dpl_A" 3 times and it kept arriving elsewhere, so the run was stopped to protect against code-skew errors. Verify that the run's deployment is still available and that queue callbacks are routed to it.
+```
+
+When the queue definitively reports that the run's deployment cannot be reached — it was deleted, or aged out of its retention window — no re-route is possible and the message omits the re-routing clause. Transient or unknown publishing failures leave the current delivery unacknowledged so the queue can redeliver it; they do not fail the run or consume this recovery budget.
+
+## Why A Run Is Pinned
+
+A run's deployment is chosen once, at [`start()`](/docs/api-reference/workflow-api/start):
+
+- By default it is the deployment that called `start()` — see [Versioning](/docs/foundations/versioning) for why runs are pinned this way.
+- With `start(workflow, args, { deploymentId })` it is the id you pass, so a run can deliberately target a deployment other than the one that created it.
+- With `deploymentId: "latest"` it is the most recent deployment for the current environment, resolved at start time.
+
+Whichever it is, that `deploymentId` is recorded on the run, and every subsequent workflow replay and step execution must happen on that deployment. Continuing on a different one is unsafe:
+
+1. **Code skew.** The workflow and step bundles on the receiving deployment may not match the code that produced the run's recorded history, so replay could diverge or produce incorrect results.
+2. **Encryption.** Step inputs and other event-log payloads are encrypted with a per-run key derived from the pinned deployment's key material. A different deployment derives the wrong key and cannot decrypt them — previously the source of a confusing [runtime-decryption-failed](/docs/errors/runtime-decryption-failed) that exhausted retries with no clear cause.
+
+So the runtime checks the pinned deployment before it executes anything, and `DEPLOYMENT_MISMATCH` names the result — instead of the mismatch surfacing later as an unrelated decryption failure.
+
+## Automatic Recovery
+
+A deployment that receives a run it does not own first tries to fix the delivery rather than fail the run:
+
+1. It re-enqueues the message **explicitly addressed** to the run's own deployment. This is strictly better-addressed than the send that misrouted, which inherited the producing deployment's ambient id.
+2. Delivery is delayed with a short exponential backoff (1s, 2s, 4s).
+3. If the run keeps arriving at the wrong deployment, the run is failed with `DEPLOYMENT_MISMATCH` after `WORKFLOW_DEPLOYMENT_MISMATCH_MAX_RETRIES` attempts (default `3`). Set it to `0` to fail on the first misrouted delivery instead.
+
+Nothing is executed on the wrong deployment during recovery: no workflow code, no step body, no `step_started`, and no hook resume. Whatever the delivery was carrying travels with it, so a pending step keeps its identity and a hook resume keeps its payload — they run on the deployment that can actually decrypt them.
+
+Recovery attempts do not create events on the run, so a run that self-heals looks completely normal. They are reported on the invocation's trace span (`workflow.deployment.pinned_id`, `workflow.deployment_mismatch.retry_count`, `workflow.deployment_mismatch.recovered`) and as a runtime warning in your function logs.
+
+## What To Do
+
+- **Re-run from the current deployment.** Trigger the workflow again from your latest deployment (or use the **Re-run** button in the Workflow Dashboard). The new run is pinned to the current deployment.
+- **Keep a run's deployment available** for the lifetime of that run. A run whose deployment has been deleted or has aged out cannot be resumed and must be re-run — recovery cannot help, so these fail on the first misrouted delivery. This applies to runs started with an explicit `deploymentId` too: pinning a run to an older deployment keeps it dependent on that deployment for its whole lifetime.
+- **Report it** if the pinned deployment was still available. Include both deployment ids and the run id from the error message, plus the trace span attributes above — a run that failed this way despite a reachable target is a routing fault worth investigating rather than something to work around.
+
+## This Error Cannot Be Caught
+
+Like other runtime signals, `DEPLOYMENT_MISMATCH` is **not catchable** inside your workflow function — the run is failed before any workflow or step code executes on the receiving deployment. Check the run status from outside instead:
+
+```typescript lineNumbers
+import { getRun } from "workflow/api";
+
+const run = getRun("wrun_abc123");
+const status = await run.status;
+if (status === "failed") {
+  console.error("Run failed");
+}
+```

docs/content/docs/v5/configuration/runtime-tuning.mdx

@@ -55,6 +55,14 @@ For example, a workflow can run a 10-minute inline step even with `WORKFLOW_REPL
 - The runtime falls back to the sequential path automatically when the consumer or backend does not attest dedup support (or the payload is too large to inline on the queue message). On the sequential path the event is written *before* dispatch and its failure fails the resume — the fallback trades that resilience away to stay safe when dedup is not enforced, it does not preserve it.
 - Set `1` to force the sequential path as a kill switch. The chosen strategy is reported on the resume span as `workflow.hook.resume_strategy`.
 
+### `WORKFLOW_DEPLOYMENT_MISMATCH_MAX_RETRIES`
+
+- Default: `3`
+- Times a delivery that reached a deployment other than the one its run is pinned to is re-routed to that deployment before the run is failed with [`DEPLOYMENT_MISMATCH`](/docs/errors/deployment-mismatch).
+- Re-routed deliveries back off exponentially (1s, 2s, 4s). Set to `0` to fail the run on the first misrouted delivery.
+- Only applies to Worlds with atomic, immutable deployments (the Vercel World). A run whose pinned deployment cannot be reached at all fails immediately regardless of this value.
+- Transient or unknown queue publishing failures use normal queue redelivery and do not consume this budget.
+
 ### `WORKFLOW_PRECONDITION_GUARD`
 
 - Default: enabled

docs/content/docs/v5/errors/deployment-mismatch.mdx

@@ -0,0 +1,71 @@
+---
+title: deployment-mismatch
+description: A workflow run was delivered to a deployment other than the one it is pinned to.
+type: troubleshooting
+summary: Understand how Workflow recovers from a misrouted delivery, and why a run eventually fails with DEPLOYMENT_MISMATCH.
+prerequisites:
+  - /docs/foundations/workflows-and-steps
+related:
+  - /docs/foundations/versioning
+  - /docs/errors/runtime-decryption-failed
+  - /docs/foundations/errors-and-retries
+---
+
+Every run is pinned to a single deployment when it starts. When a queued workflow or step callback is delivered to a **different** deployment, Workflow does not execute it there. Instead it re-routes the message to the deployment the run is pinned to, and only if the run keeps arriving elsewhere does it fail with the `DEPLOYMENT_MISMATCH` classification.
+
+This is an SDK/runtime signal, not an error thrown by your workflow code, and it is not catchable inside a workflow function.
+
+## Error Message
+
+```
+Workflow run "wrun_..." is pinned to deployment "dpl_A", but was received by deployment "dpl_B". The runtime re-routed the message to "dpl_A" 3 times and it kept arriving elsewhere, so the run was stopped to protect against code-skew errors. Verify that the run's deployment is still available and that queue callbacks are routed to it.
+```
+
+When the queue definitively reports that the run's deployment cannot be reached — it was deleted, or aged out of its retention window — no re-route is possible and the message omits the re-routing clause. Transient or unknown publishing failures leave the current delivery unacknowledged so the queue can redeliver it; they do not fail the run or consume this recovery budget.
+
+## Why A Run Is Pinned
+
+A run's deployment is chosen once, at [`start()`](/docs/api-reference/workflow-api/start):
+
+- By default it is the deployment that called `start()` — see [Versioning](/docs/foundations/versioning) for why runs are pinned this way.
+- With `start(workflow, args, { deploymentId })` it is the id you pass, so a run can deliberately target a deployment other than the one that created it.
+- With `deploymentId: "latest"` it is the most recent deployment for the current environment, resolved at start time.
+
+Whichever it is, that `deploymentId` is recorded on the run, and every subsequent workflow replay and step execution must happen on that deployment. Continuing on a different one is unsafe:
+
+1. **Code skew.** The workflow and step bundles on the receiving deployment may not match the code that produced the run's recorded history, so replay could diverge or produce incorrect results.
+2. **Encryption.** Step inputs and other event-log payloads are encrypted with a per-run key derived from the pinned deployment's key material. A different deployment derives the wrong key and cannot decrypt them — previously the source of a confusing [runtime-decryption-failed](/docs/errors/runtime-decryption-failed) that exhausted retries with no clear cause.
+
+So the runtime checks the pinned deployment before it executes anything, and `DEPLOYMENT_MISMATCH` names the result — instead of the mismatch surfacing later as an unrelated decryption failure.
+
+## Automatic Recovery
+
+A deployment that receives a run it does not own first tries to fix the delivery rather than fail the run:
+
+1. It re-enqueues the message **explicitly addressed** to the run's own deployment. This is strictly better-addressed than the send that misrouted, which inherited the producing deployment's ambient id.
+2. Delivery is delayed with a short exponential backoff (1s, 2s, 4s).
+3. If the run keeps arriving at the wrong deployment, the run is failed with `DEPLOYMENT_MISMATCH` after `WORKFLOW_DEPLOYMENT_MISMATCH_MAX_RETRIES` attempts (default `3`). Set it to `0` to fail on the first misrouted delivery instead.
+
+Nothing is executed on the wrong deployment during recovery: no workflow code, no step body, no `step_started`, and no hook resume. Whatever the delivery was carrying travels with it, so a pending step keeps its identity and a hook resume keeps its payload — they run on the deployment that can actually decrypt them.
+
+Recovery attempts do not create events on the run, so a run that self-heals looks completely normal. They are reported on the invocation's trace span (`workflow.deployment.pinned_id`, `workflow.deployment_mismatch.retry_count`, `workflow.deployment_mismatch.recovered`) and as a runtime warning in your function logs.
+
+## What To Do
+
+- **Re-run from the current deployment.** Trigger the workflow again from your latest deployment (or use the **Re-run** button in the Workflow Dashboard). The new run is pinned to the current deployment.
+- **Keep a run's deployment available** for the lifetime of that run. A run whose deployment has been deleted or has aged out cannot be resumed and must be re-run — recovery cannot help, so these fail on the first misrouted delivery. This applies to runs started with an explicit `deploymentId` too: pinning a run to an older deployment keeps it dependent on that deployment for its whole lifetime.
+- **Report it** if the pinned deployment was still available. Include both deployment ids and the run id from the error message, plus the trace span attributes above — a run that failed this way despite a reachable target is a routing fault worth investigating rather than something to work around.
+
+## This Error Cannot Be Caught
+
+Like other runtime signals, `DEPLOYMENT_MISMATCH` is **not catchable** inside your workflow function — the run is failed before any workflow or step code executes on the receiving deployment. Check the run status from outside instead:
+
+```typescript lineNumbers
+import { getRun } from "workflow/api";
+
+const run = getRun("wrun_abc123");
+const status = await run.status;
+if (status === "failed") {
+  console.error("Run failed");
+}
+```

packages/core/src/classify-error.test.ts

@@ -7,6 +7,7 @@ import {
   RuntimeDecryptionError,
   ThrottleError,
   TooEarlyError,
+  WorkflowDeploymentMismatchError,
   WorkflowNotRegisteredError,
   WorkflowRuntimeError,
   WorkflowWorldError,
@@ -49,6 +50,18 @@ describe('classifyRunError', () => {
     );
   });
 
+  it('classifies WorkflowDeploymentMismatchError as DEPLOYMENT_MISMATCH', () => {
+    expect(
+      classifyRunError(
+        new WorkflowDeploymentMismatchError(
+          'wrun_test',
+          'dpl_expected',
+          'dpl_actual'
+        )
+      )
+    ).toBe(RUN_ERROR_CODES.DEPLOYMENT_MISMATCH);
+  });
+
   it('classifies plain Error as USER_ERROR', () => {
     expect(classifyRunError(new Error('user code broke'))).toBe(
       RUN_ERROR_CODES.USER_ERROR

packages/core/src/classify-error.ts

@@ -7,6 +7,7 @@ import {
   RuntimeDecryptionError,
   StepNotRegisteredError,
   ThrottleError,
+  WorkflowDeploymentMismatchError,
   WorkflowNotRegisteredError,
   WorkflowRuntimeError,
   WorkflowWorldError,
@@ -121,6 +122,10 @@ export function classifyRunError(err: unknown): RunErrorCode {
     return RUN_ERROR_CODES.MAX_EVENTS_EXCEEDED;
   }
 
+  if (WorkflowDeploymentMismatchError.is(err)) {
+    return RUN_ERROR_CODES.DEPLOYMENT_MISMATCH;
+  }
+
   // World-layer faults — both a malformed response (contract violation) and a
   // transient infrastructure failure (throttle / 5xx / transport / timeout,
   // e.g. a firewall challenge) — are the backend's fault, not the user's.

packages/core/src/describe-error.test.ts

@@ -3,6 +3,7 @@ import {
   RUN_ERROR_CODES,
   SerializationError,
   StepNotRegisteredError,
+  WorkflowDeploymentMismatchError,
   WorkflowRuntimeError,
 } from '@workflow/errors';
 import { describe, expect, test } from 'vitest';
@@ -118,6 +119,15 @@ describe('describeError', () => {
     expect(result.hint).toContain('maximum number of events');
   });
 
+  test('WorkflowDeploymentMismatchError is attributed to the SDK with the deployment hint', () => {
+    const result = describeError(
+      new WorkflowDeploymentMismatchError('wrun', 'dpl_a', 'dpl_b')
+    );
+    expect(result.attribution).toBe('sdk');
+    expect(result.errorCode).toBe(RUN_ERROR_CODES.DEPLOYMENT_MISMATCH);
+    expect(result.hint).toContain('deployment it is pinned to');
+  });
+
   test('precomputed errorCode wins over classifyRunError when both are provided', () => {
     // A plain Error would classify as USER_ERROR, but passing REPLAY_TIMEOUT
     // explicitly overrides that — useful for callers that know the failure
@@ -220,6 +230,16 @@ describe('describeRunError', () => {
     expect(result.hint).toContain('SDK contract');
   });
 
+  test('DEPLOYMENT_MISMATCH errorCode is attributed to the SDK', () => {
+    const result = describeRunError({
+      errorCode: RUN_ERROR_CODES.DEPLOYMENT_MISMATCH,
+      errorName: 'WorkflowDeploymentMismatchError',
+    });
+    expect(result.attribution).toBe('sdk');
+    expect(result.errorCode).toBe(RUN_ERROR_CODES.DEPLOYMENT_MISMATCH);
+    expect(result.hint).toContain('deployment it is pinned to');
+  });
+
   test('RUNTIME_ERROR code without errorName still lands as SDK', () => {
     const result = describeRunError({
       errorCode: RUN_ERROR_CODES.RUNTIME_ERROR,

packages/core/src/describe-error.ts

@@ -87,6 +87,8 @@ const MAX_EVENTS_HINT =
   'The workflow exceeded the maximum number of events per run. This usually means unbounded work in the workflow function — e.g. a loop that keeps creating steps without terminating. Break long-running workflows into child workflows to stay under the limit.';
 const WORLD_CONTRACT_HINT =
   'The workflow backend returned data that violated the SDK contract. This is not retryable; please report it with the stack trace and runId.';
+const DEPLOYMENT_MISMATCH_HINT =
+  "The run was delivered to a deployment other than the deployment it is pinned to, and was stopped to protect against code-skew errors after the runtime failed to re-route it there. Verify that the run's deployment is still available and that queue callbacks route to it.";
 
 function normalizeErrorCode(code: string | undefined): RunErrorCode {
   // Values read back from persisted events are `string | undefined` — we
@@ -149,6 +151,9 @@ export function describeRunError(
       hint: WORLD_CONTRACT_HINT,
     };
   }
+  if (errorCode === RUN_ERROR_CODES.DEPLOYMENT_MISMATCH) {
+    return { attribution: 'sdk', errorCode, hint: DEPLOYMENT_MISMATCH_HINT };
+  }
   if (name === 'WorkflowRuntimeError' || name === 'StepNotRegisteredError') {
     return { attribution: 'sdk', errorCode, hint: RUNTIME_ERROR_HINT };
   }
@@ -209,6 +214,16 @@ export function describeError(
     };
   }
 
+  // Check DEPLOYMENT_MISMATCH before the generic WorkflowRuntimeError branch —
+  // WorkflowDeploymentMismatchError subclasses it, but has its own code + hint.
+  if (effectiveCode === RUN_ERROR_CODES.DEPLOYMENT_MISMATCH) {
+    return {
+      attribution: 'sdk',
+      errorCode: effectiveCode,
+      hint: DEPLOYMENT_MISMATCH_HINT,
+    };
+  }
+
   if (err instanceof WorkflowRuntimeError) {
     return {
       attribution: 'sdk',