Retry replay divergence before failing event logs (#2212)

(cherry picked from commit 813cd9a)

Author: pranaygp

.changeset/replay-divergence-redelivery.md

@@ -0,0 +1,7 @@
+---
+'@workflow/core': patch
+'@workflow/errors': patch
+'@workflow/world': patch
+---
+
+Retry transient workflow replay divergence before classifying repeated divergence as a corrupted event log.

docs/content/docs/v4/errors/corrupted-event-log.mdx

@@ -9,21 +9,21 @@ related:
   - /docs/foundations/errors-and-retries
 ---
 
-This error occurs when the Workflow runtime encounters an event in the event log that no registered consumer can process. This means the event log is in an invalid state — typically due to duplicate or orphaned events.
+This error occurs when the Workflow runtime repeatedly cannot replay events in the event log. This usually means the event log is in an invalid state, such as duplicate or orphaned events, or that a runtime determinism bug persists across retry attempts.
 
-This is a **workflow-level fatal error**. It cannot be caught or handled inside your workflow code. A corrupted event log immediately fails the entire run without executing any more user code. The run must be retried from outside the workflow.
+This is a **workflow-level fatal error**. It cannot be caught or handled inside your workflow code. The runtime first retries transient replay divergence automatically; it marks the run as failed with this error only after replay still cannot recover.
 
 ## Error Message
 
 ```
-Unconsumed event in event log: eventType=<type>, correlationId=<id>, eventId=<id>. This indicates a corrupted or invalid event log.
+Workflow replay diverged <divergenceCount> times after <maxRecoveryReplays> recovery replays; latest divergent event was <eventId>. Last divergence: <details>
 ```
 
 ## Why This Happens
 
 Workflows persist their progress as an ordered event log. During replay, the runtime processes each event in sequence — every event must be consumed by a matching callback (e.g., a step or sleep waiting for its result). When an event has no matching consumer, the runtime cannot advance past it, which would block all subsequent events and hang the workflow indefinitely.
 
-Instead of silently hanging, the runtime raises a `WorkflowRuntimeError` to fail the workflow fast and surface the problem.
+Instead of silently hanging, the runtime retries a divergent replay before failing the workflow and surfacing this terminal error.
 
 Common scenarios that produce this error:
 
@@ -45,7 +45,7 @@ npm install workflow@latest
 
 ### 2. Retry the failed run
 
-Since this is a fatal error, the run is automatically marked as `failed`. You can re-run it using the **Re-run** button in the Workflow Dashboard.
+If this error is displayed, automatic replay recovery has already been exhausted and the run has been marked as `failed`. You can re-run it using the **Re-run** button in the Workflow Dashboard.
 
 ### 3. Report the issue
 

docs/content/docs/v4/errors/index.mdx

@@ -37,6 +37,9 @@ Fix common mistakes when creating and executing workflows in the **Workflow SDK*
     <Card href="/docs/errors/corrupted-event-log" title="corrupted-event-log">
         Learn how to handle corrupted or invalid event logs.
     </Card>
+    <Card href="/docs/errors/replay-divergence" title="replay-divergence">
+        Learn how workflow replay divergence is recovered automatically.
+    </Card>
     <Card href="/docs/errors/step-not-registered" title="step-not-registered">
         Resolve step not registered errors caused by deployment mismatches.
     </Card>

docs/content/docs/v4/errors/replay-divergence.mdx

@@ -0,0 +1,27 @@
+---
+title: replay-divergence
+description: A workflow replay temporarily followed a path that did not match its recorded events.
+type: troubleshooting
+summary: Understand automatic recovery when a workflow replay diverges from its event history.
+prerequisites:
+  - /docs/foundations/workflows-and-steps
+related:
+  - /docs/errors/corrupted-event-log
+  - /docs/foundations/errors-and-retries
+---
+
+A replay divergence occurs when one invocation of a workflow cannot consume the durable event history using the promises, hooks, sleeps, or steps it created during replay.
+
+This is an SDK/runtime signal, not an error thrown by your workflow code. It is not catchable inside a workflow function.
+
+## Automatic Recovery
+
+A single divergent replay does not prove that persisted history is corrupted. For example, asynchronous delivery ordering may cause one invocation to follow the wrong side of a race while another replay can follow the recorded history correctly.
+
+The runtime automatically queues another replay when an invocation reports `REPLAY_DIVERGENCE`. No terminal `run_failed` event is written during these recovery attempts.
+
+If recovery replays continue to diverge after the retry budget is exhausted, the runtime marks the run as failed with `CORRUPTED_EVENT_LOG` and records the latest divergent event for diagnosis.
+
+## What To Do
+
+Most replay divergence signals recover without action. If a run ultimately fails with `CORRUPTED_EVENT_LOG`, update to the latest `workflow` package and report the run ID and error details if the failure persists.

docs/content/docs/v5/errors/corrupted-event-log.mdx

@@ -9,21 +9,21 @@ related:
   - /docs/foundations/errors-and-retries
 ---
 
-This error occurs when the Workflow runtime encounters an event in the event log that no registered consumer can process. This means the event log is in an invalid state — typically due to duplicate or orphaned events.
+This error occurs when the Workflow runtime repeatedly cannot replay events in the event log. This usually means the event log is in an invalid state, such as duplicate or orphaned events, or that a runtime determinism bug persists across retry attempts.
 
-This is a **workflow-level fatal error**. It cannot be caught or handled inside your workflow code. A corrupted event log immediately fails the entire run without executing any more user code. The run must be retried from outside the workflow.
+This is a **workflow-level fatal error**. It cannot be caught or handled inside your workflow code. The runtime first retries transient replay divergence automatically; it marks the run as failed with this error only after replay still cannot recover.
 
 ## Error Message
 
 ```
-Unconsumed event in event log: eventType=<type>, correlationId=<id>, eventId=<id>. This indicates a corrupted or invalid event log.
+Workflow replay diverged <divergenceCount> times after <maxRecoveryReplays> recovery replays; latest divergent event was <eventId>. Last divergence: <details>
 ```
 
 ## Why This Happens
 
 Workflows persist their progress as an ordered event log. During replay, the runtime processes each event in sequence — every event must be consumed by a matching callback (e.g., a step or sleep waiting for its result). When an event has no matching consumer, the runtime cannot advance past it, which would block all subsequent events and hang the workflow indefinitely.
 
-Instead of silently hanging, the runtime raises a `WorkflowRuntimeError` to fail the workflow fast and surface the problem.
+Instead of silently hanging, the runtime retries a divergent replay before failing the workflow and surfacing this terminal error.
 
 Common scenarios that produce this error:
 
@@ -45,7 +45,7 @@ npm install workflow@latest
 
 ### 2. Retry the failed run
 
-Since this is a fatal error, the run is automatically marked as `failed`. You can re-run it using the **Re-run** button in the Workflow Dashboard.
+If this error is displayed, automatic replay recovery has already been exhausted and the run has been marked as `failed`. You can re-run it using the **Re-run** button in the Workflow Dashboard.
 
 ### 3. Report the issue
 

docs/content/docs/v5/errors/index.mdx

@@ -37,6 +37,9 @@ Fix common mistakes when creating and executing workflows in the **Workflow SDK*
     <Card href="/docs/errors/corrupted-event-log" title="corrupted-event-log">
         Learn how to handle corrupted or invalid event logs.
     </Card>
+    <Card href="/docs/errors/replay-divergence" title="replay-divergence">
+        Learn how workflow replay divergence is recovered automatically.
+    </Card>
     <Card href="/docs/errors/step-not-registered" title="step-not-registered">
         Resolve step not registered errors caused by deployment mismatches.
     </Card>

docs/content/docs/v5/errors/replay-divergence.mdx

@@ -0,0 +1,27 @@
+---
+title: replay-divergence
+description: A workflow replay temporarily followed a path that did not match its recorded events.
+type: troubleshooting
+summary: Understand automatic recovery when a workflow replay diverges from its event history.
+prerequisites:
+  - /docs/foundations/workflows-and-steps
+related:
+  - /docs/errors/corrupted-event-log
+  - /docs/foundations/errors-and-retries
+---
+
+A replay divergence occurs when one invocation of a workflow cannot consume the durable event history using the promises, hooks, sleeps, or steps it created during replay.
+
+This is an SDK/runtime signal, not an error thrown by your workflow code. It is not catchable inside a workflow function.
+
+## Automatic Recovery
+
+A single divergent replay does not prove that persisted history is corrupted. For example, asynchronous delivery ordering may cause one invocation to follow the wrong side of a race while another replay can follow the recorded history correctly.
+
+The runtime automatically queues another replay when an invocation reports `REPLAY_DIVERGENCE`. No terminal `run_failed` event is written during these recovery attempts.
+
+If recovery replays continue to diverge after the retry budget is exhausted, the runtime marks the run as failed with `CORRUPTED_EVENT_LOG` and records the latest divergent event for diagnosis.
+
+## What To Do
+
+Most replay divergence signals recover without action. If a run ultimately fails with `CORRUPTED_EVENT_LOG`, update to the latest `workflow` package and report the run ID and error details if the failure persists.

packages/core/src/abort-controller.test.ts

@@ -6,7 +6,7 @@
  * (for real-time step propagation).
  */
 
-import { CorruptedEventLogError } from '@workflow/errors';
+import { ReplayDivergenceError } from '@workflow/errors';
 import { withResolvers } from '@workflow/utils';
 import type { Event } from '@workflow/world';
 import * as nanoid from 'nanoid';
@@ -121,7 +121,7 @@ describe('AbortController in workflow VM', () => {
       expect(controller.signal.aborted).toBe(true);
     });
 
-    it('reports a CorruptedEventLogError when abort hook_received token mismatches the controller', async () => {
+    it('reports a ReplayDivergenceError when abort hook_received token mismatches the controller', async () => {
       ctx = setupWorkflowContext([]);
       const ProbeAbortController = createCreateAbortController(ctx);
       new ProbeAbortController();
@@ -160,7 +160,8 @@ describe('AbortController in workflow VM', () => {
       new AbortController();
 
       const workflowError = await errorReceived.promise;
-      expect(workflowError).toBeInstanceOf(CorruptedEventLogError);
+      expect(workflowError).toBeInstanceOf(ReplayDivergenceError);
+      expect((workflowError as ReplayDivergenceError).eventId).toBe('evnt_0');
       expect(workflowError?.message).toContain('hook_received');
       expect(workflowError?.message).toContain('wrong-token');
       expect(workflowError?.message).toContain(probeHookItem.token);

packages/core/src/classify-error.test.ts

@@ -1,6 +1,7 @@
 import {
   CorruptedEventLogError,
   HookConflictError,
+  ReplayDivergenceError,
   RUN_ERROR_CODES,
   RuntimeDecryptionError,
   WorkflowNotRegisteredError,
@@ -17,6 +18,16 @@ describe('classifyRunError', () => {
     ).toBe(RUN_ERROR_CODES.CORRUPTED_EVENT_LOG);
   });
 
+  it('classifies ReplayDivergenceError as REPLAY_DIVERGENCE', () => {
+    expect(
+      classifyRunError(
+        new ReplayDivergenceError('replay took another path', {
+          eventId: 'event-1',
+        })
+      )
+    ).toBe(RUN_ERROR_CODES.REPLAY_DIVERGENCE);
+  });
+
   it('classifies WorkflowRuntimeError as RUNTIME_ERROR', () => {
     expect(
       classifyRunError(new WorkflowRuntimeError('corrupted event log'))

packages/core/src/classify-error.ts

@@ -1,5 +1,6 @@
 import {
   CorruptedEventLogError,
+  ReplayDivergenceError,
   RUN_ERROR_CODES,
   type RunErrorCode,
   RuntimeDecryptionError,
@@ -67,6 +68,10 @@ export function isWorldContractError(err: unknown): err is WorkflowWorldError {
 }
 
 export function classifyRunError(err: unknown): RunErrorCode {
+  if (ReplayDivergenceError.is(err)) {
+    return RUN_ERROR_CODES.REPLAY_DIVERGENCE;
+  }
+
   if (CorruptedEventLogError.is(err)) {
     return RUN_ERROR_CODES.CORRUPTED_EVENT_LOG;
   }