Various Performance Improvements (#932)

* Parallelize async operations in step handler for performance

- Parallelize getPort(), getSpanKind(), and world.steps.get() calls
- Start step_started event creation while hydrating arguments (CPU work)
- Parallelize step_completed event with trace serialization

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix race condition: await step_started before hydration

Reverts Optimization 1 to fix a race condition where hydrateStepArguments()
could throw before stepStartedPromise was awaited, causing stale step.attempt
in the catch handler and potentially allowing extra retries.

Optimizations 0 and 2 are preserved as they don't have this issue.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Skip initial world.steps.get() in step handler for performance

Eliminate the world.steps.get() HTTP call by calling step_started first
and relying on server-side validation. This saves 50-80ms per step
execution by removing one HTTP round-trip.

The server (workflow-server) now validates:
- Step not in terminal state (returns 409)
- retryAfter timestamp reached (returns 425 with Retry-After header)
- Workflow still active (returns 410 if completed)

Changes:
- Remove world.steps.get() from initial Promise.all
- Call step_started first to get step entity and validate state
- Handle 409 (terminal state) by re-queueing workflow
- Handle 425 (retryAfter not reached) by returning timeout
- Handle 410 (workflow gone) as no-op

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add retryAfter validation to local and postgres worlds

Add server-side retryAfter validation to match workflow-server behavior:
- Check retryAfter timestamp before allowing step_started
- Return HTTP 425 with retryAfter timestamp in response meta
- Clear retryAfter field when step starts successfully

This ensures consistent behavior across all world implementations
and allows the step-handler optimization to work correctly.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix step terminal state HTTP status code to 409 (Conflict)

Aligns local and postgres worlds with workflow-server, which returns
409 via InvalidOperationStateError for step in terminal state.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix world-vercel telemetry to use parent application's tracer

The world-vercel package was creating spans under a separate
'workflow-world-vercel' service name, causing HTTP spans for
workflow-server API calls (step_started, step_completed) to be
filtered out when viewing traces for the main application service.

Now uses the same 'workflow' tracer name as @workflow/core to
ensure all spans are reported under the parent application's service.

* Update /demo command to include OTEL tracing with Jaeger

- Start Jaeger container for local trace visualization
- Configure OTEL exporter environment variables for dev server
- Open Jaeger UI automatically
- Add documentation about available trace attributes

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add OTEL instrumentation to world-local and improve trace consistency

- Add telemetry.ts and instrumentObject.ts to world-local for tracing parity
  with world-vercel (world.runs, world.steps, world.events, world.hooks spans)
- Change workflow span name from uppercase "WORKFLOW" to lowercase "workflow"
  for consistency with step spans and OTEL naming conventions
- Add step.execute child span to trace actual user step function execution
  separately from step handler infrastructure

These changes enable local development to have the same observability as
production deployments, making performance analysis and debugging easier.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Use uppercase WORKFLOW and STEP span names for consistency with HTTP spans

HTTP spans use uppercase methods like "GET /path" and "POST /path".
Following the same convention, workflow and step spans now use:
- WORKFLOW <workflow-name>
- STEP <step-name>

Child spans (workflow.run, workflow.loadEvents, step.execute, world.events.create)
remain lowercase as they represent internal operations, not top-level entries.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add W3C trace context headers to step queue messages

Include traceparent and tracestate headers when queueing step
execution messages. This enables automatic trace propagation
by Vercel's infrastructure, potentially linking step invocation
spans to the parent workflow trace.

The trace carrier is now serialized once and included in both:
- Payload: for manual context restoration in step handler
- Headers: for automatic HTTP-based trace propagation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Improve OTEL tracing with service attribution and enhanced instrumentation

- Add peer.service attributes for workflow-server and VQS for Datadog service maps
- Rename queueMessage span to queue.publish for consistency
- Add step.hydrate, step.dehydrate, and workflow.replay spans
- Include event type in world.events.create span names (e.g., "world.events.create step_started")
- Add span.recordException() for errors with category classification (fatal/retryable/transient)
- Add span events for milestones: retry.scheduled, step.skipped, step.delayed
- Add HTTP semantic conventions with peer.service for world-vercel HTTP calls
- Add baggage propagation for workflow context (run_id, workflow_name)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Update changesets to reflect all PR changes

- step-handler-parallelization.md: Add race condition fix and 409 status code fix
- world-vercel-telemetry-tracer.md: Add peer.service and event type in span names
- otel-tracing-improvements.md: New changeset for comprehensive OTEL improvements

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Use span name for rpc.method to show event type in Datadog resource

Datadog derives the resource name from rpc.method attribute. Updated to
use the full span name (which includes event type) instead of just the
method name, so Datadog shows "world.events.create step_started" instead
of "world.events.create workflow-server".

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Use lazy ref resolution for events where client discards response data

Skip expensive S3 ref resolution (~200-460ms) for event types where
the client doesn't use the response entity data (step_created,
step_completed, step_failed, run_completed, etc). Only resolve refs
for run_created, run_started, and step_started where the client
reads the response.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Update changeset to include lazy ref resolution optimization

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Address PR review comments: module scope const, comment cleanup, status code audit

- Move eventsNeedingResolve Set to module scope (PR review #2)
- Rewrite changelog-style OPTIMIZATION comments as current-state docs (PR review #4)
- Fix run terminal state errors: 410 → 409 in world-local and world-postgres to
  match workflow-server's InvalidOperationStateError (409) (PR review #3)
- Fix remaining step terminal state 410 → 409 in world-postgres fallback paths

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

Author: pranaygp

.changeset/otel-tracing-improvements.md

@@ -0,0 +1,17 @@
+---
+"@workflow/core": patch
+"@workflow/world-local": patch
+---
+
+Improve OpenTelemetry tracing instrumentation
+
+- Add W3C trace context headers to step queue messages for cross-service trace linking
+- Add `peer.service` and RPC semantic conventions for external service attribution
+- Add `step.hydrate` and `step.dehydrate` spans for argument serialization visibility
+- Add `workflow.replay` span for workflow event replay tracking
+- Rename `queueMessage` span to `queue.publish` following OTEL messaging conventions
+- Add OTEL baggage propagation for workflow context (`workflow.run_id`, `workflow.name`)
+- Add span events for milestones: `retry.scheduled`, `step.skipped`, `step.delayed`
+- Enhance error telemetry with `recordException()` and error categorization (fatal/retryable/transient)
+- Use uppercase span names (WORKFLOW, STEP) for consistency with HTTP spans
+- Add world-local OTEL instrumentation matching world-vercel

.changeset/step-handler-parallelization.md

@@ -0,0 +1,12 @@
+---
+"@workflow/core": patch
+"@workflow/world-local": patch
+"@workflow/world-postgres": patch
+---
+
+Optimize step handler performance and improve server-side validation
+
+- Skip initial `world.steps.get()` call in step handler (saves one HTTP round-trip)
+- Add server-side `retryAfter` validation to local and postgres worlds (HTTP 425 when not reached)
+- Fix HTTP status code for step terminal state: return 409 (Conflict) instead of 410
+- Fix race condition: await `step_started` event before hydration to ensure correct attempt count

.changeset/world-vercel-telemetry-tracer.md

@@ -0,0 +1,10 @@
+---
+"@workflow/world-vercel": patch
+---
+
+Improve world-vercel telemetry and event creation performance
+
+- Use parent application's 'workflow' tracer instead of separate service name
+- Add `peer.service` and RPC semantic conventions for Datadog service maps
+- Include event type in `world.events.create` span names (e.g., `world.events.create step_started`)
+- Use lazy ref resolution for fire-and-forget events to skip S3 ref resolution (~200-460ms savings)

.claude/commands/demo.md

@@ -1,9 +1,51 @@
 ---
 description: Run the 7_full demo workflow
-allowed-tools: Bash(curl:*), Bash(npx workflow:*), Bash(pnpm dev)
+allowed-tools: Bash(curl:*), Bash(npx workflow:*), Bash(pnpm dev), Bash(docker *), Bash(open *)
 ---
 
+Run the demo workflow with OpenTelemetry tracing enabled.
 
-Start the $ARUGMENTS workbench (default to the nextjs turboback workbench  available in the workbenches directory). Run it in dev mode, and also start the workflow web UI (run `npx workflow web` inside the appropriate workbench directory).
+## Steps
 
-Then trigger the 7_full.ts workflow example. you can see how to trigger a specific example by looking at the trigger API route for the workbench - it is probably just a POST request using bash (maybe curl) to this endpoint: <http://localhost:3000/api/trigger\?workflowFile\=workflows/7_full.ts\&workflowFn\=handleUserSignup>>
+1. **Start Jaeger** for OTEL trace visualization (if not already running):
+   ```bash
+   docker run -d --name jaeger-otel \
+     -p 16686:16686 \
+     -p 4317:4317 \
+     -p 4318:4318 \
+     jaegertracing/jaeger:2.4.0 2>&1 || docker start jaeger-otel
+   ```
+
+2. **Open the Jaeger UI** in the browser:
+   ```bash
+   open http://localhost:16686
+   ```
+
+3. **Start the workbench** (default: nextjs-turbopack) with OTEL tracing enabled:
+   ```bash
+   cd workbench/nextjs-turbopack
+   OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318" \
+   OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" \
+   pnpm dev
+   ```
+
+   Also start the workflow web UI in a separate terminal:
+   ```bash
+   npx workflow web
+   ```
+
+4. **Trigger the 7_full.ts workflow**:
+   ```bash
+   curl -X POST "http://localhost:3000/api/trigger?workflowFile=workflows/7_full.ts&workflowFn=handleUserSignup"
+   ```
+
+5. **View traces** in Jaeger UI at http://localhost:16686 - select service `example-nextjs-workflow`
+
+## Tracing Details
+
+The traces include:
+- Step execution spans with `workflow.queue.overhead_ms`, `step.attempt`, `step.status`
+- Workflow orchestration spans with `workflow.run.status`, `workflow.events.count`
+- Queue message spans with messaging attributes
+
+$ARGUMENTS can specify a different workbench (e.g., "example" or "nextjs-webpack").