Skip to content

CI red on all PRs since Aug 1: Node 26.5.1 (undici 8.8+) dispatch stall breaks "update state while run in flight" #2648

Description

@ostapondo

Every CI run since Aug 1 fails on Unit Tests (ubuntu-latest, 26.x) with the same single failure:

FAIL tests/api.test.mts > threads crud > update state while run in flight
AssertionError: expected 'error' to be 'interrupted'

This hits every branch, including dependabot PRs that only bump frontend deps. The last green run on main (Jul 29) used Node 26.5.0; runners now resolve 26.x to 26.5.1. Nothing in the repo changed.

Reproduction

On unmodified main, CI=1 npx vitest run tests/api.test.mts in libs/langgraph-api:

Node undici (bundled) Result
26.5.0 8.7.0 pass (2/2)
26.5.1 8.9.0 fail (3/3, same assertion)

The failure needs the full file's concurrent load; the test passes in isolation via -t.

Root cause

The regression is in undici, which jumped 8.7.0 → 8.9.0 inside the Node patch release. Cross-checking by overriding the SDK's fetch with standalone undici builds:

Node fetch from Result
26.5.0 undici@8.9.0 fail
26.5.0 undici@8.8.0 fail (2/2)
26.5.1 undici@8.7.0 pass

So the change landed in undici 8.8.0. A diagnostics_channel trace of the failing POST /threads/:id/state request shows where the time goes:

undici:request:create      +0ms
undici:client:sendHeaders  +1560ms   <- request sat in the dispatcher
undici:request:headers     +1563ms

The server handled the request in 1ms (hono log). Under the suite's load (67 concurrent tests sharing one client, several long-lived SSE joins) requests intermittently stall ~1.5–2.6s client-side before their headers are written, with delays quantizing around 500ms. A candidate is the idle-socket validation change in 8.8.0 (nodejs/undici#5499), but I haven't bisected undici commits and don't yet have a standalone repro to report upstream.

Why this test in particular

It schedules a run with afterSeconds: 2 and assumes create + 500ms sleep + update-state + cancel complete inside that window. The dispatch stall pushes the cancel past it: the queue picks the run up at the 2s mark, the cancel finds a running run instead of a pending one (Attempted to cancel non-pending run in the log), and the abort surfaces in the queue worker as an error, so the run ends "error".

Fixes

  • Widening the test's window to 5s makes the sequence fit even with slow dispatch, and costs nothing since the happy path cancels the run before it starts. PR: fix(api): widen the scheduled-run window in the in-flight cancel test #2649. Verified 3/3 green on 26.5.1 and green on 26.5.0.
  • Alternatively the workflow could pin node-version: 26.5.0, but that freezes the whole matrix over one test and needs a manual unpin later.

One thing worth a maintainer's opinion: cancelling a run that has already started always ends it as "error"ops.runs.cancel only sets "interrupted" while the run is still pending, and the queue worker classifies the cancel abort like any other failure. If "interrupted" is the intended status for a cancelled running run, I'm happy to follow up with that change separately.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions