Skip to content

[BUG]: express.middleware spans retained unbounded via AsyncLocalStorage frame capture → heap OOM #9227

Closed
Closed
[BUG]: express.middleware spans retained unbounded via AsyncLocalStorage frame capture → heap OOM#9227
Labels
bugSomething isn't working

Description

@dwinrick-lever
dwinrick-lever
opened on Jul 6, 2026

Tracer Version(s)

5.110.0 (also confirmed present on master / v5.111.0)

Node.js Version(s)

24.16.0 (also reproduces on 18.20.8)

Bug Report

Under sustained traffic, an Express app instrumented by dd-trace accumulates express.middleware spans in the JS heap without bound, eventually crashing with FATAL ERROR: … JavaScript heap out of memory. In a production service the post-GC live heap climbed ~linearly to the --max-old-space-size ceiling and OOM'd roughly daily; ~93% of retained DatadogSpan objects were express.middleware.

Root cause (traced through the source): the active span is stored with AsyncLocalStorage.enterWith({ ...store, span }) (not run()) — packages/dd-trace/src/plugins/tracing.js:268-269, packages/dd-trace/src/plugins/plugin.js:105-107, and per-middleware at packages/datadog-plugin-router/src/index.js:41. Because the HTTP request event is emitted synchronously on the connection's async-context frame, this writes { ...store, span } into that frame. Any never-released async resource created during request handling (a retained AsyncResource, an event listener added to a long-lived emitter and never removed, a subscription, a cache entry, etc.) captures that frame (AsyncContextFrame / legacy kResourceStore) and pins the entire middleware span chain for the life of that resource — independent of the HTTP socket. The span is finished correctly, but the captured store snapshot still references it, so it is never collected. Real handlers create such resources constantly (DB clients, pub/sub subscriptions, timers, listeners), so retained spans grow ~N-per-request unbounded.

Retainer chain from a heap snapshot (taken after the load client closed its sockets + GC), proving it is not socket- or timer-anchored:

AsyncResource [Symbol(context_frame)] → AsyncContextFrame → <array> → Object{ ...store, span } → DatadogSpan → (parent chain, ~20-deep middleware)

Reproduction Code

Full, self-contained reproducer: https://github.com/dwinrick-lever/dd-trace-express-middleware-span-leak — npm install && ./reproduce.sh

Express + ~20 no-op middlewares + dd-trace.init(); sustained requests where each creates one never-destroyed async resource (stand-in for a real subscription/listener). Measured:

config retained per request outcome
dd-trace ON 36,730–36,748 B/req (constant every batch, dead-linear heap climb, no plateau) OOM
dd-trace OFF (same load) 90 B/req bounded/flat 10+ min
  • ~407× amplification by dd-trace. Constant bytes/request with no plateau ⇒ strictly unbounded.
  • Crashes at both --max-old-space-size=256 (~18 s) and 512 (~22 s): the larger cap crashes later but still crashes — a bounded working set would plateau below 512 MB and never crash there.
  • DD_TRACE_MIDDLEWARE_TRACING_ENABLED=false (or tracer.use('express', { middleware: false })) cuts retained spans ~96% and stops the growth.
  • Reproduces on Node 24 (AsyncContextFrame default) and Node 18 (legacy async_hooks); --no-async-context-frame does not help.

Error Logs

<--- Last few GCs --->
[1:0x...] Mark-Compact ... allocation failure; scavenge might not succeed
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Tracer Config

Default: require('dd-trace').init() with the express integration enabled; server uses HTTP keep-alive (Node default). No unusual config required to trigger it.
Workaround: DD_TRACE_MIDDLEWARE_TRACING_ENABLED=false or tracer.use('express', { middleware: false }).

Operating System

Linux (mechanism is not OS-specific).

Bundling

No Bundling

Activity

  1. dwinrick-lever
    added a commit that references this issue on Jul 6, 2026

    fix(core): release finished span from async context to prevent unboun…

    d391b22
  2. BridgeAR
    added
    bugSomething isn't working
    on Jul 7, 2026
  3. BridgeAR
    added 8 commits that reference this issue on Jul 7, 2026

    fix(core): release finished spans from captured async-context frames

    aaeb9a8

    test(plugins): detect finished-span retention at agent teardown

    1baba8f

    test(plugins): let suites raise the span-retention baseline

    620ccec

    test: tolerate upstream pending-timer span retention per suite

    4502a90

    test: release HTTP keep-alive span retainers, raise baseline to 10

    b2d84bc

    fix(core): retire finished spans without losing context

    f4dc7bd

    fix(core): retire finished spans without losing context

    5e90183

    fix(core): retire finished spans without losing context

    ddc03de

  4. bvizy-reshop commented on Jul 22, 2026

    @bvizy-reshop
    bvizy-reshop
    on Jul 22, 2026

    @BridgeAR Any update when a fix can be released? The underlying issue is creating a memory leak in our application as well

  5. BridgeAR commented on Jul 22, 2026

    @BridgeAR
    BridgeAR
    on Jul 22, 2026
    Member

    The tracer kind of behaves correct is the problem.

    It is a very tricky situation where different requirements mean we have to still have access to older memory in some situations.

    It is on the radar and I will try to focus on this next week.

  6. julwhitney13 commented on Jul 23, 2026

    @julwhitney13
    julwhitney13
    on Jul 23, 2026

    @BridgeAR thank you for looking into and working on this!! more context if its at all helpful:

    Confirming the same retention pattern in Next.js (App Router, standalone) where the documented workaround doesn't apply

    We believe we hit the same root cause in a production Next.js 16 App Router app (output: "standalone", cacheComponents: true) on dd-trace 5.113.0 i.e., the enterWith frame-capture retention described here isn't limited to express.middleware spans; the path in packages/dd-trace/src/plugins/tracing.js / plugin.js is generic to tracing plugins.

    What we observed

    • Tracing enabled: monotonic heap growth of ~23-54 MiB/h under steady production traffic, ending in a V8 heap OOM (FATAL ERROR: Ineffective mark-compacts near heap limit) at ~30h uptime, with a burst of user-facing 502s at each crash.
    • DD_TRACE_ENABLED=false, same traffic: ~7-10 MiB/h with a healthy GC sawtooth, no crashes.

    How we confirmed
    Env-var differential only, no code changes: flipped DD_TRACE_ENABLED off in staging, reproduced the flat profile, then rolled the same toggle through production. Heap growth tracked the tracer, not the app.

    Two Next.js-specific notes

    1. The next plugin's bindStart returns { ...store, span, req }, so every captured async-context frame additionally pins the IncomingMessage (and its buffers/headers), which plausibly makes per-request retained size worse than the express numbers in the issue body. (Related but distinct from Memory leak in Next.js plugin: WeakMap in #requestsBySpanId retains IncomingMessage objects #7876, which was the #requestsBySpanId WeakMap path.)
    2. The validated mitigations here (DD_TRACE_MIDDLEWARE_TRACING_ENABLED=false / middleware: false) only suppress express middleware spans. Next.js standalone users have no equivalent short of disabling tracing entirely.

    it would be great if the fix in #9292 and its tests also covered the next plugin's store shape (if not already planning to), not just express middleware chains. also always happy to help test a prerelease against our workload!

  7. bvizy-reshop commented on Aug 27, 2026

    @bvizy-reshop
    bvizy-reshop
    on Aug 27, 2026

    Hi @BridgeAR It's been a few weeks, and it seems like the PR stalled. Any chance we can prioritize a fix?

  8. BridgeAR commented on Sep 2, 2026

    @BridgeAR
    BridgeAR
    on Sep 2, 2026
    Member

    After looking further into it, it is likely related to using undici.

    Could you please deactivate it with tracer.use('undici', false) (programmatic) or DD_TRACE_UNDICI_ENABLED=false (environment variable). That should resolve the problem for now.

    The undici fix should land in the next release. I want to apologize that this took so long!

  9. BridgeAR commented on Sep 3, 2026

    @BridgeAR
    BridgeAR
    on Sep 3, 2026
    Member

    I opened a PR to fix the root issue nodejs/undici#5761

  10. BridgeAR commented on Sep 3, 2026

    @BridgeAR
    BridgeAR
    on Sep 3, 2026
    Member

    We landed a fix in dd-trace-js as well (a kind of bug workaround for the undici bug) and it will be released in the next version.

  11. BridgeAR commented on Sep 8, 2026

    @BridgeAR
    BridgeAR
    on Sep 8, 2026
    Member

    Now that the fix landed in undici and we also landed a fix in dd-trace that the issue does not surface anymore, this should be resolved.

    If you are still seeing any of the former behavior, please let us know right away, thank you!

  12. BridgeAR
    closed this as completedon Sep 8, 2026
  13. BridgeAR
    added 5 commits that reference this issue on Sep 10, 2026

    fix(core): release finished spans from captured async-context frames

    7511a80

    test(plugins): detect finished-span retention at agent teardown

    f8fb9d5

    test(plugins): let suites raise the span-retention baseline

    46df544

    test: tolerate upstream pending-timer span retention per suite

    c0e6847

    test: release HTTP keep-alive span retainers, raise baseline to 10

    44e56bd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions