Skip to content

fix(local): buffered local queues no longer drop cascaded messages (GH-3287)#3322

Merged
jeremydmiller merged 1 commit into
mainfrom
fix/local-queue-message-drop-3287
Jul 6, 2026
Merged

fix(local): buffered local queues no longer drop cascaded messages (GH-3287)#3322
jeremydmiller merged 1 commit into
mainfrom
fix/local-queue-message-drop-3287

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Fixes #3287.

Problem

A handler that cascaded more than ~10,000 messages to a buffered (in-memory) local queue silently lost everything past the first ~10k. In the reporter's reproduction, Handler A raised 20,000 messages to Handler B and only ~10,000 arrived — no error, no log.

Root cause

The buffered receiver enqueues cascaded messages through the synchronous JasperFx.Blocks.Block<T>.Post, which wrote to a channel bounded at 10,000 using a non-blocking TryWrite and silently dropped each item once the channel filled (only a Debug.WriteLine). The producer never felt back pressure, so with a slow/limited consumer everything past the bound was lost.

Fix

Paired with JasperFx 2.19.1 (JasperFx/jasperfx#485), which makes Block<T>.Post back-pressure instead of dropping and adds an Unbounded channel option.

On the Wolverine side, a buffered local queue is routinely a handler's own cascade target — Handler A running on the queue enqueues thousands of messages back onto the same queue. Bounded back pressure there would deadlock: the single reader that must drain the queue is the very thread blocked producing. Local queues are in-process, so the buffered local queue now uses an unbounded channel — cascaded messages are never dropped and never deadlock, at any degree of parallelism. Broker-backed buffered receivers keep the bounded default so a fast broker can't flood memory.

  • BufferedReceiver gains a boundedCapacity ctor parameter (bounded default preserved for broker receivers).
  • BufferedLocalQueue passes Block<Envelope>.Unbounded.
  • ShardedExecutionBlock threads the capacity through to its per-slot blocks.
  • Bumps JasperFx to 2.19.1.

Tests

New SlowTests/dropped_messages_on_full_local_queue.cs reproduces the issue: Handler1 cascades 20,000 messages to a buffered local queue (throttled to a small worker count so the producer outruns the consumer), Handler2 counts them. Asserts all 20,000 arrive.

  • Runs at parallelism 5 (the reporter's config) and 1 — parallelism 1 is the harder case, where the single reader that runs Handler1 is the only thing that can drain the queue, so naive back pressure would deadlock. The unbounded queue handles both.
  • Before: 10,004 / 20,000. After: 20,000 / 20,000.

Verified against the published JasperFx 2.19.1; full wolverine.slnx builds clean in Release.

🤖 Generated with Claude Code

…H-3287)

A handler that cascaded more than ~10,000 messages to a buffered (in-memory)
local queue silently lost everything past the first ~10k. The buffered receiver
enqueues via the synchronous JasperFx Block<T>.Post, which used a non-blocking
TryWrite against a channel bounded at 10,000 and dropped items once full.

The JasperFx side (2.19.1) makes Post back-pressure instead of dropping and adds
an Unbounded channel option. On the Wolverine side a buffered local queue is
routinely a handler's own cascade target -- Handler A running on the queue
enqueues onto the same queue -- so bounded back pressure could deadlock the very
reader that drains it. Local queues are in-process, so the buffered local queue
now uses an unbounded channel: cascaded messages are never dropped and never
deadlock, at any degree of parallelism. Broker-backed buffered receivers keep the
bounded default so a fast broker can't flood memory.

- BufferedReceiver gains a boundedCapacity ctor parameter (bounded default).
- BufferedLocalQueue passes Block<Envelope>.Unbounded.
- ShardedExecutionBlock threads the capacity through to its per-slot blocks.
- Bumps JasperFx to 2.19.1.
- Adds a SlowTests reproduction that cascades 20k messages through a buffered
  local queue at parallelism 5 and 1 and asserts all 20k arrive.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jeremydmiller
jeremydmiller merged commit 55ac873 into main Jul 6, 2026
24 of 26 checks passed
This was referenced Jul 9, 2026
This was referenced Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wolverine drops messages when 10K+ messages are queued in-memory

1 participant