fix(local): buffered local queues no longer drop cascaded messages (GH-3287)#3322
Merged
Merged
Conversation
…H-3287) A handler that cascaded more than ~10,000 messages to a buffered (in-memory) local queue silently lost everything past the first ~10k. The buffered receiver enqueues via the synchronous JasperFx Block<T>.Post, which used a non-blocking TryWrite against a channel bounded at 10,000 and dropped items once full. The JasperFx side (2.19.1) makes Post back-pressure instead of dropping and adds an Unbounded channel option. On the Wolverine side a buffered local queue is routinely a handler's own cascade target -- Handler A running on the queue enqueues onto the same queue -- so bounded back pressure could deadlock the very reader that drains it. Local queues are in-process, so the buffered local queue now uses an unbounded channel: cascaded messages are never dropped and never deadlock, at any degree of parallelism. Broker-backed buffered receivers keep the bounded default so a fast broker can't flood memory. - BufferedReceiver gains a boundedCapacity ctor parameter (bounded default). - BufferedLocalQueue passes Block<Envelope>.Unbounded. - ShardedExecutionBlock threads the capacity through to its per-slot blocks. - Bumps JasperFx to 2.19.1. - Adds a SlowTests reproduction that cascades 20k messages through a buffered local queue at parallelism 5 and 1 and asserts all 20k arrive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 9, 2026
Merged
This was referenced Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3287.
Problem
A handler that cascaded more than ~10,000 messages to a buffered (in-memory) local queue silently lost everything past the first ~10k. In the reporter's reproduction, Handler A raised 20,000 messages to Handler B and only ~10,000 arrived — no error, no log.
Root cause
The buffered receiver enqueues cascaded messages through the synchronous
JasperFx.Blocks.Block<T>.Post, which wrote to a channel bounded at 10,000 using a non-blockingTryWriteand silently dropped each item once the channel filled (only aDebug.WriteLine). The producer never felt back pressure, so with a slow/limited consumer everything past the bound was lost.Fix
Paired with JasperFx 2.19.1 (JasperFx/jasperfx#485), which makes
Block<T>.Postback-pressure instead of dropping and adds anUnboundedchannel option.On the Wolverine side, a buffered local queue is routinely a handler's own cascade target — Handler A running on the queue enqueues thousands of messages back onto the same queue. Bounded back pressure there would deadlock: the single reader that must drain the queue is the very thread blocked producing. Local queues are in-process, so the buffered local queue now uses an unbounded channel — cascaded messages are never dropped and never deadlock, at any degree of parallelism. Broker-backed buffered receivers keep the bounded default so a fast broker can't flood memory.
BufferedReceivergains aboundedCapacityctor parameter (bounded default preserved for broker receivers).BufferedLocalQueuepassesBlock<Envelope>.Unbounded.ShardedExecutionBlockthreads the capacity through to its per-slot blocks.Tests
New
SlowTests/dropped_messages_on_full_local_queue.csreproduces the issue: Handler1 cascades 20,000 messages to a buffered local queue (throttled to a small worker count so the producer outruns the consumer), Handler2 counts them. Asserts all 20,000 arrive.Verified against the published JasperFx 2.19.1; full
wolverine.slnxbuilds clean in Release.🤖 Generated with Claude Code