feat(batching): ApplyItemException to isolate poison items from a batch (GH-3289 Phase 3a)

[jeremydmiller]

Phase 3a of the #3289 batch-processing plan: item 8 (ApplyItemException, "build FIRST") plus the shared reduced-batch re-execution primitive that items 2 & 3 will build on. (Phases 0/1 merged as #3299/#3300; Phase 2 is #3301.)

I split the plan's Phase 3 into two PRs for reviewability — this one is item 8 + the primitive; ProbeIndividuallyAfter (item 2) and IsolateBatchMembers (item 3) follow in a second PR that reuses the primitive.

The problem

Today a failed batch is retried and dead-lettered as a unit — one poison message takes every other message in the batch to the DLQ with it (MessageContext.MoveToDeadLetterQueueAsync fans over Envelope.Batch). When the handler already knows which item is bad, that's needless collateral.

The feature

A batch handler throws ApplyItemException naming the poison item(s) plus an explicit disposition for the survivors. Static factories only:

throw ApplyItemException.DeadLetterAndReplayOthers(badOrder);            // DLQ it, re-run the rest
throw ApplyItemException.DeadLetterAndAckOthers(bad1, bad2);             // DLQ them, ack the rest as-is
throw ApplyItemException.DeadLetter(poison: bads, ackItems: committed);  // DLQ bads, ack what I committed, replay the remainder

Resolution is a built-in FailureRule keyed on the exception type, registered in the WolverineOptions constructor as an every-attempt ContinueWith (mirrors the existing OnException<DuplicateIncomingEnvelopeException>().Discard() / rate-limit precedents). Throwing the exception is the opt-in — no config.

ApplyItemContinuation partitions Envelope.Batch by reference identity into poison / ack / replay:

• poison → new internal MessageContext.MoveBatchMembersToDeadLetterQueueAsync — dead-letters just those members (native or durable path) without completing the batch;
• replay → BatchReplay.EnqueueReducedBatchAsync — the shared primitive: rebuilds a fresh, reduced typed T[] batch (fresh member envelopes carrying the items by reference so a subsequent ApplyItemException still maps) and enqueues it onto the batch's own local queue (runtime.Endpoints.AgentForLocalQueue(batch.Destination));
• then CompleteAsync() settles every original member + the batch.

Tests (DB-free)

src/Testing/CoreTests/Acceptance/batch_item_isolation.cs:

• ApplyItemException factory + empty/null-poison guard unit tests;
• end-to-end isolation for each disposition — asserting the replayed survivor set and that only the poison item was dead-lettered. The DLQ assertion is captured via a registered IDeadLetterInterceptor (its BeforeStoreAsync runs even under NullMessageStore, so no database is needed).

All 8 green; existing batch acceptance tests unaffected. dotnet build wolverine.slnx -c Release clean.

Scope note

Items are matched by reference identity, which is exact for a normal batch. Under CoalesceBy the handler sees the last-wins instance for a key (the Message of only one member), so today only that member is dead-lettered and the earlier same-key members fall into the survivor set. Poisoning every member that collapsed into a coalesced key is a deliberate follow-up (called out in a code comment and the docs), handled in the items 2/3 PR.

🤖 Generated with Claude Code