GH-3111: contain fatal recovery-path failures so a poison message can't crash the host

[jeremydmiller]

Hardens #3111. Scope is deliberately narrow per the issue: not fixing the unbounded allocation itself, just keeping a single un-recoverable envelope from taking the whole host down.

The trap (traced against main)

HandlerPipeline.InvokeAsync's last-resort catch (Exception e) does allocating recovery — channel.CompleteAsync(envelope) then Logger.LogException(...) — with no protection:

catch (Exception e)
{
    await channel.CompleteAsync(envelope).ConfigureAwait(false); // allocates
    Logger.LogException(e, envelope.Id);                          // allocates
    activity?.SetStatus(...);
}

When the original failure is resource exhaustion (an OutOfMemoryException from an unbounded deserialization — the field report was a Brotli decompression bomb, JasperFx/ProductSupport#11), that recovery re-allocates at the memory ceiling and re-throws. The re-thrown OOM escapes InvokeAsync, faults the receiver loop, stops the listener, and the host exits cleanly (code 0). Kubernetes reads exit 0 as a clean shutdown and restarts straight into the same un-acked poison message → permanent CrashLoopBackOff from one bad message. (This is not the circuit breaker — that's opt-in and needs 10 failures; this fires on a single message.)

The fix (issue direction #2 — "guard the last-resort catch")

Move the recovery into a guarded, allocation-safe helper. The normal path is byte-for-byte the same ack + log; the only behavioral change is that a failure of the recovery work itself is now contained instead of escaping the pipeline:

• If CompleteAsync / LogException throw (typically a re-thrown OOM), the helper swallows it so it can never fault the receiver loop and silently stop the host.
• The contained log is itself fully guarded (logging allocates too) and reports the fatal-vs-cascading nature distinctly.
• The host stays alive and keeps processing other messages. The still-un-acked envelope is left for the broker to redeliver / dead-letter via its own max-receive policy.

Mirrors the existing OOM-as-fatal precedent in EncryptingMessageSerializer (is not OutOfMemoryException).

Tests

handler_pipeline_failure_containment pins the guarantee — RecoverFromFailedProcessingAsync never throws when:

• CompleteAsync throws OutOfMemoryException,
• exception logging throws OutOfMemoryException,
• even the contained containment-log itself throws OutOfMemoryException,

and the happy path still acks + logs exactly as before.

99 pipeline-adjacent CoreTests pass; full wolverine.slnx Release build clean.

Deliberately out of scope

The issue's broader directions — a non-zero process exit on fatal conditions, a poison-counter, and a generic max-inbound-deserialized-size guard — are larger behavioral/policy changes left for a follow-up. This PR is the minimal "keep the host alive" hardening.

🤖 Generated with Claude Code