perf(dlq): index wolverine_dead_letters for replay/cleanup scans (GH-3279)

[jeremydmiller]

Closes #3279.

Problem

The durability agent's background DLQ replay (INSERT ... SELECT ... FROM wolverine_dead_letters WHERE replayable = $1) and cleanup DELETE ... WHERE replayable = $1 both filter on replayable, and the expiration sweep (DeleteExpiredDeadLetterMessagesOperation) filters on expires. With no index on those columns, each durability cycle forces a full sequential scan of the entire wolverine_dead_letters table.

The reporter (a DBA) observed these as the top-2 SQL statements by Average Active Sessions in AWS Performance Insights during an ~8M-event Marten projection rebuild — a 9+ GB dead-letter table scanned in full (Buffers: shared hit=519487 read=627396, ~1.9 s, I/O-wait dominated) on every replay cycle.

Fix

DeadLettersTable now provisions the indexes automatically as part of normal schema creation/migration — no configuration, no opt-in. The replayable index is always added (that column always exists); the expires index is added only when DeadLetterQueueExpirationEnabled is on (that's when the column exists).

Backend	Index
PostgreSQL	partial index WHERE replayable, partial WHERE expires is not null
SQL Server	filtered index WHERE ([replayable]=1), filtered WHERE ([expires] IS NOT NULL)
SQLite	partial index WHERE replayable = 1, partial WHERE expires is not null
MySQL	plain index on replayable / expires (no filtered-index support)
Oracle	plain index on replayable / expires (WHERE-clause partial indexes are 23c+ only)

Partial/filtered indexes stay tiny because replayable rows are deleted from the DLQ immediately after being moved back to the inbox.

Predicate round-trip (why the exact predicate strings matter)

A filtered-index predicate that doesn't round-trip through the DB's system catalogs makes Weasel see a perpetual "index changed" delta and drop + recreate the index on every startup. SQL Server stores filter definitions in a canonical bracketed form (([replayable]=(1))), so the configured predicate uses [replayable]=1 to match after canonicalization; Postgres uses the boolean / is not null forms. New idempotency tests create the schema and assert FindDeltaAsync().Difference == SchemaPatchDifference.None for Postgres, SQL Server, and SQLite (the three backends that emit predicates). The SQL Server test caught exactly this drift before the predicate was corrected.

Gating

Automatic, no flag. The partial index is negligible in size/write cost, both hot queries filter on the column, and the deployments that need it most are precisely the large-DLQ ones that would never think to flip an opt-in. Documented a caveat for operators upgrading with an already-huge DLQ table: the migration's CREATE INDEX takes a write-blocking lock, so they may want to pre-create the index with CREATE INDEX CONCURRENTLY / WITH (ONLINE = ON).

Verification

• New idempotency tests pass on PostgreSQL, SQL Server, and SQLite (indexes created and stable/no-drift).
• Existing Postgres DLQ replay/expiration tests still green (no regression to the startup migration path).
• Full wolverine.slnx builds clean in Release (0 warnings, 0 errors).

🤖 Generated with Claude Code