perf(postgresql-transport): index dequeue path + robust idempotency#3278
Merged
Conversation
The PostgreSQL queue transport's dequeue (ORDER BY timestamp LIMIT n FOR UPDATE SKIP LOCKED) had no supporting index, so each poll scanned + sorted the queue table. - Add a btree index on the queue table's `timestamp` column so the ordered LIMIT is an index scan. Benchmarked: drain 69k -> 98k msg/s, deep-pop (batch 50 over 30k backlog) median 3.1ms -> 0.8ms. Additive, no migration risk. (Index name is run through PostgresqlIdentifier.Shorten for NAMEDATALEN, per GH-2942.) - Replace locale-fragile idempotency checks (e.Message.Contains("duplicate key value")) with the SQLSTATE unique-violation code (PostgresException.SqlState == 23505). Deliberately NOT adding a clustered-storage opt-in (the SQL Server transport's OptimizeQueueThroughput()): PostgreSQL tables are heaps with no clustered index, so a bigint-identity "seq" redesign benchmarked as marginal (deep-pop 0.8 -> 0.6ms, drain flat) — the timestamp index already captures the available benefit. Documented accordingly, including the autovacuum note for high-churn queues. Batched send (outbox -> queue via a batch) is deferred to a follow-up, matching the SQL Server transport work. Includes a manual A/B/C benchmark (skipped in CI). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jun 29, 2026
Open
Merged
This was referenced Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Postgres counterpart to #3277. The PostgreSQL queue transport's dequeue (
ORDER BY timestamp LIMIT n FOR UPDATE SKIP LOCKED) had no supporting index, so each poll scanned and sorted the queue table.Changes
timestampcolumn so the orderedLIMITdequeue is an index scan. No SQL changes (ordering stays ontimestamp); the index name goes throughPostgresqlIdentifier.Shortenfor NAMEDATALEN (PostgressqlQueue name size limit #2942).e.Message.Contains("duplicate key value")with the SQLSTATE unique-violation code (PostgresException.SqlState == 23505).Why no
OptimizeQueueThroughput()hereSQL Server got an opt-in clustered-storage layout (#3277) that gave a ~350× drain improvement. That does not translate to Postgres: PG tables are heaps with no clustered index, so the physical-clustering win doesn't exist. Benchmarked A/B/C to confirm before deciding:
timestampindex (this PR)bigintidentityseq(rejected)The
seqredesign is marginal, so adding the knob would mean a table migration for ~no gain and misleading cross-DB parity. The doc instead points at autovacuum tuning as the real operational lever for high-churn PG queues.Follow-up (not in this PR)
Batched outbox→queue send, matching the SQL Server transport plan.
Validation
wolverine.slnxRelease build clean.🤖 Generated with Claude Code