Skip to content

perf(postgresql-transport): index dequeue path + robust idempotency#3278

Merged
jeremydmiller merged 1 commit into
mainfrom
perf/postgresql-transport
Jun 29, 2026
Merged

perf(postgresql-transport): index dequeue path + robust idempotency#3278
jeremydmiller merged 1 commit into
mainfrom
perf/postgresql-transport

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Summary

The Postgres counterpart to #3277. The PostgreSQL queue transport's dequeue (ORDER BY timestamp LIMIT n FOR UPDATE SKIP LOCKED) had no supporting index, so each poll scanned and sorted the queue table.

Changes

  • Index the queue table timestamp column so the ordered LIMIT dequeue is an index scan. No SQL changes (ordering stays on timestamp); the index name goes through PostgresqlIdentifier.Shorten for NAMEDATALEN (PostgressqlQueue name size limit #2942).
  • Robust idempotency: replaced locale-fragile e.Message.Contains("duplicate key value") with the SQLSTATE unique-violation code (PostgresException.SqlState == 23505).
  • Docs: new "Dequeue Performance" note in the PostgreSQL guide.

Why no OptimizeQueueThroughput() here

SQL Server got an opt-in clustered-storage layout (#3277) that gave a ~350× drain improvement. That does not translate to Postgres: PG tables are heaps with no clustered index, so the physical-clustering win doesn't exist. Benchmarked A/B/C to confirm before deciding:

Variant drain deep-pop median (30k backlog) p95
baseline (uuid PK, no index) 69,634/s 3.1 ms 3.9 ms
+ timestamp index (this PR) 98,401/s 0.8 ms 1.0 ms
+ bigint identity seq (rejected) 97,308/s 0.6 ms 0.8 ms

The seq redesign is marginal, so adding the knob would mean a table migration for ~no gain and misleading cross-DB parity. The doc instead points at autovacuum tuning as the real operational lever for high-churn PG queues.

Follow-up (not in this PR)

Batched outbox→queue send, matching the SQL Server transport plan.

Validation

  • ✅ Full wolverine.slnx Release build clean.
  • 24 PG transport tests pass locally against PostgreSQL 17 (the Postgres dev environment is reliable, so unlike perf(sqlserver-transport): index dequeue path + opt-in clustered queue layout #3277 the integration suite — including schema provisioning with the new index and idempotent double-send — was validated locally).
  • A manual A/B/C benchmark is included (skipped in CI).

🤖 Generated with Claude Code

The PostgreSQL queue transport's dequeue (ORDER BY timestamp LIMIT n FOR UPDATE
SKIP LOCKED) had no supporting index, so each poll scanned + sorted the queue
table.

- Add a btree index on the queue table's `timestamp` column so the ordered LIMIT
  is an index scan. Benchmarked: drain 69k -> 98k msg/s, deep-pop (batch 50 over
  30k backlog) median 3.1ms -> 0.8ms. Additive, no migration risk. (Index name is
  run through PostgresqlIdentifier.Shorten for NAMEDATALEN, per GH-2942.)
- Replace locale-fragile idempotency checks (e.Message.Contains("duplicate key
  value")) with the SQLSTATE unique-violation code (PostgresException.SqlState ==
  23505).

Deliberately NOT adding a clustered-storage opt-in (the SQL Server transport's
OptimizeQueueThroughput()): PostgreSQL tables are heaps with no clustered index,
so a bigint-identity "seq" redesign benchmarked as marginal (deep-pop 0.8 ->
0.6ms, drain flat) — the timestamp index already captures the available benefit.
Documented accordingly, including the autovacuum note for high-churn queues.

Batched send (outbox -> queue via a batch) is deferred to a follow-up, matching
the SQL Server transport work.

Includes a manual A/B/C benchmark (skipped in CI).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jeremydmiller
jeremydmiller merged commit 6638212 into main Jun 29, 2026
26 checks passed
This was referenced Jun 29, 2026
This was referenced Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant