fix: scope pub/sub channels to their pool - #1285
Conversation
The channel registry is a process-global map keyed by channel name alone, shared by every listener via `CHANNELS.clone()`. Each listener owns a connection to one database, and PostgreSQL scopes NOTIFY to a database, so that key is missing a dimension. The visible consequence is in `listen()`: when the name is already present it returns a subscriber on the existing channel and never sends `LISTEN` to its own pool. So the first pool to register a name owns the only real subscription; later subscribers from other pools attach to it and receive that pool's notifications, while notifications raised in their own database reach nobody. Key the registry on the pool instead: database, user and shard, as configured in pgdog. Two databases using the same channel name now get their own channel and their own `LISTEN`, and so do two users of the same database -- they have separate pools and separate backend connections, so sharing one entry left whichever of them lost the race silently dependent on the other. The identity deliberately comes from `Shard`'s identifier and number rather than the server address, so it survives a replica being promoted underneath the listener: `init_pub_sub` rebuilds against the new primary, which has a different host but the same pgdog-side identity. Nesting the map (pool -> channel -> state) rather than flattening the key into one struct keeps the scoping structural. The restart path had the mirror of the same bug -- it re-subscribed to every key in the shared map, issuing `LISTEN` for channels owned by other pools, which it can never receive and which channel teardown does not reach, since the `UNLISTEN` only runs against the connection the notification arrived on. It now iterates one pool's channels by construction rather than filtering, and the receive path no longer builds a key per notification. `SHOW LISTENERS` and the `pub_sub_*` metrics gain database, user and shard: the channel name is no longer a unique row or series identity.
There was a problem hiding this comment.
Pull request overview
Fixes a correctness bug in PgDog’s Postgres pub/sub listener where channel registry entries were process-global (keyed only by channel name), causing listeners on different pools (db/user/shard) to incorrectly share subscriptions and miss their own LISTENs. The PR scopes channel registry state to the pool identity so each pool maintains its own LISTEN lifecycle and stats, and updates observability surfaces accordingly.
Changes:
- Scope the pub/sub channel registry by
(database, user, shard)viaPoolKeyand return stats keyed by(pool, channel)rather than channel name alone. - Update
SHOW LISTENERSoutput and listener metrics to includedatabase,user, andsharddimensions. - Update shard pub/sub initialization to pass pool identity into
PubSubListener::new, and add unit tests covering cross-pool non-sharing.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| pgdog/src/backend/pub_sub/listener.rs | Introduces pool-scoped registry keys, adjusts subscribe/resubscribe/unsubscribe logic accordingly, and adds unit tests for cross-pool behavior. |
| pgdog/src/backend/pool/shard/mod.rs | Updates pub/sub listener initialization to provide identifier + shard to the listener constructor. |
| pgdog/src/admin/show_listeners.rs | Extends SHOW LISTENERS columns/rows to include pool identity (database/user/shard) for uniqueness. |
| pgdog/src/stats/listeners.rs | Extends Prometheus metric labels with database/user/shard to match the new stats key shape. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if let Some(unsub) = unsub { | ||
| channels.lock().remove(&unsub); | ||
| if let Some(channels) = channels.lock().get_mut(pool_key) { | ||
| channels.remove(&unsub); | ||
| } | ||
| server.send(&vec![Request::Unsubscribe(unsub).into()].into()).await?; | ||
| } |
| /// Get stats for all channels. | ||
| pub fn stats() -> HashMap<String, StatsSnapshot> { | ||
| pub fn stats() -> HashMap<ChannelKey, StatsSnapshot> { | ||
| CHANNELS | ||
| .lock() | ||
| .iter() | ||
| .map(|(name, channel)| (name.to_string(), channel.stats.get())) | ||
| .flat_map(|(pool, channels)| { | ||
| channels | ||
| .iter() | ||
| .map(|(channel, state)| (pool.channel(channel), state.stats.get())) | ||
| }) | ||
| .collect() |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
The bug
The pub/sub channel registry is a process-global map keyed by channel name alone, shared by every listener via
CHANNELS.clone():Each
PubSubListenerowns a connection to one database, and PostgreSQL scopesNOTIFYto a database, so that key is missing a dimension.It surfaces in
listen(). When the name is already present, it returns a subscriber on the existing channel and returns early — without ever sendingLISTENto its own pool:So the first pool to register a name owns the only real subscription. Later subscribers from other pools attach to it and receive that pool's notifications, while notifications raised in their own database reach nobody.
listen_creates_channel_once_and_reuses_italready pins the no-second-LISTENbehaviour; it just never exercised it across pools.The restart path has the mirror of the same bug. It re-subscribed to every key in the shared map, issuing
LISTENfor channels owned by other pools — subscriptions that connection can never receive, and which channel teardown does not clean up, since theUNLISTENruns only against the connection the notification arrived on.The fix
Key the registry on the pool — database, user and shard, as configured in pgdog — and nest the map (
pool -> channel -> state) so the scoping is structural rather than a filter that a future change can forget.Two notes on the choice of identity:
Shard's identifier and number, not the server address.init_pub_subrebuilds the listener against the new primary when a replica is promoted, which has a different host but the same pgdog-side identity. Keying on host/port would silence every pre-promotion subscriber.useris included. Pools are per(user, database), so two users of the same database have separate pools and separate backend connections. Sharing one entry left whichever of them lost the race silently dependent on the other's subscription, and — once the other's task restarted and re-subscribed — produced duplicate delivery.Resharding behaviour is deliberately unchanged: a channel re-maps to a different shard when the shard count changes, which this patch neither fixes nor worsens.
User-visible changes
SHOW LISTENERSgainsdatabase,userandshardcolumns, and thepub_sub_listeners/pub_sub_listener_received/pub_sub_listener_droppedmetrics gain the matching labels. Both are required for uniqueness — the channel name is no longer a unique row or series identity, and without the labels two pools sharing a channel name would emit duplicate Prometheus series. Flagging it since anything parsing that admin output or scraping those series will see the new shape.Testing
cargo fmt --all -- --check— cleancargo clippy -p pgdog --all-targets -- -D warnings— cleancargo check -p pgdog --all-targets— cleanLISTEN: across databases, across users, and across shards. Plus one covering the key's components.To be straight about the limits: I was not able to execute the test suite locally (the full codegen build didn't finish in my environment), so the new tests are compiled and reviewed but not run — CI will be the first real execution. There is also no integration coverage, because
integration/pub_sub/pgdog.tomldeclares a single[[databases]], so two databases sharing a channel name cannot occur there. Happy to extend that config if you'd like the fix covered end to end.Left alone deliberately
listen()still does not roll back its registry entry if the subsequentSubscribesend fails — the entry then survives with noLISTENbehind it, and the early return hands every later caller a dead receiver. It predates this change, and removing the entry after the lock is released could strand a concurrent subscriber, so it wants its own fix rather than a bolt-on here.