Skip to content

fix: scope pub/sub channels to their pool - #1285

Open
4thel00z wants to merge 1 commit into
pgdogdev:mainfrom
4thel00z:fix/scope-pub-sub-channels-per-database
Open

fix: scope pub/sub channels to their pool#1285
4thel00z wants to merge 1 commit into
pgdogdev:mainfrom
4thel00z:fix/scope-pub-sub-channels-per-database

Conversation

@4thel00z

Copy link
Copy Markdown

The bug

The pub/sub channel registry is a process-global map keyed by channel name alone, shared by every listener via CHANNELS.clone():

type Channels = Arc<Mutex<HashMap<String, Channel>>>;
static CHANNELS: Lazy<Channels> = ...;

Each PubSubListener owns a connection to one database, and PostgreSQL scopes NOTIFY to a database, so that key is missing a dimension.

It surfaces in listen(). When the name is already present, it returns a subscriber on the existing channel and returns early — without ever sending LISTEN to its own pool:

if let Some(channel) = guard.get(channel_name) {
    return Ok(Listener::new(channel));   // no LISTEN is sent
}

So the first pool to register a name owns the only real subscription. Later subscribers from other pools attach to it and receive that pool's notifications, while notifications raised in their own database reach nobody. listen_creates_channel_once_and_reuses_it already pins the no-second-LISTEN behaviour; it just never exercised it across pools.

The restart path has the mirror of the same bug. It re-subscribed to every key in the shared map, issuing LISTEN for channels owned by other pools — subscriptions that connection can never receive, and which channel teardown does not clean up, since the UNLISTEN runs only against the connection the notification arrived on.

The fix

Key the registry on the pool — database, user and shard, as configured in pgdog — and nest the map (pool -> channel -> state) so the scoping is structural rather than a filter that a future change can forget.

Two notes on the choice of identity:

  • It comes from Shard's identifier and number, not the server address. init_pub_sub rebuilds the listener against the new primary when a replica is promoted, which has a different host but the same pgdog-side identity. Keying on host/port would silence every pre-promotion subscriber.
  • user is included. Pools are per (user, database), so two users of the same database have separate pools and separate backend connections. Sharing one entry left whichever of them lost the race silently dependent on the other's subscription, and — once the other's task restarted and re-subscribed — produced duplicate delivery.

Resharding behaviour is deliberately unchanged: a channel re-maps to a different shard when the shard count changes, which this patch neither fixes nor worsens.

User-visible changes

SHOW LISTENERS gains database, user and shard columns, and the pub_sub_listeners / pub_sub_listener_received / pub_sub_listener_dropped metrics gain the matching labels. Both are required for uniqueness — the channel name is no longer a unique row or series identity, and without the labels two pools sharing a channel name would emit duplicate Prometheus series. Flagging it since anything parsing that admin output or scraping those series will see the new shape.

Testing

  • cargo fmt --all -- --check — clean
  • cargo clippy -p pgdog --all-targets -- -D warnings — clean
  • cargo check -p pgdog --all-targets — clean
  • Three unit tests added, driving two listeners over one shared registry and asserting each is sent its own LISTEN: across databases, across users, and across shards. Plus one covering the key's components.

To be straight about the limits: I was not able to execute the test suite locally (the full codegen build didn't finish in my environment), so the new tests are compiled and reviewed but not run — CI will be the first real execution. There is also no integration coverage, because integration/pub_sub/pgdog.toml declares a single [[databases]], so two databases sharing a channel name cannot occur there. Happy to extend that config if you'd like the fix covered end to end.

Left alone deliberately

listen() still does not roll back its registry entry if the subsequent Subscribe send fails — the entry then survives with no LISTEN behind it, and the early return hands every later caller a dead receiver. It predates this change, and removing the entry after the lock is released could strand a concurrent subscriber, so it wants its own fix rather than a bolt-on here.

The channel registry is a process-global map keyed by channel name alone,
shared by every listener via `CHANNELS.clone()`. Each listener owns a
connection to one database, and PostgreSQL scopes NOTIFY to a database, so
that key is missing a dimension.

The visible consequence is in `listen()`: when the name is already present it
returns a subscriber on the existing channel and never sends `LISTEN` to its
own pool. So the first pool to register a name owns the only real
subscription; later subscribers from other pools attach to it and receive
that pool's notifications, while notifications raised in their own database
reach nobody.

Key the registry on the pool instead: database, user and shard, as configured
in pgdog. Two databases using the same channel name now get their own channel
and their own `LISTEN`, and so do two users of the same database -- they have
separate pools and separate backend connections, so sharing one entry left
whichever of them lost the race silently dependent on the other.

The identity deliberately comes from `Shard`'s identifier and number rather
than the server address, so it survives a replica being promoted underneath
the listener: `init_pub_sub` rebuilds against the new primary, which has a
different host but the same pgdog-side identity.

Nesting the map (pool -> channel -> state) rather than flattening the key into
one struct keeps the scoping structural. The restart path had the mirror of
the same bug -- it re-subscribed to every key in the shared map, issuing
`LISTEN` for channels owned by other pools, which it can never receive and
which channel teardown does not reach, since the `UNLISTEN` only runs against
the connection the notification arrived on. It now iterates one pool's
channels by construction rather than filtering, and the receive path no longer
builds a key per notification.

`SHOW LISTENERS` and the `pub_sub_*` metrics gain database, user and shard:
the channel name is no longer a unique row or series identity.
Copilot AI review requested due to automatic review settings July 30, 2026 00:41
@CLAassistant

CLAassistant commented Jul 30, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a correctness bug in PgDog’s Postgres pub/sub listener where channel registry entries were process-global (keyed only by channel name), causing listeners on different pools (db/user/shard) to incorrectly share subscriptions and miss their own LISTENs. The PR scopes channel registry state to the pool identity so each pool maintains its own LISTEN lifecycle and stats, and updates observability surfaces accordingly.

Changes:

  • Scope the pub/sub channel registry by (database, user, shard) via PoolKey and return stats keyed by (pool, channel) rather than channel name alone.
  • Update SHOW LISTENERS output and listener metrics to include database, user, and shard dimensions.
  • Update shard pub/sub initialization to pass pool identity into PubSubListener::new, and add unit tests covering cross-pool non-sharing.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
pgdog/src/backend/pub_sub/listener.rs Introduces pool-scoped registry keys, adjusts subscribe/resubscribe/unsubscribe logic accordingly, and adds unit tests for cross-pool behavior.
pgdog/src/backend/pool/shard/mod.rs Updates pub/sub listener initialization to provide identifier + shard to the listener constructor.
pgdog/src/admin/show_listeners.rs Extends SHOW LISTENERS columns/rows to include pool identity (database/user/shard) for uniqueness.
pgdog/src/stats/listeners.rs Extends Prometheus metric labels with database/user/shard to match the new stats key shape.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 394 to 399
if let Some(unsub) = unsub {
channels.lock().remove(&unsub);
if let Some(channels) = channels.lock().get_mut(pool_key) {
channels.remove(&unsub);
}
server.send(&vec![Request::Unsubscribe(unsub).into()].into()).await?;
}
Comment on lines 96 to 106
/// Get stats for all channels.
pub fn stats() -> HashMap<String, StatsSnapshot> {
pub fn stats() -> HashMap<ChannelKey, StatsSnapshot> {
CHANNELS
.lock()
.iter()
.map(|(name, channel)| (name.to_string(), channel.stats.get()))
.flat_map(|(pool, channels)| {
channels
.iter()
.map(|(channel, state)| (pool.channel(channel), state.stats.get()))
})
.collect()
@codecov

codecov Bot commented Jul 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.18310% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pgdog/src/backend/pub_sub/listener.rs 96.66% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants