Skip to content

fix(pool): eliminate abandon monitor race for blocking commands (#6971) - #7063

Draft
affonsov wants to merge 6 commits into
mainfrom
fix/java-pool-abandon-monitor-blocking-race
Draft

affonsov wants to merge 6 commits into
mainfrom
fix/java-pool-abandon-monitor-blocking-race

Conversation

@affonsov

@affonsov affonsov commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR fixes a race condition in the ClientPool abandon monitor (#6971) and migrates the Node.js pool to use glide-core::ClientPool directly (#6887), removing the TypeScript-side state duplication that existed since the pool was first introduced.

Bug (#6971): The abandon monitor could silently reclaim a pooled client executing a blocking command (BLPOP, XREAD BLOCK, etc.), causing the in-flight future to fail with a ClosingException. Root cause: mark_client_blocking() used try_lock() on the pools TokioMutex`. Under lock contention the flag was never set, and the monitor treated the client as idle-abandoned.

Refactor (#6887): Since the pool was introduced in #6338, Nodes ClientPoolmaintained its own TypeScript-side idle stack, active map, waiter queue, and abandon monitor — duplicating state thatglide-core::ClientPoolalready tracks for Java, Go, and Python. This PR removes that duplication by wiringClientPool.ts` to delegate all pool state to Rust.

Issue link

Closes #6971 — ClientPool abandon monitor can reclaim a client parked on a blocking command
Closes #6887 — Node: Refactor pool state management into Rust core

Features / Behaviour Changes

  • The ClientPool abandon monitor now correctly skips clients executing blocking commands under all conditions, including high pool mutex contention, for Java, Go, Python, and Node.
  • mark_client_blocking() (which used try_lock) is removed. All bindings use the lock-free get_blocking_flag(client_id) API from glide-core.
  • Node.js ClientPool.ts now delegates all pool state to glide-core::ClientPool via the standard Rust pool APIs (createPool, poolTryAcquire, poolAcquireBlocking, poolRelease, poolMetrics, poolDestroy). The TypeScript-side idle stack, active map, waiter queue, and abandon monitor are removed.
  • create_handle_for_client() extracted from create_direct_client() in node/rust-client/src/lib.rs. Pool-managed Rust connections are wrapped as full GlideClientHandle N-API objects at acquire time, preserving the complete command API surface for TypeScript callers.
  • The fix is available to all future language clients (C#, PHP, Ruby) — calling glide_core::pool::get_blocking_flag(client_id) in the dispatch path is sufficient.
  • Public API of ClientPool is unchanged.

Implementation

Root cause and fix (glide-core/src/pool.rs)

mark_client_blocking() acquired the pools TokioMutexviatry_lock(). If the lock was held at the moment a blocking command was dispatched, try_lock()returnedErrand theis_blocking flag was silently never set. The abandon monitors next scan saw is_blocking = false and discarded the client.

Fix: a BLOCKING_FLAG_REGISTRY: DashMap<u64, Arc<AtomicBool>> is added to glide-core/src/pool.rs. The Arc is created when a client enters the pool (add_client) and is shared between PooledClient.is_blocking (read by the monitor) and the registry (read by bindings via get_blocking_flag(client_id)). Each binding calls arc.store(true, Ordering::Release) synchronously on the calling thread, before any spawn() or channel send. An UnmarkOnDrop RAII guard inside the async task clears the flag on all exit paths.

Node pool migration (node/rust-client/src/pool.rs, node/src/ClientPool.ts)

create_handle_for_client(client, push_rx, wake_tsfn, inflight_limit, provided_client_id) is extracted from create_direct_client(). It accepts a provided_client_id: Option<u64> — when Some(id) is given (pool path), the handles client_id matches the pools tracking key; when None (standalone path), a new ID is allocated from NEXT_CLIENT_ID. This ensures poolRelease(pool_id, client.getClientId()) finds the correct in_use entry.

pool_build_handle(client_id, wake_callback) retrieves the existing Rust Client from the glide-core scope registry and wraps it as a full GlideClientHandle with a fresh worker thread and response buffer — no new connection is opened.

close_for_pool_release() stops the worker thread without unregistering from the scope registry, so the same underlying Rust connection can be reused across pool borrow cycles.

On-demand connection creation is triggered in both pool_try_acquire and pool_acquire_blocking via a shared maybe_spawn_on_demand_creation() helper, preserving the maxSize pool growth behaviour.

Key attention areas for reviewers

  • glide-core/src/pool.rsBLOCKING_FLAG_REGISTRY and CLIENT_TO_POOL lifecycle (add on creation, reset-to-false on return-to-idle, remove on permanent discard and destroy)
  • java/src/lib.rspre_blocking_arc set before get_runtime().spawn(); UnmarkOnDrop guard handles all exit paths including routing errors via ? and task cancellation
  • ffi/src/lib.rs — same pre-spawn pattern; _arc_guard for single-command path
  • node/rust-client/src/lib.rscreate_handle_for_client extraction; close_for_pool_release
  • node/rust-client/src/pool.rsmaybe_spawn_on_demand_creation; pool_build_handle; on-demand creation in both acquire paths; discard draining
  • node/src/ClientPool.ts — all TS-side state removed; delegates to Rust pool APIs

Limitations

  • Go/Python fix has no dedicated integration test; the two new glide-core unit tests (abandon_monitor_skips_blocking_clients, abandon_monitor_evicts_non_blocking_abandoned_clients) cover the core invariant without a Valkey server.
  • pool_acquire_blocking uses a 5 ms polling loop rather than a condvar wait (pre-existing limitation, not introduced by this PR).
  • All batch/script executions are conservatively marked as blocking for the full dispatch duration (pre-existing behavior, consistent with prior semantics).
  • Node pool integration tests are not run as part of this PR`s CI (Node integration tests require a full native addon build); existing unit tests and the TypeScript build are verified clean.

Testing

New unit tests added (glide-core/src/pool.rs — no Valkey server required):

Verified:

  • 190/190 glide-core unit tests pass
  • Java pool integration tests pass
  • cargo clippy clean across all 4 crates (glide-core, java/glide-rs, ffi, node/rust-client)
  • Java spotless lint clean
  • Node TypeScript build (npm run build:ts) clean
  • CHANGELOG.md updated

Checklist

  • This Pull Request is related to one issue.
  • Commit message has a detailed description of what changed and why.
  • Tests are added or updated.
  • CHANGELOG.md and documentation files are updated.
  • Linters have been run and Prettier has been run.
  • Destination branch is correct — main.
  • Create merge commit if merging release branch into main, squash otherwise.
  • Make sure to update the documentation in the valkey-glide-docs repository if necessary.

@valkey-review-bot valkey-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lock-free flag removes the original pool-mutex race, but the new registry/lifecycle code introduces correctness problems in Node and on overlapping dispatches.

Comment thread node/rust-client/src/pool.rs Outdated
/// Starts at a high offset to avoid collision with glide-core managed pool IDs
/// (which start at 1 and increment monotonically from POOL_REGISTRY::register_pool).
/// The two namespaces are disjoint as long as fewer than 2^62 glide-core pools are created.
static NEXT_POOL_ID: AtomicU64 = AtomicU64::new(1 << 62); // 4_611_686_018_427_387_904

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pool_allocate_id() is exposed to TypeScript as a number, but 2^62 is above JavaScript's Number.MAX_SAFE_INTEGER (2^53 - 1). At this magnitude adjacent fetch_add(1) results round to the same JS value (the ULP is 1024), so many ClientPool instances receive an indistinguishable poolId; their CLIENT_TO_POOL, monitor-handle, and discard-queue entries then overwrite or drain one another. Keep this allocator within the safe integer range (or expose the ID as a BigInt) and separate the two pool kinds without relying on an unsafe numeric offset.

@affonsov affonsov Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved by the Node pool migration. pool_allocate_id() no longer exists — ClientPool.ts now calls createPool() which delegates to glide-core::register_pool(), same as Java, Go, and Python. No separate TS-side pool ID is needed.

Comment thread ffi/src/pool_ffi.rs
@@ -227,20 +234,22 @@ pub unsafe extern "C" fn glide_pool_create(
if pool.state.load(AtomicOrdering::Acquire) != POOL_RUNNING {
return;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the pool is destroyed while this client is being created, this branch returns without releasing adapter_ptr. create_pool_client() has already transferred an Arc<ClientAdapter> into the raw pointer and get_pool_clients() is populated only below, so glide_pool_destroy() has no way to find/drop it. The same early-return shape exists in the two on-demand creation paths below. Reconstruct and drop the raw Arc before returning when POOL_RUNNING is false.

@affonsov affonsov Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. All three early-return paths after create_pool_client() now reconstruct and drop the Arc before returning:

unsafe { drop(Arc::from_raw(adapter_ptr as *const ClientAdapter)); }

Comment thread java/src/lib.rs
// Guard arms immediately on task entry — flag was already set true before spawn.
// This ensures the flag is cleared on every exit path: normal completion,
// early ?-return inside the inner async block, and task cancellation.
let _unmark_guard = UnmarkOnDrop(pre_blocking_arc.clone());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A single AtomicBool plus one UnmarkOnDrop per request is not safe when requests overlap on the same multiplexed pooled client. For example, two concurrent BLPOPs both store true; when the first future completes, its guard stores false while the second is still blocking, allowing the abandon monitor to reclaim the client. Node and FFI have the same last-writer-wins behavior. Track an active-blocking count (increment before dispatch, decrement on drop, and let the monitor test count > 0) rather than independently clearing a shared boolean.

@affonsov affonsov Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Safe by design: the pool enforces exclusive borrow — only one caller holds a pooled client at a time via try_acquire/release_client_async. Concurrent blocking commands on the same pooled client are not possible through the pool API. This invariant is documented in the BLOCKING_FLAG_REGISTRY doc comment.

@affonsov
affonsov force-pushed the fix/java-pool-abandon-monitor-blocking-race branch from 881aee4 to a9bcb70 Compare September 11, 2026 19:35
@affonsov
affonsov force-pushed the fix/java-pool-abandon-monitor-blocking-race branch from a9bcb70 to 27c3a48 Compare September 11, 2026 20:26
@affonsov
affonsov force-pushed the fix/java-pool-abandon-monitor-blocking-race branch from 27c3a48 to 32b6dea Compare September 11, 2026 21:20
@affonsov
affonsov force-pushed the fix/java-pool-abandon-monitor-blocking-race branch from 32b6dea to b5034ab Compare September 11, 2026 21:33
@affonsov
affonsov force-pushed the fix/java-pool-abandon-monitor-blocking-race branch from b5034ab to 07539a7 Compare September 11, 2026 21:37
@affonsov
affonsov force-pushed the fix/java-pool-abandon-monitor-blocking-race branch from 07539a7 to 5a51afc Compare September 11, 2026 21:52
@affonsov
affonsov force-pushed the fix/java-pool-abandon-monitor-blocking-race branch from d8e608d to 5dcd472 Compare September 14, 2026 20:42
…ssue #6971)

Signed-off-by: affonsov <67347924+affonsov@users.noreply.github.com>
…ace (#6971)

The abandon monitor can reclaim a pooled client parked on a blocking
command (BLPOP, XREAD BLOCK, etc.) because mark_client_blocking() used
try_lock() on the pool TokioMutex. Under lock contention the flag was
never set, so the monitor treated the client as idle-abandoned.

Fix: at borrow time, try_acquire() now returns the Arc<AtomicBool> for
the client alongside its client_id. jni_pool stores this Arc in a new
lockless DashMap (handle_id -> Arc<AtomicBool>). The JNI dispatch path
sets is_blocking directly via .store() — no pool lock, no try_lock, no
race.

mark_client_blocking() and get_client_blocking_flag() are retained for
the non-Java FFI path (ffi/src/lib.rs) which still uses try_lock().
Java JNI callers now use the lock-free Arc path exclusively.
Adds: JNI_POOL_BLOCKING_FLAG_MAP populated at borrow time.

Signed-off-by: affonsov <67347924+affonsov@users.noreply.github.com>
…e pool to core (#6971, #6887)

Fixes #6971: BLOCKING_FLAG_REGISTRY in glide-core eliminates the try_lock race.
Fixes #6887: Node ClientPool.ts now uses glide-core::ClientPool via createPool/
poolTryAcquire/poolAcquireBlocking/poolRelease/poolMetrics/poolDestroy, removing
all TS-managed idle/active/waiter state.

Key changes:
- Add BLOCKING_FLAG_REGISTRY and CLIENT_TO_POOL to glide-core/src/pool.rs
- Extract create_handle_for_client() and run_worker_message() from
  create_direct_client() in node/rust-client/src/lib.rs
- Add closeForPoolRelease() to GlideClientHandle (stops worker without
  removing from scope registry, enabling safe client reuse across pool cycles)
- Add pool_build_handle() to node/rust-client/src/pool.rs (JIT wraps a
  pool-acquired client in a fresh N-API handle with per-acquire wake callback)
- Remove CLIENT_ACTIVITY, DISCARDED_CLIENTS, TS_MONITOR_HANDLES, NEXT_POOL_ID,
  pool_allocate_id(), pool_register_client(), pool_unregister_client(),
  pool_start_monitor(), pool_stop_monitor(), start_ts_abandon_monitor(),
  refresh_ts_activity() from node/rust-client/src/pool.rs
- Add serializeConnectionRequest(), createClientFromHandle() to BaseClient
- Add serializeConfig(), createFromHandle() to GlideClient and GlideClusterClient
- Rewrite ClientPool.ts to delegate all pool state to glide-core
- C#, PHP, Ruby and future clients get the fix for free via get_blocking_flag()

Signed-off-by: affonsov <67347924+affonsov@users.noreply.github.com>
Signed-off-by: affonsov <67347924+affonsov@users.noreply.github.com>
… notification race

The refactoring of create_direct_client moved Client::new onto the
glide-core multi-thread runtime, creating a window where PubSub push
notifications (subscription confirmations) arrived before the push
listener task was running. Messages buffered silently with no
wake_callback fired, causing PubSub tests to time out.

Restore the original ordering: Client::new runs inside spawn_pinned
so the push listener is set up atomically with the connection.
create_handle_for_client is retained for the pool path only, where
clients are pre-built and never have PubSub subscriptions.

Signed-off-by: affonsov <67347924+affonsov@users.noreply.github.com>
…Sub race)

- Move create_test_glide_client before mod tests to satisfy clippy::items-after-test-module
- Fix broken rustdoc intra-doc links in node/rust-client
- Fix UnmarkOnDrop scope for MIRI compilation in ffi/src/lib.rs
- Move deferred.resolve after spawn_local(push listener) in create_direct_client
  to eliminate race where cluster-mode push notifications arrive before the
  listener task is scheduled

Signed-off-by: affonsov <67347924+affonsov@users.noreply.github.com>
@affonsov
affonsov force-pushed the fix/java-pool-abandon-monitor-blocking-race branch from 5dcd472 to e02732b Compare September 14, 2026 22:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant