Skip to content

fix(RavenDB): Node sequence startup race#3352

Merged
jeremydmiller merged 2 commits into
JasperFx:mainfrom
danielwinkler:fix/node-startup-race
Jul 10, 2026
Merged

fix(RavenDB): Node sequence startup race#3352
jeremydmiller merged 2 commits into
JasperFx:mainfrom
danielwinkler:fix/node-startup-race

Conversation

@danielwinkler

Copy link
Copy Markdown
Contributor

Problem

During startup, when multiple Wolverine nodes registered concurrently against RavenDB, PersistAsync could hand out duplicate AssignedNodeNumbers. Each concurrent call loaded the same NodeSequence document, incremented Count to the same value, and committed — so two nodes could end up with the same assigned number.

Fix

RavenDbMessageStore.PersistAsync now retries the load/increment/store cycle inside a bounded loop, relying on RavenDB cluster-wide transaction concurrency (compare-exchange atomic guards on nodes/sequence) to detect the collision:

  • Catches ClusterTransactionConcurrencyException / ConcurrencyException and retries (up to 25 attempts).
  • Uses linear backoff with jitter to avoid synchronized retries (thundering herd) under startup contention.
  • Throws a clear InvalidOperationException (with the last conflict as inner exception) if the sequence keeps conflicting past the attempt limit.

Tests

Added concurrently_persisting_nodes_assigns_unique_node_numbers to the RavenDB node-persistence compliance suite. It fires 20 PersistAsync calls simultaneously (released via a shared TaskCompletionSource) and asserts the returned numbers — and the persisted DB state — form exactly 1..20 with no duplicates.

Notes

  • No public API changes; behavior change is limited to node registration retry semantics.
  • Cluster-wide transactions still enforce per-document atomic guards even with UseOptimisticConcurrency = false, which is the mechanism the retry leans on.

Want me to commit and push this, or adjust the tone/length for the repo's PR conventions?

@jeremydmiller

Copy link
Copy Markdown
Member

@danielwinkler I'd rather just get this in if you're okay with that.

@jeremydmiller
jeremydmiller merged commit 2fe097d into JasperFx:main Jul 10, 2026
25 checks passed
This was referenced Jul 10, 2026
This was referenced Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants