Skip to content

Fix blocksToDelete shipper initialization race - #7753

Open
Rohithmatham12 wants to merge 1 commit into
cortexproject:masterfrom
Rohithmatham12:fix/6413-blocks-to-delete-race
Open

Fix blocksToDelete shipper initialization race#7753
Rohithmatham12 wants to merge 1 commit into
cortexproject:masterfrom
Rohithmatham12:fix/6413-blocks-to-delete-race

Conversation

@Rohithmatham12

Copy link
Copy Markdown

What this changes

  • Initializes the per-tenant Thanos shipper before assigning userDB.db in createTSDB.
  • Keeps blocksToDelete from observing a partially initialized userTSDB where db is visible but shipper is still being written.
  • Leaves the shipped-block cache initialization after the TSDB is fully assigned, preserving the existing behavior.

Why
tsdb.Open installs userDB.blocksToDelete as a callback and can run block reload work concurrently. Previously createTSDB assigned userDB.db before creating userDB.shipper, so the callback could read userDB.shipper while the constructor wrote it. Creating the shipper before making db visible removes that race window without adding locks to the callback path.

Fixes #6413.

Testing

  • git diff --check
  • GOCACHE=/private/tmp/cortex-go-cache-6413 GOMODCACHE=/private/tmp/cortex-go-mod-cache go test -tags "netgo slicelabels" ./pkg/ingester -run "TestIngester_shipBlocks|TestIngester_dontShipBlocksWhenTenantDeletionMarkerIsPresent|TestIngester_TSDB"
  • GOCACHE=/private/tmp/cortex-go-cache-6413-full GOMODCACHE=/private/tmp/cortex-go-mod-cache go test -tags "netgo slicelabels" ./pkg/ingester

Signed-off-by: Rohithmatham12 <rohithmatham@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Race condition on blocksToDelete

1 participant