Batch node creation to avoid oversized Bolt transactions - #316
Merged
Conversation
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Mirror the edge-batching pattern on
create_nodes: chunknodeListin Python (batch_size=5000) and send one Bolt transaction per chunk, with progress logging.Motivation
In production we observed the Neo4j driver hitting:
while a graph-creation job was writing nodes for a large repo. Cause:
create_nodespasses the entirenodeListin a single Bolt transaction. For large repos this is a multi-MB UNWIND payload that the server takes long enough to process that AuraDB / network middleboxes close the underlying connection — the next driver call surfaces asdefunct connectionand the routing table refresh fails.create_edgesalready chunks server-side viaapoc.periodic.iterate, but without a Python-level batch loop the wire payload is unbounded the same way. This PR fixes onlycreate_nodes(the smaller patch that addresses the symptom we observed); the same treatment forcreate_edgesis a sensible follow-up.Behavior
len(nodeList)items.len(nodeList)/ 5000) transactions, each carrying ≤ 5000 nodes. The innerapoc.periodic.iteratebatch size stays at 100 (unchanged), so server-side processing semantics are identical.Creating N nodes in batches of M) and a per-batch progress line, matching the style already used increate_edges.Note on prod rollout
The LSP server pins
blarifytofeat/frontend-prompt-templatesin itspyproject.toml. This PR lands ondev, which has a different module layout (blarify/db_managers/neo4j_manager.py) than the pinned branch (blarify/repositories/graph_db_manager/neo4j_manager.py). To get this fix into prod we'll need either:dev→feat/frontend-prompt-templates, orfeat/frontend-prompt-templatesapplying the same idea to its newerNeo4jManager.create_nodes(which sends the wholenodeListin oneexecute_writeand only chunks via APOC).Test plan
poetry run ruff check blarify/db_managers/neo4j_manager.py— clean (verified locally).poetry run pyright blarify/db_managers/neo4j_manager.py— no new errors in this file (8 pre-existing errors in__init__are out of scope).create_graphagainst a large repo and watch forCreating N nodes in batches of 5000/ per-batch lines in logs; confirm nodefunct connection/ routing errors mid-write.