Skip to content

Batch node creation to avoid oversized Bolt transactions - #316

Merged
berrazuriz1 merged 1 commit into
devfrom
feat/batch-create-nodes
May 25, 2026
Merged

Batch node creation to avoid oversized Bolt transactions#316
berrazuriz1 merged 1 commit into
devfrom
feat/batch-create-nodes

Conversation

@berrazuriz1

Copy link
Copy Markdown
Contributor

Summary

Mirror the edge-batching pattern on create_nodes: chunk nodeList in Python (batch_size=5000) and send one Bolt transaction per chunk, with progress logging.

Motivation

In production we observed the Neo4j driver hitting:

neo4j.io: <CONNECTION> error: Failed to read from defunct connection IPv4Address(('p-0b67...
neo4j.pool: Unable to retrieve routing information

while a graph-creation job was writing nodes for a large repo. Cause: create_nodes passes the entire nodeList in a single Bolt transaction. For large repos this is a multi-MB UNWIND payload that the server takes long enough to process that AuraDB / network middleboxes close the underlying connection — the next driver call surfaces as defunct connection and the routing table refresh fails.

create_edges already chunks server-side via apoc.periodic.iterate, but without a Python-level batch loop the wire payload is unbounded the same way. This PR fixes only create_nodes (the smaller patch that addresses the symptom we observed); the same treatment for create_edges is a sensible follow-up.

Behavior

  • Before: one Bolt transaction with len(nodeList) items.
  • After: ceil(len(nodeList) / 5000) transactions, each carrying ≤ 5000 nodes. The inner apoc.periodic.iterate batch size stays at 100 (unchanged), so server-side processing semantics are identical.
  • Adds two log lines: a one-line summary (Creating N nodes in batches of M) and a per-batch progress line, matching the style already used in create_edges.

Note on prod rollout

The LSP server pins blarify to feat/frontend-prompt-templates in its pyproject.toml. This PR lands on dev, which has a different module layout (blarify/db_managers/neo4j_manager.py) than the pinned branch (blarify/repositories/graph_db_manager/neo4j_manager.py). To get this fix into prod we'll need either:

  • merge devfeat/frontend-prompt-templates, or
  • a sibling commit on feat/frontend-prompt-templates applying the same idea to its newer Neo4jManager.create_nodes (which sends the whole nodeList in one execute_write and only chunks via APOC).

Test plan

  • poetry run ruff check blarify/db_managers/neo4j_manager.py — clean (verified locally).
  • poetry run pyright blarify/db_managers/neo4j_manager.py — no new errors in this file (8 pre-existing errors in __init__ are out of scope).
  • Manual: run a create_graph against a large repo and watch for Creating N nodes in batches of 5000 / per-batch lines in logs; confirm no defunct connection / routing errors mid-write.

@berrazuriz1
berrazuriz1 merged commit eda0ae1 into dev May 25, 2026
5 of 6 checks passed
@berrazuriz1
berrazuriz1 deleted the feat/batch-create-nodes branch May 25, 2026 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant