bulkload-db: retry chunk uploads and fail loud on incomplete loads

[kevinschaper]

Problem

bulkload_duckdb silently swallowed per-chunk upload failures. A chunk that timed out (or returned a 5xx) was caught, logged, and counted as 0 — but the load still finished and reported success with a short document count. The CLI then printed Successfully committed <N> documents and exited 0.

In a Monarch KG release build this shipped a Solr entity core with only ~728k of ~1.46M documents (and the association core similarly short) while the Jenkins job passed as SUCCESS. The downstream symptom was get_entity() returning None for genes/diseases that were present in the same release's graph — because their chunk had timed out during indexing and was never retried or flagged.

Root cause in the logs:

Processing 1462594 rows ... with 12 parallel workers
Error processing DuckDB chunk at offset 0: ('Connection aborted.', TimeoutError('timed out'))
... (many chunks time out) ...
Progress: 562,594/1,462,594 documents loaded
DuckDB upload complete!   # <- still "success", exit 0

Fix

• _upload_docs_to_solr: retries transient failures (timeouts, connection resets, 5xx) with exponential backoff. Re-POSTing a chunk is idempotent because every doc carries a unique id, so a retry overwrites rather than duplicates — safe even when a prior attempt timed out client-side but Solr had already indexed the batch. After exhausting retries it raises instead of returning 0.
• bulkload_duckdb: tracks failed chunks and raises unless total_loaded == total_rows; the outer handler re-raises rather than returning 0, so the CLI exits non-zero on any incomplete load.

Validation

Against a local Solr with the patched loader:

• Full load of 1,462,594 rows → committed exactly 1,462,594, exit 0, zero retries needed at the default 4 workers.
• Fail-loud: pointed at an unreachable Solr → retried with backoff, then RuntimeError: Incomplete Solr load ... uploaded 0/172,957 rows ... Refusing to report success, exit 1.
• Subsequently used to build a full Monarch release index (entity 1,462,594 / association 15,211,571 / sssom 1,276,312) — all cores complete, counts matched.

Builds on the streaming bulkload_duckdb already on main (#22 / 1a9195a).