Skip to content

feat: streaming merger for lattice-compatible distributed aggregation#8

Merged
cigrainger merged 2 commits into
mainfrom
cg/streaming-merger
Mar 23, 2026
Merged

feat: streaming merger for lattice-compatible distributed aggregation#8
cigrainger merged 2 commits into
mainfrom
cg/streaming-merger

Conversation

@cigrainger

@cigrainger cigrainger commented Mar 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Dux.Remote.StreamingMerger — folds worker IPC results incrementally using lattice merge operations instead of batching all results for a single SQL merge query
  • Coordinator auto-selection — checks streaming_compatible? from PipelineSplitter; uses streaming for SUM/COUNT/MIN/MAX pipelines, falls back to batch for non-lattice aggregates (MEDIAN, etc.)
  • Telemetry — emits [:dux, :distributed, :streaming_merge] events after each worker result is folded, with progress metadata (workers_complete, workers_total)
  • Worker failure trackingrecord_failure/1 counts failed workers without crashing the merge
  • Unordered fan-out — streaming path uses ordered: false in Task.async_stream for lower latency

Benchmarks

Metric Value
Streaming MIN+MAX (2 workers) 3.29ms
Streaming SUM+COUNT grouped (2 workers) 9.41ms
Distributed filter+agg (2 workers, streaming) 4.09ms

Test plan

  • Happy path: fold SUM, fold MIN/MAX, fold grouped, progress tracking, failure recording
  • Sad path: empty IPC, no folds → empty Dux, all workers fail, mixed lattice+non-lattice, filter-only pipeline
  • Adversarial: 100 groups, repeated folding (5x), nil group keys, AVG rewrite end-to-end
  • Wicked: streaming vs batch idempotence for MIN/MAX, many groups × many workers, fold order commutativity
  • Property tests: SUM commutativity (3 partitions, any order), MIN/MAX commutativity
  • Integration: SUM+COUNT, MIN+MAX, grouped SUM, 3-worker SUM, penguins dataset
  • Peer tests: 5 cross-node streaming tests (SUM+COUNT, MIN+MAX, grouped, AVG rewrite across 3 peers, penguins)
  • Telemetry: streaming_merge events emitted per worker
  • 568 total tests (38 properties, 487 tests, 43 doctests), 0 failures, 0 credo issues

🤖 Generated with Claude Code

cigrainger and others added 2 commits March 23, 2026 14:24
Adds StreamingMerger that folds worker IPC results incrementally using
lattice merge operations instead of batching all results for a single
SQL merge query. The Coordinator auto-selects streaming vs batch based
on PipelineSplitter's streaming_compatible? flag.

Benefits: lower memory (don't hold all IPC binaries simultaneously),
lower latency (merge starts when first worker finishes), and telemetry
events per worker completion for progressive result tracking.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…peer tests + benchmarks

Adds missing test categories per CLAUDE.md quality bar:
- Sad path: empty IPC, no folds, all workers fail, mixed lattice
- Adversarial: 100 groups, repeated folding, nil group keys, AVG rewrite
- Wicked: streaming vs batch idempotence, many groups × many workers,
  fold order commutativity
- Property: SUM commutativity, MIN/MAX commutativity
- Peer: 5 cross-node streaming tests (SUM+COUNT, MIN+MAX, grouped,
  AVG rewrite, penguins dataset)

Benchmarks: streaming MIN+MAX 3.29ms, SUM+COUNT grouped 9.41ms.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@cigrainger
cigrainger merged commit 1f6c531 into main Mar 23, 2026
3 checks passed
@cigrainger
cigrainger deleted the cg/streaming-merger branch March 23, 2026 03:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant