feat: streaming merger for lattice-compatible distributed aggregation#8
Merged
Conversation
Adds StreamingMerger that folds worker IPC results incrementally using lattice merge operations instead of batching all results for a single SQL merge query. The Coordinator auto-selects streaming vs batch based on PipelineSplitter's streaming_compatible? flag. Benefits: lower memory (don't hold all IPC binaries simultaneously), lower latency (merge starts when first worker finishes), and telemetry events per worker completion for progressive result tracking. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…peer tests + benchmarks Adds missing test categories per CLAUDE.md quality bar: - Sad path: empty IPC, no folds, all workers fail, mixed lattice - Adversarial: 100 groups, repeated folding, nil group keys, AVG rewrite - Wicked: streaming vs batch idempotence, many groups × many workers, fold order commutativity - Property: SUM commutativity, MIN/MAX commutativity - Peer: 5 cross-node streaming tests (SUM+COUNT, MIN+MAX, grouped, AVG rewrite, penguins dataset) Benchmarks: streaming MIN+MAX 3.29ms, SUM+COUNT grouped 9.41ms. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Dux.Remote.StreamingMerger— folds worker IPC results incrementally using lattice merge operations instead of batching all results for a single SQL merge querystreaming_compatible?from PipelineSplitter; uses streaming for SUM/COUNT/MIN/MAX pipelines, falls back to batch for non-lattice aggregates (MEDIAN, etc.)[:dux, :distributed, :streaming_merge]events after each worker result is folded, with progress metadata (workers_complete, workers_total)record_failure/1counts failed workers without crashing the mergeordered: falsein Task.async_stream for lower latencyBenchmarks
Test plan
🤖 Generated with Claude Code