feat(connectors): Clickhouse Sink Connector by kriti-sc · Pull Request #2886 · apache/iggy

kriti-sc · 2026-03-06T19:42:08Z

Which issue does this PR close?

Closes #2539

Rationale

Clickhouse is a real-time data analytics engine, and very popular in modern analytics architectures.

What changed?

This PR introduces a Clickhouse Sink Connector that enables writing data from Iggy to Clickhouse.

The Clickhouse writing logic is heavily inspired by the official Clickhouse Kafka Connector.

Local Execution

Produced messages 30456 + 29060 rows with schema user_id: String, user_type: u8, email: String, source: String, state: String, created_at: DateTime, message: String using sample data producer.
Consumed messages using the Clickhouse sink and into the particular Clickhouse table.
Verified schema and number of rows in Clickhouse.
Added unit tests and e2e tests, both passing.

Images 1&2: Produced 30456 + 29060 rows into Iggy in two batches
Image 3: Verified schema and number of rows in Clickhouse

AI Usage

Which tools? (e.g., GitHub Copilot, Claude, ChatGPT) Claude Code
Scope of usage? (e.g., autocomplete, generated functions, entire implementation) generated functions
How did you verify the generated code works correctly? Manual testing by producing data into Iggy and then running the sink and verifying insertion into Clickhouse, unit tests and e2e tests for different Clickhouse insert configurations.
Can you explain every line of the code if asked? Yes

codecov · 2026-03-06T19:43:18Z

Codecov Report

❌ Patch coverage is 82.56210% with 358 lines in your changes missing coverage. Please review.
✅ Project coverage is 47.42%. Comparing base (8678ead) to head (11c2f47).

Files with missing lines	Patch %	Lines
...ore/connectors/sinks/clickhouse_sink/src/binary.rs	82.95%	122 Missing and 56 partials ⚠️
...ore/connectors/sinks/clickhouse_sink/src/schema.rs	76.45%	30 Missing and 75 partials ⚠️
...ore/connectors/sinks/clickhouse_sink/src/client.rs	72.68%	55 Missing and 10 partials ⚠️
core/connectors/sinks/clickhouse_sink/src/lib.rs	97.27%	3 Missing and 1 partial ⚠️
core/connectors/sinks/clickhouse_sink/src/sink.rs	60.00%	4 Missing ⚠️
core/connectors/sinks/clickhouse_sink/src/body.rs	98.80%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #2886       +/-   ##
=============================================
- Coverage     74.57%   47.42%   -27.15%     
  Complexity      937      937               
=============================================
  Files          1249     1252        +3     
  Lines        123564   110008    -13556     
  Branches      99837    86313    -13524     
=============================================
- Hits          92143    52172    -39971     
- Misses        28438    55018    +26580     
+ Partials       2983     2818      -165

Components	Coverage Δ
Rust Core	`40.92% <82.56%> (-34.81%)`	⬇️
Java SDK	`58.57% <ø> (ø)`
C# SDK	`69.41% <ø> (-0.45%)`	⬇️
Python SDK	`81.06% <ø> (ø)`
PHP SDK	`83.57% <ø> (ø)`
Node SDK	`91.35% <ø> (+0.12%)`	⬆️
Go SDK	`40.25% <ø> (ø)`

Files with missing lines	Coverage Δ
core/connectors/sinks/clickhouse_sink/src/body.rs	`98.80% <98.80%> (ø)`
core/connectors/sinks/clickhouse_sink/src/lib.rs	`97.27% <97.27%> (ø)`
core/connectors/sinks/clickhouse_sink/src/sink.rs	`60.00% <60.00%> (ø)`
...ore/connectors/sinks/clickhouse_sink/src/client.rs	`72.68% <72.68%> (ø)`
...ore/connectors/sinks/clickhouse_sink/src/schema.rs	`76.45% <76.45%> (ø)`
...ore/connectors/sinks/clickhouse_sink/src/binary.rs	`82.95% <82.95%> (ø)`

... and 362 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-03-17T00:36:38Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

hubcio

overall good direction, just needs a little bit polishing

github-actions · 2026-04-02T00:40:00Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

github-actions · 2026-04-18T00:43:17Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

…ickhouse-sink

…house-sink

kriti-sc · 2026-06-02T19:45:07Z

/ready

ryerraguntla · 2026-06-14T03:38:55Z

/author could you look into it

kriti-sc · 2026-06-14T07:15:08Z

/ready

ryerraguntla · 2026-06-14T15:46:23Z

+
+/// Build a RowBinaryWithDefaults body.
+/// Each `Payload::Json` message is serialised to binary using the table schema.
+pub(crate) fn build_row_binary_body(


body.rs:58-77 (build_row_binary_body) vs module doc body.rs:18-23 — propagates Err from serialize_row on first bad row, failing whole batch. Doc claims "mixed batch never causes complete failure" — true for json/string builders, false here. Fix: skip-and-log bad row (match other builders) or document fail-fast as intentional poison-pill in README Reliability section.

ryerraguntla · 2026-06-14T15:51:48Z

+                        if attempts >= max_retries {
+                            error!(
+                                "Insert failed after {attempts} attempts (HTTP {status}): {body_text}"
+                            );


client.rs:243-256 (insert) — Error::CannotStoreData used for BOTH retryable (429/5xx/network) and non-retryable 4xx. Other connectors doris_sink/influxdb_sink use Error::PermanentHttpError for 4xx, per SDK doc "so circuit breakers are not tripped by bad data" (sdk/src/lib.rs:420-425).
Fix: 4xx branch → Error::PermanentHttpError.

ryerraguntla · 2026-06-14T15:54:10Z

+    container: ContainerAsync<GenericImage>,
+    pub base_url: String,
+}
+


fixtures/clickhouse/container.rs:90-96 — ClickHouseContainer::start() missing.with_container_name(fixtures::unique_container_name("clickhouse")). postgres/influxdb fixtures set this; fixtures/mod.rs:32-36 docs iggy-test-* sweep convention (docker ps -aqf 'name=^iggy-test-'). Without it, container gets random testcontainers name, invisible to cleanup sweep, leaks on crash/SIGKILL. Fix: add the call.

ryerraguntla · 2026-06-14T16:03:24Z

/author

add clickhouse sink

4c2ae73

kriti-sc added 2 commits March 7, 2026 01:18

fix linting errors

7c7430b

fix linting errors

fc34e4d

abonander reviewed Mar 6, 2026

View reviewed changes

Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs

kriti-sc requested a review from abonander March 7, 2026 10:58

atharvalade requested changes Mar 7, 2026

View reviewed changes

Comment thread core/connectors/sinks/clickhouse_sink/src/binary.rs Outdated

hubcio mentioned this pull request Mar 15, 2026

feat(connectors): add ClickHouse sink connector #2891

Closed

github-actions Bot added S-stale Inactive issue or pull request and removed S-stale Inactive issue or pull request labels Mar 17, 2026

hubcio requested changes Mar 18, 2026

View reviewed changes

kriti-sc and others added 4 commits March 26, 2026 00:20

Merge branch 'master' into clickhouse-sink

4cd3eb8

fix ipv4 datatype binary conversion

9813d1d

map tuple object payloads with name not positionally

e6abb9b

set default headers

2fa7213

github-actions Bot added S-stale Inactive issue or pull request and removed S-stale Inactive issue or pull request labels Apr 2, 2026

kriti-sc and others added 5 commits April 3, 2026 23:08

address type conversion issues

68ddfb1

address type conversion issues

9e7c817

Merge branch 'master' into clickhouse-sink

8770526

linting fix

f282f36

implement PartialEq

75b9b58

kriti-sc requested a review from hubcio April 3, 2026 19:56

Merge branch 'master' into clickhouse-sink

236bd15

github-actions Bot added S-stale Inactive issue or pull request and removed S-stale Inactive issue or pull request labels Apr 18, 2026

Merge branch 'master' into clickhouse-sink

015f51a

kriti-sc and others added 12 commits May 30, 2026 22:15

Address: Optimize String Key Serialization

ce772ac

Address: Optimize Return Type

fd4c433

Address: Optimize INSERT query reconstruction efficiency.

21b1bc3

Address: Optimize Buffer Allocation Strategy

83ad28c

Address: Support Backtick Identifiers in Tuples

717ff20

Address: JSONEachRow Requires Newline Delimiter

2bbe3b4

Merge branch 'pr-2886/jsoneachrow-requires-newline-delimiter' into cl…

e01bc5b

…ickhouse-sink

Merge branch 'master' into clickhouse-sink

eb0c953

Address: Non-Deterministic Map Iteration Issue

3279498

Merge branch 'clickhouse-sink' of github.com:kriti-sc/iggy into click…

c5e4a73

…house-sink

fix linting errors

aa7ad2e

fix linting errors

ec1064f

github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jun 2, 2026

kriti-sc and others added 3 commits June 8, 2026 01:46

Merge branch 'master' into clickhouse-sink

0b9c94e

make headers uniform

bc7ce0a

Merge branch 'master' into clickhouse-sink

11c2f47

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jun 14, 2026

github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Jun 14, 2026

ryerraguntla reviewed Jun 14, 2026

View reviewed changes

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Jun 14, 2026

Conversation

kriti-sc commented Mar 6, 2026

Which issue does this PR close?

Rationale

What changed?

Local Execution

AI Usage

Uh oh!

codecov Bot commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Mar 17, 2026

Uh oh!

hubcio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 2, 2026

Uh oh!

github-actions Bot commented Apr 18, 2026

Uh oh!

kriti-sc commented Jun 2, 2026

Uh oh!

ryerraguntla commented Jun 14, 2026

Uh oh!

kriti-sc commented Jun 14, 2026

Uh oh!

ryerraguntla Jun 14, 2026

Choose a reason for hiding this comment

Uh oh!

ryerraguntla Jun 14, 2026

Choose a reason for hiding this comment

Uh oh!

ryerraguntla Jun 14, 2026

Choose a reason for hiding this comment

Uh oh!

ryerraguntla commented Jun 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov Bot commented Mar 6, 2026 •

edited

Loading