Skip to content

[flink] Upgrade sink connector to new API version#205

Merged
wuchong merged 5 commits into
apache:mainfrom
michaelkoepf:flink/connectors-upgrade-to-new-sink-api
Feb 27, 2025
Merged

[flink] Upgrade sink connector to new API version#205
wuchong merged 5 commits into
apache:mainfrom
michaelkoepf:flink/connectors-upgrade-to-new-sink-api

Conversation

@michaelkoepf

@michaelkoepf michaelkoepf commented Dec 17, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #132

Upgrade sink connector from deprecated RichSinkFunction (will be removed in Flink 2.0) to new Sink interface

Tests

  • UT (adapted to new implementation, test case logic unchanged): com.alibaba.fluss.connector.flink.sink.FlinkSinkWriterTest.java
  • IT (unchanged, because external interface did not change): com.alibaba.fluss.connector.flink.sink.FlinkTableSinkITCase.java

API and Format

n/a

Documentation

n/a

@michaelkoepf michaelkoepf mentioned this pull request Dec 17, 2024
Closed
2 tasks
@ChaomingZhangCN

ChaomingZhangCN commented Dec 17, 2024

Copy link
Copy Markdown
Contributor

LGTM, cc @wuchong.

@michaelkoepf michaelkoepf changed the title [flink] Upgraded sink connector to new API version [flink] Upgrade sink connector to new API version Dec 17, 2024
wuchong
wuchong reviewed Feb 22, 2025
View reviewed changes

@wuchong wuchong left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the late reviewing and thanks @michaelkoepf for the contribution. The changes looks good to me in general. Could you rebase the branch to latest main branch to resolve the conflicts?

Comment thread ...onnector-flink/src/test/java/com/alibaba/fluss/connector/flink/sink/FlinkSinkWriterTest.java Outdated
Comment thread ...rs/fluss-connector-flink/src/main/java/com/alibaba/fluss/connector/flink/sink/FlinkSink.java Outdated
@michaelkoepf michaelkoepf force-pushed the flink/connectors-upgrade-to-new-sink-api branch 4 times, most recently from 9c08511 to 2da3bdc Compare February 22, 2025 14:54
@michaelkoepf

michaelkoepf commented Feb 22, 2025

Copy link
Copy Markdown
Contributor Author

@wuchong addressed feedback and rebased, ptal

wuchong
wuchong approved these changes Feb 24, 2025
View reviewed changes

@wuchong wuchong left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updating. LGTM.

@wuchong wuchong force-pushed the flink/connectors-upgrade-to-new-sink-api branch from 2da3bdc to 5f93c5f Compare February 25, 2025 08:12
@wuchong

wuchong commented Feb 25, 2025

Copy link
Copy Markdown
Member

Hi @michaelkoepf , I know why the CI is stuck on FlussDatabaseSyncSourceITCase. It can be easily reproduced in local. The reason is the hard cast here will fail, because we have upgraded from SinkFunctionProvider to SinkV2Provider in this PR, and the cast in this line will fail. So we have to upgrade TestingDatabaseSyncSink to SinkV2 as well. Could you help to fix it?

https://github.com/alibaba/fluss/blob/70229b8687297295fe29650016aed3a77137dbc1/fluss-lakehouse/fluss-lakehouse-paimon/src/test/java/com/alibaba/fluss/lakehouse/paimon/testutils/TestingDatabaseSyncSink.java#L92-L96

@michaelkoepf

michaelkoepf commented Feb 25, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

@wuchong i will look into the suggested issue.

@michaelkoepf michaelkoepf force-pushed the flink/connectors-upgrade-to-new-sink-api branch from 2132b2a to 2da3bdc Compare February 25, 2025 23:52
@michaelkoepf
[flink] Upgraded sink connector to new API version
89cd63f 
- Upgraded sink connector from deprecated RichSinkFunction to new Sink interface
- Adapted corresponding unit test cases

Issue apache#132
@michaelkoepf
[flink] Upgraded sink connector to new API version
2e41a99 
- Adapted access modifiers
- Consistent naming

Issue apache#132
@michaelkoepf
Addressed PR feedback
3922439
@michaelkoepf
Adapted integration tests to flink sink v2
6dd724f
@michaelkoepf michaelkoepf force-pushed the flink/connectors-upgrade-to-new-sink-api branch from 2da3bdc to 6dd724f Compare February 26, 2025 23:05
@michaelkoepf

michaelkoepf commented Feb 26, 2025

Copy link
Copy Markdown
Contributor Author

@wuchong fixed

[INFO] Fluss : ............................................ SUCCESS [  1.906 s]
[INFO] Fluss : Test utils ................................. SUCCESS [  3.650 s]
[INFO] Fluss : Common ..................................... SUCCESS [ 15.666 s]
[INFO] Fluss : Metrics : .................................. SUCCESS [  0.077 s]
[INFO] Fluss : Metrics : Prometheus ....................... SUCCESS [  2.893 s]
[INFO] Fluss : Metrics : JMX .............................. SUCCESS [  2.509 s]
[INFO] Fluss : ProtoGen : ................................. SUCCESS [  0.049 s]
[INFO] Fluss : ProtoGen : Generator ....................... SUCCESS [  0.367 s]
[INFO] Fluss : ProtoGen : Maven Plugin .................... SUCCESS [  4.008 s]
[INFO] Fluss : RPC ........................................ SUCCESS [ 19.378 s]
[INFO] Fluss : Server ..................................... SUCCESS [04:28 min]
[INFO] Fluss : Client ..................................... SUCCESS [04:58 min]
[INFO] Fluss : FileSystems : .............................. SUCCESS [  0.053 s]
[INFO] Fluss : FileSystems : Hadoop FS shaded ............. SUCCESS [  3.810 s]
[INFO] Fluss : FileSystems : Hadoop FS .................... SUCCESS [ 20.107 s]
[INFO] Fluss : FileSystems : OSS FS ....................... SUCCESS [ 11.021 s]
[INFO] Fluss : FileSystems : S3 FS ........................ SUCCESS [  7.628 s]
[INFO] Fluss : Connector : ................................ SUCCESS [  0.062 s]
[INFO] Fluss : Connector : Flink .......................... SUCCESS [06:06 min]
[INFO] Fluss : Lakehouse : ................................ SUCCESS [  0.047 s]
[INFO] Fluss : Lakehouse : Paimon ......................... SUCCESS [03:22 min]
[INFO] Fluss : Lakehouse : CLI ............................ SUCCESS [  7.450 s]
[INFO] Fluss : Dist ....................................... SUCCESS [ 12.843 s]

wuchong
wuchong approved these changes Feb 27, 2025
View reviewed changes

@wuchong wuchong left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@wuchong wuchong merged commit 0b623e4 into apache:main Feb 27, 2025
@michaelkoepf michaelkoepf mentioned this pull request Feb 28, 2025
Closed
2 tasks
@michaelkoepf michaelkoepf deleted the flink/connectors-upgrade-to-new-sink-api branch March 16, 2025 19:00
ZmmBigdata pushed a commit to ZmmBigdata/fluss that referenced this pull request Jun 20, 2025
@michaelkoepf @ZmmBigdata
[flink] Upgrade sink connector to new SinkV2 API (apache#205)
1d639f9
Ugbot pushed a commit to Ugbot/fluss that referenced this pull request Apr 26, 2026
@michaelkoepf
[flink] Upgrade sink connector to new SinkV2 API (apache#205)
72dc339
fresh-borzoni pushed a commit to fresh-borzoni/fluss that referenced this pull request Jul 3, 2026
@linguoxuan
test: add it for produce & scan log for all supported datatypes (apac…
1bc399b 
…he#205)
fresh-borzoni pushed a commit that referenced this pull request Jul 4, 2026
@linguoxuan
test: add it for produce & scan log for all supported datatypes (#205)
24c4634
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wuchong wuchong wuchong approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature] Use new sink api for Flink connector

3 participants

@michaelkoepf @ChaomingZhangCN @wuchong