[File system] Add Google Cloud Storage integration by gkatzioura · Pull Request #631 · apache/fluss

gkatzioura · 2025-03-18T23:51:40Z

Purpose

Linked issue: close #630

Brief change log

Added the GSFileSystem based on the given Hadoop Google Cloud Storage file system

Tests

API and Format

No

Documentation

It adds support for the Google Cloud Storage Filesystem

gkatzioura · 2025-03-19T07:37:25Z

Will add integration tests. Putting temporary to draft

gkatzioura · 2025-03-20T10:00:14Z

Added an integration test based on FileSystemBehaviorTestSuite.
For the integration test to run it requires a Google Cloud Storage bucket declared on the variable IT_CASE_GS_BUCKET
and a path of the service account in the IT_CASE_GS_ACCESS_KEY. The service account file needs to be mounted during the CI process.

Since the service account key is in a file form and not in a string form I see two options.

Changing the ci.yaml to read the service account file (base64) from secrets and store it to a path.
During the integration test initialization read the service account file (base64) and store it to a path.

In the pr I followed the second approach.

wuchong · 2025-03-22T09:45:17Z

@luoyuxia could you help to review it?

wuchong · 2025-03-22T09:45:32Z

@gkatzioura , could you check how Flink tests the gs-filesystem in CI?

gkatzioura · 2025-04-04T08:01:10Z

Hi @wuchong , after checking there are not tests for the actual filesystem integration.
Will proceed on adding a mock server thus avoiding the need for integrating with an actual service account.

wuchong · 2025-04-04T09:58:53Z

Thank you @gkatzioura , please ping me when you have finished the mock server.

gkatzioura · 2025-04-08T08:55:40Z

Hi @wuchong & @luoyuxia
I implemented a server simulating Google Cloud Storage, thus the now the test runs pointing to that server. No credentials or actual GCP account needed.
The test is similar to the ones found on the FileSystemTest.java.
Should we need to bring back the Google Cloud Storage integration test can do on demand.

wuchong · 2025-04-16T02:55:59Z

checkstyle are failed

2025-04-14T15:08:32.2197637Z [INFO] --- maven-jar-plugin:2.4:jar (default) @ fluss-fs-oss ---
2025-04-14T15:08:32.2261014Z [INFO] There are 2 errors reported by Checkstyle 9.3 with /tools/maven/checkstyle.xml ruleset.
2025-04-14T15:08:32.2283668Z [ERROR] src/test/java/com/alibaba/fluss/fs/gs/GSPathServerHandler.java:[42,1] (javadoc) MissingJavadocType: Missing a Javadoc comment.
2025-04-14T15:08:32.2289030Z [ERROR] src/test/java/com/alibaba/fluss/fs/gs/MockAuthServer.java:[35,1] (javadoc) MissingJavadocType: Missing a Javadoc comment.

wuchong · 2025-04-16T02:56:14Z

@luoyuxia could you help to review this PR?

luoyuxia · 2025-04-16T02:57:51Z

@luoyuxia could you help to review this PR?

Sure, now I got some free time. I'll help review these days.

luoyuxia

@gkatzioura Thanks for the greate work! Left some comments.

wuchong · 2025-05-10T13:31:24Z

Hi @gkatzioura , you can "request review" again when you have addressed all the comments, so that the reviewer can get the notification in time.

luoyuxia

@gkatzioura Could you please rebase main branch and add licencese since currently licences check is enabeld.

luoyuxia · 2025-05-14T06:38:59Z

+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.AssertionsForClassTypes.assertThatThrownBy;
+
+class GSFileSystemPluginTest {


Please don't forget this comment.

luoyuxia · 2025-06-03T08:48:07Z

@gkatzioura Thanks for the great work. I'll have a final check. Hope to merge in later days

luoyuxia

@gkatzioura Thanks for the work. LGTM! I append a commit to add licenses and refine test. Please take a look. If you have no further comments, I'll merge later

* [rust] Add Fluss 1.x protocol support to the admin client Add 27 new admin methods to FlussAdmin: - Database/table extensions: list_database_summaries, alter_database, alter_table, get_table_stats - KV snapshot operations: get_latest_kv_snapshots, get_kv_snapshot_metadata, create_kv_snapshot_lease, get_lake_snapshot - ACL management: create_acls, list_acls, drop_acls - Cluster configuration: describe_cluster_configs, alter_cluster_configs - Server management: add_server_tag, remove_server_tag, rebalance, list_rebalance_progress, cancel_rebalance - Producer offsets: register_producer_offsets, get_producer_offsets, delete_producer_offsets - Monitoring: get_cluster_health, list_remote_log_manifests - KV snapshots: list_kv_snapshots, release_kv_snapshot_lease, drop_kv_snapshot_lease * Address reviewer feedback: add domain enums, type aliases, expose readable - Add ClusterHealthStatus enum (Green/Yellow/Red/Unknown) to cluster_health.rs - Add RebalanceStatus enum (NotStarted..Timeout) to rebalance.rs - Replace raw i32/i64 with BucketId/TableId/PartitionId aliases in kv_snapshot.rs, lake_snapshot.rs, rebalance.rs, producer_offsets.rs - Expose readable parameter in admin.get_lake_snapshot() instead of hardcoding None Addresses reviewer feedback on PR apache#631. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Apply type aliases to remaining metadata types (table_stats, kv_snapshot_lease) Extends the BucketId/TableId/PartitionId alias consistency fix to table_stats.rs (BucketStatsRequest, BucketStats) and kv_snapshot_lease.rs (KvSnapshotLeaseForBucket, KvSnapshotLeaseForTable). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update admin.rs to use GoalType/ServerTag enums after rebase Now that pr/3 provides GoalType and ServerTag enums in the RPC wrappers, update the admin client methods to use them in their public signatures too. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Apply TableId/PartitionId/BucketId type aliases to admin methods Update the new admin method signatures introduced by this PR to use the i64/i32 type aliases from lib.rs instead of raw primitives, matching the underlying RPC message wrappers. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Address reviewer feedback round 2 - Add `RegisterProducerResult` enum mirroring Java's `RegisterResult` (Created=0, AlreadyExists=1); `register_producer_offsets` now returns `Option<RegisterProducerResult>` instead of raw `Option<i32>`. - Expose `comment: Option<&str>` on `alter_database` and the `AlterDatabaseRequest` wrapper (was hardcoded to `None`). - Replace fully-qualified paths with imports: `BucketStatsRequest` in admin.rs, `PbDatabaseSummary` in metadata/database.rs, and `AlterConfig` self-reference in metadata/table_change.rs. - Rename `create_acls(acl)` to `create_acls(acls)` and `drop_acls(acl_filter)` to `drop_acls(acl_filters)` for plural consistency with the `Vec` argument type. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: warmbupt <warmbupt@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* [rust] Add Fluss 1.x protocol support to the admin client Add 27 new admin methods to FlussAdmin: - Database/table extensions: list_database_summaries, alter_database, alter_table, get_table_stats - KV snapshot operations: get_latest_kv_snapshots, get_kv_snapshot_metadata, create_kv_snapshot_lease, get_lake_snapshot - ACL management: create_acls, list_acls, drop_acls - Cluster configuration: describe_cluster_configs, alter_cluster_configs - Server management: add_server_tag, remove_server_tag, rebalance, list_rebalance_progress, cancel_rebalance - Producer offsets: register_producer_offsets, get_producer_offsets, delete_producer_offsets - Monitoring: get_cluster_health, list_remote_log_manifests - KV snapshots: list_kv_snapshots, release_kv_snapshot_lease, drop_kv_snapshot_lease * Address reviewer feedback: add domain enums, type aliases, expose readable - Add ClusterHealthStatus enum (Green/Yellow/Red/Unknown) to cluster_health.rs - Add RebalanceStatus enum (NotStarted..Timeout) to rebalance.rs - Replace raw i32/i64 with BucketId/TableId/PartitionId aliases in kv_snapshot.rs, lake_snapshot.rs, rebalance.rs, producer_offsets.rs - Expose readable parameter in admin.get_lake_snapshot() instead of hardcoding None Addresses reviewer feedback on PR #631. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Apply type aliases to remaining metadata types (table_stats, kv_snapshot_lease) Extends the BucketId/TableId/PartitionId alias consistency fix to table_stats.rs (BucketStatsRequest, BucketStats) and kv_snapshot_lease.rs (KvSnapshotLeaseForBucket, KvSnapshotLeaseForTable). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update admin.rs to use GoalType/ServerTag enums after rebase Now that pr/3 provides GoalType and ServerTag enums in the RPC wrappers, update the admin client methods to use them in their public signatures too. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Apply TableId/PartitionId/BucketId type aliases to admin methods Update the new admin method signatures introduced by this PR to use the i64/i32 type aliases from lib.rs instead of raw primitives, matching the underlying RPC message wrappers. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * Address reviewer feedback round 2 - Add `RegisterProducerResult` enum mirroring Java's `RegisterResult` (Created=0, AlreadyExists=1); `register_producer_offsets` now returns `Option<RegisterProducerResult>` instead of raw `Option<i32>`. - Expose `comment: Option<&str>` on `alter_database` and the `AlterDatabaseRequest` wrapper (was hardcoded to `None`). - Replace fully-qualified paths with imports: `BucketStatsRequest` in admin.rs, `PbDatabaseSummary` in metadata/database.rs, and `AlterConfig` self-reference in metadata/table_change.rs. - Rename `create_acls(acl)` to `create_acls(acls)` and `drop_acls(acl_filter)` to `drop_acls(acl_filters)` for plural consistency with the `Vec` argument type. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: warmbupt <warmbupt@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

gkatzioura marked this pull request as draft March 19, 2025 07:37

polyzos changed the title ~~630~~ [Feature] Add GSFileSystem Support Mar 19, 2025

polyzos added the component=filesystem label Mar 19, 2025

polyzos changed the title ~~[Feature] Add GSFileSystem Support~~ 630 Mar 19, 2025

polyzos assigned gkatzioura Mar 19, 2025

gkatzioura marked this pull request as ready for review March 20, 2025 09:20

gkatzioura changed the title ~~630~~ [File system] Add Google Cloud Storage integration Mar 20, 2025

wuchong reviewed Apr 12, 2025

View reviewed changes

Comment thread fluss-filesystems/fluss-fs-gs/pom.xml Outdated

gkatzioura marked this pull request as draft April 14, 2025 06:27

gkatzioura marked this pull request as ready for review April 14, 2025 08:30

luoyuxia reviewed Apr 16, 2025

View reviewed changes

gkatzioura mentioned this pull request Apr 27, 2025

[File system] Enable clients access the Google File System through the FileSystem security token receiver API #801

Open

2 tasks

wuchong requested a review from luoyuxia May 10, 2025 13:30

gkatzioura requested a review from wuchong May 10, 2025 15:00

luoyuxia reviewed May 14, 2025

View reviewed changes

gkatzioura force-pushed the 630 branch 2 times, most recently from 694a42f to c465149 Compare May 30, 2025 07:56

luoyuxia force-pushed the 630 branch from 9e3b0f4 to 37324fe Compare June 3, 2025 11:57

luoyuxia approved these changes Jun 3, 2025

View reviewed changes

luoyuxia force-pushed the 630 branch 2 times, most recently from 0b8f285 to bec6304 Compare June 3, 2025 13:18

gkatzioura and others added 2 commits June 4, 2025 09:35

[filesystem/gs] Introduce google filesystem

8258a07

[filesystem/gs] Refine test and add licenses

baa3978

luoyuxia force-pushed the 630 branch from bec6304 to baa3978 Compare June 4, 2025 01:35

luoyuxia merged commit 478469d into apache:main Jun 4, 2025
2 of 4 checks passed

polyzos added this to the v0.8 milestone Jun 17, 2025

Uh oh!

Conversation

gkatzioura commented Mar 18, 2025

Purpose

Brief change log

Tests

API and Format

Documentation

Uh oh!

gkatzioura commented Mar 19, 2025

Uh oh!

gkatzioura commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wuchong commented Mar 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wuchong commented Mar 22, 2025

Uh oh!

gkatzioura commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wuchong commented Apr 4, 2025

Uh oh!

gkatzioura commented Apr 8, 2025

Uh oh!

Uh oh!

wuchong commented Apr 16, 2025

Uh oh!

wuchong commented Apr 16, 2025

Uh oh!

luoyuxia commented Apr 16, 2025

Uh oh!

luoyuxia left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wuchong commented May 10, 2025

Uh oh!

luoyuxia left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

luoyuxia May 14, 2025

Choose a reason for hiding this comment

Uh oh!

luoyuxia commented Jun 3, 2025

Uh oh!

luoyuxia left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gkatzioura commented Mar 20, 2025 •

edited

Loading

wuchong commented Mar 22, 2025 •

edited

Loading

gkatzioura commented Apr 4, 2025 •

edited

Loading