Skip to content

feat(storage): organize storage credential configs for security - #19147

Merged
bohutang merged 26 commits into
databendlabs:mainfrom
bohutang:feat/storage-disable-ec2-metadata
Dec 25, 2025
Merged

feat(storage): organize storage credential configs for security#19147
bohutang merged 26 commits into
databendlabs:mainfrom
bohutang:feat/storage-disable-ec2-metadata

Conversation

@bohutang

@bohutang bohutang commented Dec 24, 2025

Copy link
Copy Markdown
Member

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

  • Added global credential-chain controls: StorageConfig now exposes runtime-only disable_config_load and disable_instance_profile, and CredentialChainConfig replaces the previous S3-specific helper so DataOperator seeds the standardized policy for every storage operator instance.

  • Tightened stage runtime policy: StageInfo carries a non-persisted allow_credential_chain flag, init_stage_operator honors it alongside role_arn, and the system history stage is auto-opted in while other stages stay locked down unless they explicitly require the ambient chain.

For compatibility, the existing disable_credential_loader field is treated as "disallow credential chain".

Tests

  • Unit Test (cargo check)
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions Bot added the pr-feature this PR introduces a new feature to the codebase label Dec 24, 2025
@bohutang
bohutang marked this pull request as draft December 24, 2025 01:26

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/meta/proto-conv/src/config_from_to_protobuf_impl.rs
@bohutang
bohutang force-pushed the feat/storage-disable-ec2-metadata branch from 7d22460 to 726bd10 Compare December 24, 2025 03:41
@everpcpc
everpcpc force-pushed the feat/storage-disable-ec2-metadata branch from d9b0c36 to 145421c Compare December 24, 2025 04:55
@bohutang
bohutang force-pushed the feat/storage-disable-ec2-metadata branch from 145421c to cd23ebb Compare December 24, 2025 04:57
@everpcpc everpcpc changed the title feat(storage): add s3 disable_ec2_metadata feat(storage): organize credential configs for security Dec 24, 2025
@everpcpc everpcpc changed the title feat(storage): organize credential configs for security feat(storage): organize storage credential configs for security Dec 24, 2025
@everpcpc

Copy link
Copy Markdown
Member

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/common/storage/src/operator.rs
@everpcpc everpcpc added the ci-cloud Build docker image for cloud test label Dec 24, 2025
@github-actions

Copy link
Copy Markdown
Contributor

Docker Image for PR

  • tag: pr-19147-2d725c2-1766571613

note: this image tag is only available for internal use.

@everpcpc everpcpc added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Dec 24, 2025
@github-actions

Copy link
Copy Markdown
Contributor

Docker Image for PR

  • tag: pr-19147-ae89f6e-1766593278

note: this image tag is only available for internal use.

@everpcpc everpcpc added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Dec 24, 2025
@github-actions

Copy link
Copy Markdown
Contributor

Docker Image for PR

  • tag: pr-19147-1d751e5-1766603706

note: this image tag is only available for internal use.

@everpcpc

Copy link
Copy Markdown
Member

@codex review

@everpcpc
everpcpc marked this pull request as ready for review December 24, 2025 19:30
@bohutang

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/common/storage/src/operator.rs
@bohutang
bohutang merged commit ebcad91 into databendlabs:main Dec 25, 2025
185 of 192 checks passed
dantengsky added a commit to dantengsky/fuse-query that referenced this pull request Apr 15, 2026
When allow_insecure is true in storage config, enable the credential
chain so EC2 instance metadata (IMDS) can provide IAM role credentials.

This was removed in ebcad91 (databendlabs#19147), causing ATTACH TABLE to fail
with 403 when relying on EC2 IAM role without explicit AK/SK.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-cloud Build docker image for cloud test pr-feature this PR introduces a new feature to the codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants