Skip to content

Add s3_compatible_url_schemes setting to route additional URL schemes to the S3 filesystem#340

Open
plusplusjiajia wants to merge 1 commit into
duckdb:mainfrom
plusplusjiajia:fix-oss
Open

Add s3_compatible_url_schemes setting to route additional URL schemes to the S3 filesystem#340
plusplusjiajia wants to merge 1 commit into
duckdb:mainfrom
plusplusjiajia:fix-oss

Conversation

@plusplusjiajia

Copy link
Copy Markdown

Motivation

Many object stores are S3-compatible but use their own URL scheme (e.g. oss://, cos://, obs://). Data lake catalogs (e.g. Iceberg REST catalogs) return table locations using these schemes, which the client cannot control — DuckDB currently fails with "No such file or directory"
because no filesystem recognizes the prefix, even though the S3 filesystem could serve these requests via a custom endpoint.

httpfs already handles this pattern for specific vendors with the built-in r2:// and gcs:///gs:// schemes. This PR generalizes that into a setting instead of hardcoding more vendor schemes one by one.

Changes

  • New s3_compatible_url_schemes setting (VARCHAR, comma-separated): registered schemes are recognized by S3FileSystem::TryGetPrefix / CanHandleFile and included in the default scope of newly created S3 secrets.
  • The setting is global-only (default_scope = GLOBAL; explicit SET SESSION is rejected) — URL-to-filesystem routing is instance-level by nature.
  • Schemes of built-in filesystems (s3://, gcs://, r2://, http://, https://, hf://, file://, ...) are rejected to prevent hijacking their routing.
  • RESET restores the default (empty) behavior; covered by tests.

Default behavior is unchanged: with the setting unset, routing and secret scopes are identical to before.

Usage

SET s3_compatible_url_schemes = 'oss';
CREATE SECRET (TYPE S3, KEY_ID '...', SECRET '...', ENDPOINT 'oss-cn-hangzhou.aliyuncs.com');
SELECT * FROM 'oss://my-bucket/path/file.parquet';

Notes

- Considered a per-instance design (threading the scheme list through S3AuthParams), but CanHandleFile/TryGetPrefix are static with no context available; the process-level registry with global-only SET keeps semantics consistent and the change minimal.
- The settings list in duckdb-web should get a new entry once this lands.

@plusplusjiajia

Copy link
Copy Markdown
Author

Hi @carlopi, could you trigger CI when you get a chance? This generalizes the hardcoded r2:///gcs:// scheme handling into a s3_compatible_url_schemes setting, so S3-compatible stores like oss:// work via their native scheme. No behavior change when unset; tests included. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant