Add s3 access grants feature

[alisoleimani]

S3 access grants(https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-grants.html) is a solution to control access to s3 objects at scale. It is used by some big companies to control access to s3 objects.
We need this feature to be able to use our data lake.

[romangershgorin]

@Mytherin sorry to tag you directly, but as an active contributor to this repository, could you please help with review of this MR?

[guillesd]

@alisoleimani could you provide an example of how this would be used?

[ivanovro]

Thanks for the followup @guillesd!

@alisoleimani could you provide an example of how this would be used?

As we’re (@romangershgorin & @alisoleimani) part of the same organisation and share an interest in making this capability available, I can add some context on the use cases behind it.

Standard AWS IAM policies have practical limits on the number of statements that can be attached to a role. In large organisations, where access is managed at a fine granularity (for example, per bucket prefix or even per table mapped to S3 paths), these limits quickly become a bottleneck for scalable data access management.

S3 Access Grants address this by providing short-lived, vended credentials that are dynamically issued for specific bucket prefixes. This allows access to be managed centrally and independently of IAM role definitions, avoiding the need to constantly update or expand IAM policies as access requirements evolve.

[romangershgorin]

@alisoleimani should it be also added here to allow enabling access grants for secret? I had to add it here in duckdb-aws extension also, to allow usage like this

CREATE SECRET secret (
        TYPE s3,
        PROVIDER credential_chain,
        CHAIN process,
        S3_ACCESS_GRANTS_ENABLED true
);

[alisoleimani]

@guillesd Roman already described the use case.

@romangershgorin I rebased the code to the latest main so it must be buildable now.

To use this feature you first need to enable it, either from attributes or environment var.

SET s3_access_grants_enabled=1;

export DUCKDB_S3_ACCESS_GRANTS_ENABLED=1

Now for every read/write to S3, we first check if there is a valid s3 access grant fo that s3 prefix using your identity or not, if your identity has access to that prefix using access grant we update the credentials to what access grants returns, otherwise we use your current credentials.

The core idea is to have the same functionality as https://github.com/aws/boto3-s3-access-grants-plugin

[alisoleimani]

example from our datalake:

[alisoleimani]

@alisoleimani should it be also added here to allow enabling access grants for secret? I had to add it here in duckdb-aws extension also, to allow usage like this

CREATE SECRET secret (
        TYPE s3,
        PROVIDER credential_chain,
        CHAIN process,
        S3_ACCESS_GRANTS_ENABLED true
);

Depends if we want to add it or not, I do not see a use case for that yet, however if duckdb team asks us to add it to secret we can follow your suggestion

[Tmonster]

I need to take a longer look at this and I've left a couple of comments.

My main thought, however, is that this seems very aws specific, is there a reason you did not implement this in the duckdb-aws extension? Possibly the aws-cpp-sdk has a number of optimizations for this workflow?

[Tmonster]

I think these will be available across all DuckDB connections. So if multiple connections are using s3 access grants and secrets, it's possible they may pollute the caches, don't know how desireable that is

[alisoleimani]

Good point, how can I create cache instances per connection?

[alisoleimani]

I need to take a longer look at this and I've left a couple of comments.

My main thought, however, is that this seems very aws specific, is there a reason you did not implement this in the duckdb-aws extension? Possibly the aws-cpp-sdk has a number of optimizations for this workflow?

As I wanted this flow to happen in s3 access I needed it in s3fs, if there is another possibility to have the same functionality I am open to consider it