Is there a way to use an impersonated service account with GCS?

[lrntct]

Hi,

Thanks for this tool!

I am trying to connect to a GCP bucket using an impersonated service account and get the following error.
I tried to pass a callback function with the credential_provider argument, but get the same error.
Is it a limitation of obstore, is there a work-around?

obstore.exceptions.GenericError: Generic GCS error: Unable to decode service account file: unknown variant `impersonated_service_account`, expected `service_account` or `authorized_user` at line 12 column 40
Debug source:
Generic {
    store: "GCS",
    source: DecodeCredentials {
        source: Error("unknown variant `impersonated_service_account`, expected `service_account` or `authorized_user`", line: 12, column: 40),
    },
}

[kylebarron]

I don't know anything specific about Google's impersonated service accounts; I haven't used those myself.

But the credential provider mechanism allows for any arbitrary credential handling. So if there's a way to authenticate with google.auth, then it should work. Keep in mind that if you're passing in a value of credential_provider then you don't have to pass in any other auth parameters

You haven't shared any code; can you share how you're creating the store?

[lrntct]

Here's a basic way to reproduce the issue:

import environ
from obstore.store import GCSStore

env = environ.Env()
env.read_env(".env")

bucket_name = env("BUCKET_USER_DATA")

gcs_store = GCSStore(bucket_name)

And the traceback:

 gcs_store = GCSStore(bucket_name)
                ^^^^^^^^^^^^^^^^^^^^^
obstore.exceptions.GenericError: Generic GCS error: Unable to decode service account file: unknown variant `impersonated_service_account`, expected `service_account` or `authorized_user` at line 12 column 40

Debug source:
Generic {
    store: "GCS",
    source: DecodeCredentials {
        source: Error("unknown variant `impersonated_service_account`, expected `service_account` or `authorized_user`", line: 12, column: 40),
    },
}

Proving that the access works with the google python package, and passing credential_provider explicitly:

import environ
from google.auth import default as google_auth_default
from google.cloud import storage
from obstore.store import GCSStore

env = environ.Env()
env.read_env(".env")

bucket_name = env("BUCKET_USER_DATA")


def get_gcp_credentials():
    credentials, project = google_auth_default()
    return credentials


# The bucket is accessible through Google
# This also works without passing the credentials explicitly
storage_client = storage.Client(credentials=get_gcp_credentials())  
bucket = storage_client.bucket(bucket_name)
blobs = bucket.list_blobs()

# Obstore
gcs_store = GCSStore(bucket_name, credential_provider=get_gcp_credentials)

The error persists:

gcs_store = GCSStore(bucket_name, credential_provider=get_gcp_credentials)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
obstore.exceptions.GenericError: Generic GCS error: Unable to decode service account file: unknown variant `impersonated_service_account`, expected `service_account` or `authorized_user` at line 12 column 40

Debug source:
Generic {
    store: "GCS",
    source: DecodeCredentials {
        source: Error("unknown variant `impersonated_service_account`, expected `service_account` or `authorized_user`", line: 12, column: 40),
    },
}

[kylebarron]

This isn't reproducible because I don't know what is being fetched from your environment.

You're hitting this error
https://github.com/apache/arrow-rs-object-store/blob/12ef9bc22c56228b25f3b56a78879e9a45a07fc4/src/gcp/credential.rs#L64-L65

which means that the credentials file is getting fetched and read
https://github.com/apache/arrow-rs-object-store/blob/12ef9bc22c56228b25f3b56a78879e9a45a07fc4/src/gcp/credential.rs#L314-L324

Which means that you're still passing in a path to a credentials file. If you're using the credential provider, you shouldn't be passing in other credential information, otherwise obstore will try to use native authentication. See the auth doc: https://developmentseed.org/obstore/latest/authentication/

[lrntct]

I'm creating a local ADC file following the instructions from Google's documentation
gcloud auth application-default login --impersonate-service-account SERVICE_ACCT_EMAIL

I'll try the link you gave me.

[lrntct]

If I pass a GoogleCredentialProvider I still get the same error:

import environ
from google.auth import default as google_auth_default
from google.cloud import storage
from obstore.auth.google import GoogleCredentialProvider
from obstore.store import GCSStore

env = environ.Env()
env.read_env(".env")

bucket_name = env("BUCKET_USER_DATA")

def get_gcp_credentials():
    credentials, project = google_auth_default()
    return credentials

credential_provider = GoogleCredentialProvider(credentials=get_gcp_credentials())

# The bucket is accessible through Google
# This also works without passing the credentials explicitly
storage_client = storage.Client(credentials=get_gcp_credentials())
bucket = storage_client.bucket(bucket_name)
blobs = bucket.list_blobs()

# Obstore
gcs_store = GCSStore(bucket_name, credential_provider=credential_provider)

[lrntct]

If this helps:
print(get_gcp_credentials()) gives me:
<google.auth.impersonated_credentials.Credentials object at 0x7e4f16fca870>

[kylebarron]

If I pass a GoogleCredentialProvider I still get the same error:

You still have other relevant keys in your environment that means that obstore is trying to do its own native authentication.

The way to test this is to call

credential_provider = GoogleCredentialProvider(credentials=get_gcp_credentials())
credentials = credential_provider()

If that works, then the credential provider is working.

You need to make sure you have no values defined here in your environment to make sure that only the credential provider is used.