tiledbsoma.open() on S3 hangs with no error when credentials come from an AWS SSO profile (aws sso login)

[royassis]

What happened

Opening a SOMA object on S3 from a shell authenticated via aws sso login (an SSO profile only — no static keys, no AWS_* env vars). All of the following are from the same shell:

• aws s3 ls s3://<bucket>/<prefix>/ (AWS CLI) → succeeds — the data is reachable and the SSO creds are valid for S3
• boto3 reads from S3 → succeeds
• tiledbsoma.open(...) → hangs (no return, no error)
• Setting valid AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN (e.g. via aws configure export-credentials --format env) → the same open() returns immediately
• With a stale / invalid static key instead → open() returns quickly with 403 InvalidAccessKeyId

Things we tried that did not change the hang:

• Refreshing the SSO session (running aws sso login again)
• vfs.s3.connect_timeout_ms / vfs.s3.request_timeout_ms = 5000
• AWS_EC2_METADATA_DISABLED=true

Repro

import tiledbsoma

# Shell authenticated via `aws sso login` (SSO profile only; no AWS_* env vars).
# `aws s3 ls s3://<bucket>/<prefix>/` succeeds in this same shell (data is reachable).
ctx = tiledbsoma.SOMATileDBContext()
tiledbsoma.open("s3://<private-bucket>/<prefix>/", soma_type="SOMAExperiment", context=ctx)
# -> hangs (no return, no error)

Environment

• tiledbsoma 2.3.0 (latest on PyPI)
• OS: Linux (Ubuntu on WSL2)
• Python 3.12 and 3.13
• Region: us-east-1

Question

Is a hang expected when credentials can't be resolved from the chain, or should open() surface an error here? Happy to share more detail or test a patch.

Filed with AI assistance (Claude), based on hands-on debugging; behavior reproduced manually.

[aaronwolen]

Thanks for the detailed report and for isolating this so cleanly.

I suspect this issue can be traced to TileDB Embedded, which bundles its own copy of the AWS SDK for C++, separate from the boto3/botocore chain your AWS CLI and Python examples use. The timeout parameters you tried (vfs.s3.connect_timeout_ms, vfs.s3.request_timeout_ms) didn't help because they control individual S3 request timeouts, not the credential resolution step that happens before any S3 connection is opened.

I'll need to discuss this with the storage engine team to understand the best approach for resolving this issue. For now, exporting your SSO session credentials as environment variables is probably the most reliable workaround (as you noted).

[royassis]

Thanks @aaronwolen appreciate it

[aaronwolen]

Just to double check we're on the right track, could you try running the following script in the same env where you encountered the hang? It only requires tiledb-py (not tiledbsoma).

"""
Run in a shell authenticated via `aws sso login`, with no AWS_* env vars set.
"""
import signal
import sys
import tiledb

S3_URI = "s3://<your-bucket>/<your-prefix>/"  # replace with the same URI you used before
TIMEOUT_SECS = 10


def _timeout(signum, frame):
    print(f"\nTIMEOUT after {TIMEOUT_SECS}s -- credential resolution hung")
    sys.exit(1)


signal.signal(signal.SIGALRM, _timeout)

print(f"tiledb version: {tiledb.version()}")
print(f"Target URI:     {S3_URI}")

ctx = tiledb.Ctx()
vfs = tiledb.VFS(ctx=ctx)

print(f"Calling vfs.ls() -- will timeout after {TIMEOUT_SECS}s ...")
signal.alarm(TIMEOUT_SECS)
try:
    entries = list(vfs.ls(S3_URI))
    signal.alarm(0)
    print(f"Success: {len(entries)} entries")
except Exception as exc:
    signal.alarm(0)
    print(f"Error: {exc}")

[royassis]

roya@LP-0267:/tmp$ uv run --no-project --python=3.13 --with "tiledb" python
Installed 3 packages in 97ms
Python 3.13.0 (main, Oct 16 2024, 03:23:02) [Clang 18.1.8 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import signal
... import sys
... import tiledb
...
... S3_URI = "s3://<redacted>"  # replace with the same URI you used before
... TIMEOUT_SECS = 10
...
...
... def _timeout(signum, frame):
...     print(f"\nTIMEOUT after {TIMEOUT_SECS}s -- credential resolution hung")
...     sys.exit(1)
...
...
... signal.signal(signal.SIGALRM, _timeout)
...
... print(f"tiledb version: {tiledb.version()}")
... print(f"Target URI:     {S3_URI}")
...
... ctx = tiledb.Ctx()
... vfs = tiledb.VFS(ctx=ctx)
...
... print(f"Calling vfs.ls() -- will timeout after {TIMEOUT_SECS}s ...")
... signal.alarm(TIMEOUT_SECS)
... try:
...     entries = list(vfs.ls(S3_URI))
...     signal.alarm(0)
...     print(f"Success: {len(entries)} entries")
... except Exception as exc:
...     signal.alarm(0)
...     print(f"Error: {exc}")
...
tiledb version: (0, 36, 1)
Target URI:     s3://<redacted>
Calling vfs.ls() -- will timeout after 10s ...
9
Error: S3: Error while listing with prefix 's3://<redacted>' and delimiter '/'[Error Type: 23] [HTTP Response Code: 403] [Exception: InvalidAccessKeyId] [Remote IP: 16.15.229.54] [Request ID: M1HRN6VE4PE02B8G] [Headers: 'content-type' = 'application/xml' 'date' = 'Wed, 24 Jun 2026 19:21:58 GMT' 'server' = 'AmazonS3' 'transfer-encoding' = 'chunked' 'x-amz-bucket-region' = 'us-east-1' 'x-amz-id-2' = '<redacted>' 'x-amz-request-id' = 'M1HRN6VE4PE02B8G'] : The AWS Access Key Id you provided does not exist in our records.

[aaronwolen]

Thanks, that's a really useful data point. The InvalidAccessKeyId means the bundled AWS SDK did resolve a credential, just the wrong one.

Would you mind running again with trace logging on and email me the two log files it produces?

I also pinned tiledb to 0.36.0 so it matches the same version of tiledb core used by tiledbsoma v2.3.0:

uv run --no-project --python=3.13 --with "tiledb==0.36.0" python repro.py

# repro.py
import signal, sys, tiledb

S3_URI = "s3://<your-bucket>/<your-prefix>/"
TIMEOUT_SECS = 10

config = tiledb.Config({
    "vfs.s3.logging_level": "TRACE", 
    "config.logging_level": "5",
})

def _timeout(signum, frame):
    print(f"\nTIMEOUT after {TIMEOUT_SECS}s -- credential resolution hung")
    sys.exit(1)

signal.signal(signal.SIGALRM, _timeout)
print(f"tiledb version: {tiledb.version()}")

vfs = tiledb.VFS(ctx=tiledb.Ctx(config))
signal.alarm(TIMEOUT_SECS)
try:
    entries = list(vfs.ls(S3_URI))
    signal.alarm(0)
    print(f"Success: {len(entries)} entries")
except Exception as exc:
    signal.alarm(0)
    print(f"Error: {exc}")

This should produce aws_sdk_<date>.log and tiledb_s3_<date>.log in your current directory. Thanks again for the assistance!

[royassis]

Hey @aaronwolen mailed those to you

[aaronwolen]

Thanks for sharing! I think we've pinned down the issue. It looks like the SSO itself is working fine (the bundled SDK reads your cached SSO token and builds the correct request) but there's a TLS problem related to the CA certificate bundle.

error setting certificate file: /etc/pki/tls/certs/ca-bundle.crt
curlCode: 77, Problem with the SSL CA cert (path? access rights?)

Could you try setting these two environment variables to point at your system CA bundle, then re-run your original tiledbsoma.open() call (no other changes needed):

export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
export AWS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

If that works then you should be unblocked and it confirms our diagnosis, which will require a bug fix in the core C++ engine.