S3 async clients (CRT and Netty) always use CONNECT tunneling with a proxy, even for plaintext http:// endpoints

[MingWangSong]

Describe the bug

When an HTTP proxy is configured, the asynchronous S3 clients (S3AsyncClient.crtBuilder() and S3AsyncClient.builder()) always establish the proxy connection with a CONNECT tunnel, even when endpointOverride uses the plaintext http:// scheme. The synchronous S3Client backed by ApacheHttpClient correctly uses forwarding mode for the same configuration.

Many corporate proxies deliberately disallow the CONNECT method (tunnels defeat traffic inspection and can be used to bypass egress controls) and only support forwarding. Against such a proxy the async S3 clients cannot be used at all, while the sync client works.

Note that the emitted request is self-contradictory: the client sends CONNECT host:80 — it knows the target is plaintext (hence port 80) yet still negotiates a tunnel as if TLS were in use.

Regression Issue

• Select this option if this issue appears to be a regression.

Not a regression — reproduced identically on 2.33.0 and 2.54.4.

Expected Behavior

With endpointOverride = http://<host> and a proxy configured, the client should use forwarding mode and send an absolute-form request, matching both curl -x http://<proxy> http://<host>/ and the SDK's own ApacheHttpClient:

GET http://<host>/<bucket>?list-type=2 HTTP/1.1

Current Behavior

The client sends:

CONNECT <host>:80 HTTP/1.1

Against a proxy that rejects CONNECT, the request fails with:

software.amazon.awssdk.core.exception.SdkClientException:
Failed to send the request: Proxy-based connection establishment failed because the CONNECT call failed

Behavior matrix observed (SDK 2.54.4, aws-crt 0.48.4) — identical configuration, only the HTTP client differs:

HTTP client	http:// endpoint	https:// endpoint
ApacheHttpClient (sync S3Client)	GET http://host/... ✅	CONNECT host:443 ✅
CRT (S3AsyncClient.crtBuilder())	CONNECT host:80 ❌	CONNECT host:443 ✅
Netty (S3AsyncClient.builder())	CONNECT host:80 ❌	CONNECT host:443 ✅

The same CRT client is also inconsistent with itself depending on whether a proxy is configured:

Scenario	endpoint	First bytes sent
CRT, no proxy	http://	GET /<bucket>?list-type=2 HTTP/1.1 (plaintext) ✅
CRT, no proxy	https://	TLS ClientHello ✅
CRT, with proxy	http://	CONNECT host:80 ❌

So the CRT client honours the endpoint scheme when connecting directly, but stops honouring it once a proxy is involved.

Reproduction Steps

1. Start a minimal proxy stub that prints the first line it receives and closes the connection:

# stub_proxy.py
import socket, sys, threading
port = int(sys.argv[1])
s = socket.socket(); s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(("127.0.0.1", port)); s.listen(8)
def handle(c):
    data = c.recv(4096)
    line = data.split(b"\r\n")[0].decode("latin1") if data else "(empty)"
    if data[:1] == b"\x16": line = "<TLS ClientHello>"
    print(f"port {port} FIRST LINE: {line}", flush=True)
    c.close()
while True:
    c, _ = s.accept(); threading.Thread(target=handle, args=(c,), daemon=True).start()

python3 stub_proxy.py 8888 &
python3 stub_proxy.py 8889 &
python3 stub_proxy.py 8890 &

2. Run each client against a plaintext endpoint:

import java.net.URI;
import software.amazon.awssdk.auth.credentials.*;
import software.amazon.awssdk.http.apache.ApacheHttpClient;
import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.*;
import software.amazon.awssdk.services.s3.crt.*;

public class ProxyModeRepro {
    static final URI ENDPOINT = URI.create("http://example-s3.invalid");   // plaintext
    static final AwsCredentialsProvider CREDS =
        StaticCredentialsProvider.create(AwsBasicCredentials.create("ak", "sk"));

    public static void main(String[] args) throws Exception {
        // CRT -> proxy on 8888
        S3AsyncClient.crtBuilder()
            .endpointOverride(ENDPOINT).region(Region.US_EAST_1).forcePathStyle(true)
            .credentialsProvider(CREDS)
            .httpConfiguration(S3CrtHttpConfiguration.builder()
                .proxyConfiguration(S3CrtProxyConfiguration.builder()
                    .scheme("http").host("127.0.0.1").port(8888).build())
                .build())
            .build()
            .listObjectsV2(r -> r.bucket("test-bucket")).exceptionally(t -> null).join();

        // Netty -> proxy on 8889
        S3AsyncClient.builder()
            .endpointOverride(ENDPOINT).region(Region.US_EAST_1).forcePathStyle(true)
            .credentialsProvider(CREDS)
            .httpClientBuilder(NettyNioAsyncHttpClient.builder()
                .proxyConfiguration(software.amazon.awssdk.http.nio.netty.ProxyConfiguration.builder()
                    .scheme("http").host("127.0.0.1").port(8889).build()))
            .build()
            .listObjectsV2(r -> r.bucket("test-bucket")).exceptionally(t -> null).join();

        // Apache (sync) -> proxy on 8890
        try {
            S3Client.builder()
                .endpointOverride(ENDPOINT).region(Region.US_EAST_1).forcePathStyle(true)
                .credentialsProvider(CREDS)
                .httpClientBuilder(ApacheHttpClient.builder()
                    .proxyConfiguration(software.amazon.awssdk.http.apache.ProxyConfiguration.builder()
                        .endpoint(URI.create("http://127.0.0.1:8890")).build()))
                .build()
                .listObjectsV2(r -> r.bucket("test-bucket"));
        } catch (Exception ignored) { }
        System.exit(0);
    }
}

3. Observed output:

port 8888 FIRST LINE: CONNECT example-s3.invalid:80 HTTP/1.1     <- CRT     (unexpected)
port 8889 FIRST LINE: CONNECT example-s3.invalid:80 HTTP/1.1     <- Netty   (unexpected)
port 8890 FIRST LINE: GET http://example-s3.invalid/test-bucket?list-type=2 HTTP/1.1   <- Apache (expected)

Possible Solution

Root cause (CRT path). aws-c-http decides the proxy connection type from whether TLS options were supplied for the main connection (proxy.h):

AWS_HPCT_HTTP_LEGACY = 0,
/* If tls options are provided (for the main connection) then treat the proxy as a tunneling proxy
   If tls options are not provided (for the main connection), then treat the proxy as a forwarding proxy */

HttpProxyOptions defaults connectionType to Legacy and CrtConfigurationUtils.resolveProxy never calls setConnectionType(...), so this rule always applies. Meanwhile S3NativeClientConfiguration creates a TlsContext unconditionally, with no regard for the endpoint scheme:

// S3NativeClientConfiguration.java
clientTlsContextOptions = TlsContextOptions.createDefaultClient()...
this.tlsContext = new TlsContext(clientTlsContextOptions);

and S3CrtAsyncHttpClient passes it to the CRT S3 client unconditionally:

// S3CrtAsyncHttpClient.java
.withTlsContext(s3NativeClientConfiguration.tlsContext())

So the main connection always "has TLS options" and Legacy always resolves to tunneling — even though the client will not actually use TLS for an http:// endpoint. In other words, "the client holds a TLS context" is being treated as "this connection uses TLS".

Notably, aws-crt-java's own generic HttpClientConnectionManager already gates this correctly by scheme:

// HttpClientConnectionManager.java
boolean useTls = HTTPS.equals(uri.getScheme());
...
useTls && tlsContext != null ? tlsContext.getNativeHandle() : 0,

Suggested fix (verified). Set the connection type explicitly when the endpoint is plaintext, in S3NativeClientConfiguration's constructor right after resolveProxy (endpointOverride is already assigned at that point):

this.proxyOptions = resolveProxy(builder.httpConfiguration.proxyConfiguration(), tlsContext).orElse(null);
if (this.proxyOptions != null && this.endpointOverride != null
        && "http".equalsIgnoreCase(this.endpointOverride.getScheme())) {
    this.proxyOptions.setConnectionType(HttpProxyConnectionType.Forwarding);
}

I verified this directly against the CRT S3Client (bypassing the SDK wrapper) with the same stub proxy:

connectionType	tlsContext passed	First line sent
Legacy (current)	yes	CONNECT host:80
Legacy	no	CONNECT host:80
Forwarding	yes	GET http://host:80/...
Forwarding	no	GET http://host:80/...

Two findings worth highlighting:

• Simply not passing the TlsContext does not help — aws-c-s3 still tunnels. The connection type has to be set explicitly.
• Forwarding combined with a non-null TlsContext works fine and does not trigger the "configuration error" mentioned in the AWS_HPCT_HTTP_FORWARD doc comment (that refers to TLS on the tunnel destination). So the fix does not need to touch tlsContext handling at all.

This requires no new public API — HttpProxyConnectionType.Forwarding and HttpProxyOptions.setConnectionType() are already public in aws-crt-java.

For the Netty client the cause is different: AwaitCloseChannelPoolMap unconditionally uses Http1TunnelConnectionPool whenever a proxy matches, and there is no forwarding implementation in netty-nio-client at all. That is a larger change and may warrant a separate issue — happy to split it out if preferred.

Minor note: in Forwarding mode CRT emits GET http://host:80/path with the explicit default port, whereas ApacheHttpClient and curl emit GET http://host/path. Both are valid absolute-form request targets per RFC 9112, but you may want to normalise this.

Additional Information/Context

Workaround for affected users: switch to the synchronous S3Client with ApacheHttpClient, which selects the proxy mode from the endpoint scheme correctly.

I'm happy to submit a PR with the fix and tests if the approach looks acceptable.

AWS Java SDK version used

2.54.4 (also reproduced on 2.33.0); software.amazon.awssdk.crt:aws-crt:0.48.4

JDK version used

openjdk version "11.0.31" 2026-04-21
OpenJDK Runtime Environment (build 11.0.31+11-post-1ubuntu1-24.04.2-Ubuntu)

Operating System and version

Ubuntu 24.04 (kernel 6.17)