Skip to content

Speed up the Kinesis test suite - #1674

Open
tomazfernandes wants to merge 1 commit into
awspring:mainfrom
tomazfernandes:speed-up-kinesis-tests
Open

Speed up the Kinesis test suite#1674
tomazfernandes wants to merge 1 commit into
awspring:mainfrom
tomazfernandes:speed-up-kinesis-tests

Conversation

@tomazfernandes

@tomazfernandes tomazfernandes commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Motivation

Follow-up to #1665, which enabled parallel reactor builds and brought CI from about 15 minutes to about 9.

With modules now building concurrently, wall clock is set by the longest dependency chain. The two Kinesis modules are the slowest in the reactor. This covers spring-cloud-aws-kinesis; the binder is a separate change.

Test code only. Test counts are unchanged.

Changes

Run the integration test classes concurrently. The module had no parallel execution configured, so its classes ran one after another even though each uses its own streams, tables and consumer group. Enable concurrent classes and keep methods on the same thread.

The shared Localstack container start is now guarded, since Testcontainers does not document start() as thread safe and several @BeforeAll methods can reach it at once.

Gate stream creation. Concurrent CreateStream calls can exceed the limit for streams being created at the same time. Creation goes through a helper that allows three at once, which is why the callers now route through it. Creating one at a time was measured and costs about 50 seconds.

Shorten the KCL startup intervals. The KCL classes are single test fixtures whose time is spent in scheduler startup, and they ran with the library defaults for parent shard polling of 10 seconds, failover of 10 seconds, scheduler initialization backoff of 1 second and task backoff of 500 milliseconds. A single worker against Localstack has no fleet to coordinate with, so those intervals are latency with nothing to buy.

Result

Measured on CI against current main.

JDK Before After
17 4:33 1:46
21 4:50 1:59
24 4:34 2:00

Green on all three JDKs.

@tomazfernandes

tomazfernandes commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

@maciejwalkowiak @MatejNedic these configuration changes on Kinesis tests along with #1681 further cuts our build time by about about 2:20 off wall time, roughly 27%.

Let me know if you have any concerns.

@tomazfernandes
tomazfernandes force-pushed the speed-up-kinesis-tests branch from fb20f4d to b74a35d Compare August 9, 2026 22:10
@tomazfernandes tomazfernandes changed the title Speed up Kinesis test suites Speed up the Kinesis test suite Aug 9, 2026
The module had no parallel execution configured, so its classes ran one after
another even though each uses its own streams, tables and consumer group.
Enable concurrent classes and keep methods on the same thread.

The shared Localstack container start is now guarded, since Testcontainers does
not document start() as thread safe and several @BeforeAll methods can reach it
at once.

Concurrent CreateStream calls can also exceed the limit for streams being
created at the same time. Creation goes through a helper that allows three at
once rather than retrying: CreateStream is not idempotent and Kinesis answers
ResourceInUseException once the stream exists, so retrying a call that had
already succeeded surfaced a second error that is not retryable either. Gating
keeps the suite under the limit instead of reacting after it is hit.

The KCL classes are single test fixtures whose time is scheduler startup rather
than assertions, and they ran with the library defaults for parent shard
polling of 10 seconds, failover of 10 seconds, scheduler initialization backoff
of 1 second and task backoff of 500 milliseconds. A single worker against
Localstack has no fleet to coordinate with, so those intervals are latency with
nothing to buy.
@tomazfernandes
tomazfernandes force-pushed the speed-up-kinesis-tests branch from b74a35d to 42da4e6 Compare August 9, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant