Speed up the Kinesis test suite - #1674
Open
tomazfernandes wants to merge 1 commit into
Open
Conversation
tomazfernandes
requested review from
MatejNedic and
maciejwalkowiak
as code owners
August 8, 2026 19:49
Contributor
Author
|
@maciejwalkowiak @MatejNedic these configuration changes on Kinesis tests along with #1681 further cuts our build time by about about 2:20 off wall time, roughly 27%. Let me know if you have any concerns. |
tomazfernandes
force-pushed
the
speed-up-kinesis-tests
branch
from
August 9, 2026 22:10
fb20f4d to
b74a35d
Compare
The module had no parallel execution configured, so its classes ran one after another even though each uses its own streams, tables and consumer group. Enable concurrent classes and keep methods on the same thread. The shared Localstack container start is now guarded, since Testcontainers does not document start() as thread safe and several @BeforeAll methods can reach it at once. Concurrent CreateStream calls can also exceed the limit for streams being created at the same time. Creation goes through a helper that allows three at once rather than retrying: CreateStream is not idempotent and Kinesis answers ResourceInUseException once the stream exists, so retrying a call that had already succeeded surfaced a second error that is not retryable either. Gating keeps the suite under the limit instead of reacting after it is hit. The KCL classes are single test fixtures whose time is scheduler startup rather than assertions, and they ran with the library defaults for parent shard polling of 10 seconds, failover of 10 seconds, scheduler initialization backoff of 1 second and task backoff of 500 milliseconds. A single worker against Localstack has no fleet to coordinate with, so those intervals are latency with nothing to buy.
tomazfernandes
force-pushed
the
speed-up-kinesis-tests
branch
from
August 9, 2026 22:35
b74a35d to
42da4e6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Follow-up to #1665, which enabled parallel reactor builds and brought CI from about 15 minutes to about 9.
With modules now building concurrently, wall clock is set by the longest dependency chain. The two Kinesis modules are the slowest in the reactor. This covers
spring-cloud-aws-kinesis; the binder is a separate change.Test code only. Test counts are unchanged.
Changes
Run the integration test classes concurrently. The module had no parallel execution configured, so its classes ran one after another even though each uses its own streams, tables and consumer group. Enable concurrent classes and keep methods on the same thread.
The shared Localstack container start is now guarded, since Testcontainers does not document
start()as thread safe and several@BeforeAllmethods can reach it at once.Gate stream creation. Concurrent
CreateStreamcalls can exceed the limit for streams being created at the same time. Creation goes through a helper that allows three at once, which is why the callers now route through it. Creating one at a time was measured and costs about 50 seconds.Shorten the KCL startup intervals. The KCL classes are single test fixtures whose time is spent in scheduler startup, and they ran with the library defaults for parent shard polling of 10 seconds, failover of 10 seconds, scheduler initialization backoff of 1 second and task backoff of 500 milliseconds. A single worker against Localstack has no fleet to coordinate with, so those intervals are latency with nothing to buy.
Result
Measured on CI against current main.
Green on all three JDKs.