CI: CIAWS and CIPolecat jobs chronically hit the 20-minute execution timeout

[jeremydmiller]

Problem

The CIAWS and CIPolecat GitHub Actions jobs frequently fail with
The job has exceeded the maximum execution time of 20m0s rather than a real test assertion — the
suites don't finish within the 20-minute cap. This happened across multiple PRs this cycle (e.g.
#3345, #3346, #3347), forcing manual "is this a real failure?" triage every time and blocking
clean green CI.

• CIAWS: the LocalStack-backed SNS/SQS tests (notably AmazonSnsPerTenantConnectionTests.*) are
  slow and push the job over 20m. Retries burn even more time.
• CIPolecat: the Polecat suite (incl. saga/storage tests) times out the job; individual tests
  retry but the job wall-clock exceeds 20m. (Note: on Bump critter-stack deps (JasperFx 2.24.1, Marten 9.14.0, Weasel 9.16.2, Polecat [4.8.0,6.0.0)) #3348 CIPolecat did finish and pass, so it's
  borderline — right at the cap.)

Impact

• Real failures are masked by / indistinguishable from timeouts without opening each job log.
• "Merge when green" is unachievable because these two jobs are almost always red.

Options to consider

• Raise the per-job timeout-minutes for these two jobs (quickest).
• Split the AWS suite (SNS vs SQS, or per-tenant tests) into parallel jobs.
• Speed up LocalStack startup / reuse, and the Polecat suite's per-test setup.
• Investigate whether the SNS per-tenant tests have grown disproportionately slow.

Filed as a follow-up so the release isn't gated on the CI infra timing (per maintainer direction).

[jeremydmiller]

Update (2026-07-08): CIAWS is now temporarily disabled in .github/workflows/tests.yml (commented out of the test matrix) so it stops blocking the release. The AWS tests themselves pass — verified locally the suite is not broken, it's the ~20-minute LocalStack SNS/SQS job wall-clock that exceeds the job timeout. Re-enable CIAWS as part of resolving this issue (raise its timeout, split the suite, and/or speed up LocalStack startup).

[jeremydmiller]

Split shipped in #3417. Six shards, all green on real CI, slowest is 5m9s against the 20m cap — and no job needed its timeout raised. I also removed the pre-existing one-off timeout-minutes: 30 on CIPolecat, since the split made it unnecessary. Every job is back on a uniform 20.

new job	CI time	contents
CIAWSSqs	5m09s	SQS minus the compliance batteries (106 tests)
CIAWSSqsCompliance	4m23s	the four *SendingAndReceivingCompliance batteries (93 tests)
CIAWSSns	3m29s	whole SNS project (118 tests)
CIPolecatWorkflow	5m00s	AggregateHandlerWorkflow + Bugs (20 classes)
CIPolecatSagas	4m24s	Sagas / Distribution / Subscriptions / Dcb / Publishing (19 classes)
CIPolecat	4m55s	everything else (18 classes)

Two things I got wrong in this issue

1. The SNS per-tenant tests are not the hot spot. I fingered AmazonSnsPerTenantConnectionTests as "notably slow." It is 0.5 seconds for 3 tests. The whole SNS project is 90s. There was no repeated-LocalStack-setup bug left to find — commit f302484fc (#3364) already fixed that and re-enabled CIAWS in the matrix, so the "re-enable CIAWS" step of this issue was already done on main before I started.

The actual AWS hog is SQS (6m27s locally), dominated by send_and_receive (121s for 2 tests) and end_to_end_with_named_broker (60s for 1). I was looking at the wrong project.

2. Polecat's cost is per-CLASS, not per-test. Test bodies sum to only ~173s of the project's 8m14s. The other two-thirds is per-class fixture cost — host bootstrap plus SqlServer schema. So the shards are balanced by test-class count (20/19/18), not test count. Balancing by test duration, which is what I'd have done on instinct, would have produced a badly skewed split.

That second point is the more useful one to remember: for these suites, the fixture is the cost. Adding a test to an existing class is nearly free; adding a class is not.

On not silently losing coverage

The partition is proven exhaustive against baseline TRX output — 235/235 Polecat and 199/199 SQS tests land in exactly one shard, no gaps and no overlaps. CIPolecat is deliberately defined as the negation of the other two shards' namespace lists, so a newly added namespace lands there automatically instead of quietly falling out of CI. That failure mode — a split that silently stops running things — is the one I most wanted to avoid.

Honest limits

Sharding shrinks but does not eliminate the flaky-retry cost: each retried test still pays a fresh host + broker bootstrap, so a badly flaky shard can still burn wall-clock. It just has ~15 minutes of headroom now instead of zero.

Leaving this open until #3417 merges.

[jeremydmiller]

Closed by #3417, now merged. CIAWS is back in the matrix, the AWS and Polecat suites are split across parallel shards, and every job is back on a uniform 20-minute cap with the slowest at 5m9s. CIPolecat's one-off timeout-minutes: 30 exemption is gone too.

'Merge when green' means something again.