CI: conformance-smoke shards hit the 60-min timeout-minutes cap; slowest shard cancels and blocks merges

[proggeramlug]

The conformance-smoke gap-suite shards have timeout-minutes: 60, and the slowest shard is running right up against it — intermittently (sometimes) or consistently (some PRs) exceeding 60 min and getting cancelled. Because conformance-smoke-complete requires every shard to succeed, one cancelled shard fails the whole check, blocking merges on green code.

Evidence (this week)

PR	symptom
#6449	shard 1 cancelled at ~60 min on the original run, 2 reruns, AND a fresh commit — 4× consistently
#6452	shard 1 cancelled on original + rerun (2×)
#6444	a shard cancelled on the original run; a fresh full re-shard passed → borderline, not always-over
#6434, #6387	one shard cancelled at the cap while the other 7 passed

In every case the completed shards had zero untriaged failures — the code is fine; the shard just doesn't finish in 60 min.

Why it's not concurrency-cancellation

The cancellations happen on runs that were not superseded by a newer push (e.g. #6449's current head with nothing after it), and the durations sit at ~55–60 min (the timeout-minutes: 60 value), so these are job timeouts, not cancel-in-progress.

Cost

A rerun helps only when the shard is borderline (#6444). For a consistently-over shard (#6449) reruns never clear it, so the PR can't merge without an admin bypass. This is now a recurring per-PR tax.

Options

1. Rebalance the 8 shards — the split is by test-name hash, so a few slow tests can pile onto one shard. Balance by measured duration, or split into more shards.
2. Raise timeout-minutes for conformance-smoke (e.g. 60 → 90) as a stopgap.
3. Speed up the slow tests — CI: stop truncating gap-suite output, add per-test summary #796 (closed) noted an O(n²) normalize_output in the harness; if that's still in the diff path it inflates the slow shard.

Not blaming any single PR — this is CI-infra. Filing so it's tracked rather than absorbed as rerun friction. Related: the chronic cargo-test 120-min-cap timeouts.

[proggeramlug]

Interim fix landed in #6478 — raised the per-shard timeout-minutes 55→75, which unblocks merges immediately (the deterministic shard-1 55:18 timeout now has ~20 min headroom).

Reopening to track the durable fix: raise the shard count 8 → 12 in the conformance-smoke matrix so no single --shard N/8 slice approaches the cap as the gap suite keeps growing. The timeout bump is a headroom band-aid; re-sharding is the scalable fix (per-shard wall time drops ~33%, back into the comfortable 22-30 min range).

[proggeramlug]

The reported timeout mechanism was removed by #6478: conformance-smoke shards now have a 75-minute limit, giving the measured ~55-minute heavy shard about 20 minutes of headroom. If a shard again approaches the new ceiling, that should be tracked with fresh timing data rather than keeping the obsolete 60-minute-cap report open.