ci(vibevoice): skip the ASR transcription e2e on release tag builds#10567
Merged
Conversation
The `tests-vibevoice-cpp-grpc-transcription` job downloads the vibevoice ASR model (`vibevoice-asr-q4_k.gguf`, ~10 GB) and decodes it through the e2e-backends harness. On release tag pushes the detect step forces the full matrix (run-all=true), so this job runs and consistently times out: the inner `go test -timeout 30m` cannot pull a 10 GB file from HuggingFace's throttled Xet CDN within budget (curl --max-time 600 x5 retries overruns the deadline), leaving an orphaned curl and a 30m panic. It has been red on every release (v4.5.3/4/5). Guard the job's `if` with `!startsWith(github.ref, 'refs/tags/')` so it no longer runs on tag/release builds. It still runs on PRs and branch pushes that touch vibevoice-cpp, so real regressions are caught off the release path. A proper fix (a small ASR test GGUF) can re-enable it on tags later. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
mudler
added a commit
that referenced
this pull request
Jul 10, 2026
#10766) The vibevoice transcription e2e hangs until the go test timeout when the HF CDN is slow: the ASR Q4_K model is >10 GB, downloadFile capped every curl attempt at --max-time 600 (needs a sustained ~17 MB/s to fit), and curl's --retry restarts from byte zero, so no attempt ever makes forward progress. This killed the job twice on PR #10764 and previously forced skipping it on release tags (#10567). Replace the wall-clock cap with stall detection (--speed-limit 1 MiB/s over --speed-time 120s) and resume from the bytes already on disk with -C -, retrying from Go because curl does not re-evaluate the resume offset on its internal retries. Resume against the HF Xet CDN was verified by killing a transfer mid-flight and confirming the next invocation appended (114 MB -> 235 MB, GGUF magic intact). Also parameterize the suite timeout (BACKEND_TEST_TIMEOUT, default 30m) and raise it to 120m for the vibevoice transcription wrapper: a 10 GB download plus 25 specs does not fit in 30m even on a good day, and the job-level GHA timeout there is already 150m. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
tests-vibevoice-cpp-grpc-transcriptiondownloads the vibevoice ASR model (vibevoice-asr-q4_k.gguf, ~10 GB — the only ASR quants published are 10.4 GB and 13.9 GB) and decodes it through the e2e-backends gRPC harness.On release tag pushes the detect step forces the full matrix (
run-all=true), so this job runs. It then consistently times out: the innergo test -timeout 30mcannot pull a 10 GB file from HuggingFace's throttled Xet CDN within budget (curl --max-time 600× 5 retries overruns the 30-min deadline), leaving an orphanedcurland apanic: test timed out after 30m0s. It has been red on every recent release (v4.5.3 / v4.5.4 / v4.5.5).What changed
Guard the job's
ifwith!startsWith(github.ref, 'refs/tags/')so it no longer runs on tag/release builds:It still runs on PRs and branch pushes that touch
vibevoice-cpp, so real regressions are caught off the release path. Only this one job is affected; the vibevoice TTS smoke/gRPC tests (smaller model) are untouched.Follow-up
A proper fix is to publish a small ASR test GGUF and point
test-extra-backend-vibevoice-cpp-transcriptionat it, then drop this guard so the test runs on tags again.Assisted-by: Claude:claude-opus-4-8 [Claude Code]