fix(oci): retry layer downloads on transient network errors#10579
Merged
Conversation
Installing large backend images (e.g. vLLM/vLLM-omni, several GiB) over the Web UI could fail with "failed to download layer 0: unexpected EOF" when a single connection to the registry dropped mid-stream. The whole install then failed with no recovery, and since the download is not resumable, retrying from the UI restarted from zero and usually hit the same blip again - so users saw it as a consistent, size-correlated failure (issue #10577). The registry transport already retries manifest/digest fetches via defaultRetryPredicate (GetImage/GetImageDigest), but the per-layer data stream in DownloadOCIImageTar bypassed it entirely: layer.Compressed() + xio.Copy ran exactly once. Extract the per-layer copy into downloadLayerToFile, which retries on the same transient errors (unexpected EOF, EOF, EPIPE, ECONNRESET, connection refused) with exponential backoff, truncating any partial data before each retry. Non-retryable errors and context cancellation still fail fast. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
mudler
approved these changes
Jun 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #10577.
Installing large backend images through the Web UI (e.g.
vllm/vllm-omni, which are several GiB) intermittently fails with:The reporter guessed it was a file-size threshold. It isn't - it's that a large layer takes long enough that the single connection to the registry occasionally drops mid-stream. The download is not resumable, so each retry from the UI restarts from zero and tends to hit the same blip, which is why it reads as a consistent, size-correlated failure. This has been reported more than once.
Root cause
pkg/oci/image.goalready definesdefaultRetryPredicate(matchingunexpected EOF,EOF,EPIPE,ECONNRESET,connection refused) and wires it into the registry transport for manifest/digest fetches (GetImage,GetImageDigest). But the actual layer data stream inDownloadOCIImageTarbypassed it:layer.Compressed()+xio.Copyran exactly once with no retry and no recovery.Fix
Extract the per-layer copy into
downloadLayerToFile, which retries on the same transient errors with exponential backoff, truncating any partial data before each retry. Non-retryable errors and context cancellation still fail fast. Default is 3 retries (4 attempts total).Tests
Added internal unit tests (
pkg/oci/layer_internal_test.go) with a fakev1.Layerthat drops the stream mid-transfer:Full
pkg/ocisuite passes (including the live registry pull);golangci-lint --new-from-merge-base=origin/masteris clean.Note (out of scope)
The reporter has an NVIDIA RTX 3060 but the failing image was the ROCm/hipblas variant (
...-gpu-rocm-hipblas-vllm-omni). Even with this download fix, that backend won't run on their hardware - worth a separate look at meta-backend variant selection / what the UI offered. Not addressed here.Assisted-by: Claude:claude-opus-4-8 [Claude Code]