fix: fall back to full index when pull/checkout targets mix .dvc files and paths - #11076
Open
adarshsm wants to merge 1 commit into
Open
fix: fall back to full index when pull/checkout targets mix .dvc files and paths#11076adarshsm wants to merge 1 commit into
adarshsm wants to merge 1 commit into
Conversation
…s and paths `index_from_targets` builds a merged per-target index when every target is a stage or `.dvc` file, otherwise it falls back to the full repo index. The fallback signal is `index is None`, but `index` is also the loop variable that holds each per-target index. When a target list mixes a `.dvc`-file target (which parses) with a granular path inside a tracked directory (which raises `StageFileDoesNotExistError`), the loop set `index` from the `.dvc` target before the granular target failed, so the `except` left a *partial* index in place. The `index is None` fallback was then skipped, and that partial index — which knows nothing about the directory — was used with the full target list. Consequences (both from treeverse#11075): - fresh-clone state: the granular target is silently skipped (never checked out); - if the tracked directory has drifted, `checkout`'s `_check_can_delete` looks up a key absent from the partial index's `storage_map` and dies with an uncaught `KeyError`/`StorageKeyError`. All-data-path target lists already worked, but only because the first target fails immediately and leaves `index is None`. Reset `index = None` in the `except` so a partial parse falls back to the full repo index with the original targets, matching that working path. Added a regression test covering both the silent-skip and the drift-crash cases. Fixes treeverse#11075
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #11076 +/- ##
==========================================
+ Coverage 90.68% 90.98% +0.30%
==========================================
Files 504 505 +1
Lines 39795 41153 +1358
Branches 3141 3263 +122
==========================================
+ Hits 36087 37443 +1356
- Misses 3042 3071 +29
+ Partials 666 639 -27 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #11075.
Problem
dvc pull(anddvc checkout) with a target list that mixes a.dvc-file target with a granular path inside a tracked directory misbehaves two ways, both reported in #11075:KeyError/StorageKeyError:Each target form works on its own, and an all-data-path target list works too (that's the documented workaround).
Root cause
index_from_targets(dvc/repo/index.py) builds a merged per-target index when all targets are stages/.dvcfiles, and otherwise falls back to the full repo index. The fallback is gated onindex is None— butindexis also the loop variable holding each per-target index:With a mixed list like
["single.csv.dvc", "datadir/f1.txt"], the.dvctarget setsindex, then the granular path raises. Theexceptdoespass, leaving the partial single-target index in place, so theindex is Nonefallback is skipped and that partial index — which knows nothing about the directory — is used with the full target list. The directory target is dropped (silent skip), and when the dir has drifted,checkout._check_can_deletelooks up a key that isn't in the partial index'sstorage_mapand raises.All-data-path lists already work only because their first target fails immediately, so
indexstaysNoneand the fallback fires.Fix
Reset
index = Nonein theexcept, so a partial parse falls back to the full repo index with the original targets — the same working path that all-data-path target lists already take. One line.Test
Added
test_pull_mixed_dvcfile_and_granular_targets(tests/func/test_data_cloud.py) covering both the silent-skip and the drift-crash cases with alocal_remote. It fails onmain(skip +KeyError) and passes with the fix. Verifiedtests/func/test_checkout.py(48),tests/func/test_repo_index.py(19), andtests/unit/repo/(109) still pass;ruff check/formatclean.