Skip to content

fix: fall back to full index when pull/checkout targets mix .dvc files and paths - #11076

Open
adarshsm wants to merge 1 commit into
treeverse:mainfrom
adarshsm:fix/mixed-dvcfile-granular-targets
Open

fix: fall back to full index when pull/checkout targets mix .dvc files and paths#11076
adarshsm wants to merge 1 commit into
treeverse:mainfrom
adarshsm:fix/mixed-dvcfile-granular-targets

Conversation

@adarshsm

@adarshsm adarshsm commented Aug 3, 2026

Copy link
Copy Markdown

Fixes #11075.

Problem

dvc pull (and dvc checkout) with a target list that mixes a .dvc-file target with a granular path inside a tracked directory misbehaves two ways, both reported in #11075:

  • Fresh-clone state: the granular target is silently skipped — pull exits 0 but never checks it out.
  • Drifted directory (any untracked file inside the tracked dir): checkout dies with an uncaught KeyError / StorageKeyError:
    ERROR: unexpected error - ('datadir', 'f1.txt')
    

Each target form works on its own, and an all-data-path target list works too (that's the documented workaround).

Root cause

index_from_targets (dvc/repo/index.py) builds a merged per-target index when all targets are stages/.dvc files, and otherwise falls back to the full repo index. The fallback is gated on index is None — but index is also the loop variable holding each per-target index:

index: Optional[Index] = None
if targets and all(targets) and not with_deps and not recursive:
    indexes = []
    try:
        for target in targets:
            ...
            if file and not name:
                index = Index.from_file(repo, file)   # .dvc target: succeeds
            else:
                index = Index(repo, stages=list(repo.stage.collect(target)))
            indexes.append(index)
    except (StageFileDoesNotExistError, StageNotFound):
        pass                                          # <-- leaves partial `index`
    else:
        index = Index.from_indexes(repo, indexes)
        targets = None

if index is None:                                     # <-- skipped: index is partial
    index = repo.index
return index.targets_view(targets, ...)

With a mixed list like ["single.csv.dvc", "datadir/f1.txt"], the .dvc target sets index, then the granular path raises. The except does pass, leaving the partial single-target index in place, so the index is None fallback is skipped and that partial index — which knows nothing about the directory — is used with the full target list. The directory target is dropped (silent skip), and when the dir has drifted, checkout._check_can_delete looks up a key that isn't in the partial index's storage_map and raises.

All-data-path lists already work only because their first target fails immediately, so index stays None and the fallback fires.

Fix

Reset index = None in the except, so a partial parse falls back to the full repo index with the original targets — the same working path that all-data-path target lists already take. One line.

Test

Added test_pull_mixed_dvcfile_and_granular_targets (tests/func/test_data_cloud.py) covering both the silent-skip and the drift-crash cases with a local_remote. It fails on main (skip + KeyError) and passes with the fix. Verified tests/func/test_checkout.py (48), tests/func/test_repo_index.py (19), and tests/unit/repo/ (109) still pass; ruff check/format clean.

…s and paths

`index_from_targets` builds a merged per-target index when every target is a
stage or `.dvc` file, otherwise it falls back to the full repo index. The
fallback signal is `index is None`, but `index` is also the loop variable that
holds each per-target index. When a target list mixes a `.dvc`-file target
(which parses) with a granular path inside a tracked directory (which raises
`StageFileDoesNotExistError`), the loop set `index` from the `.dvc` target
before the granular target failed, so the `except` left a *partial* index in
place. The `index is None` fallback was then skipped, and that partial index —
which knows nothing about the directory — was used with the full target list.

Consequences (both from treeverse#11075):
- fresh-clone state: the granular target is silently skipped (never checked out);
- if the tracked directory has drifted, `checkout`'s `_check_can_delete` looks
  up a key absent from the partial index's `storage_map` and dies with an
  uncaught `KeyError`/`StorageKeyError`.

All-data-path target lists already worked, but only because the first target
fails immediately and leaves `index is None`. Reset `index = None` in the
`except` so a partial parse falls back to the full repo index with the original
targets, matching that working path.

Added a regression test covering both the silent-skip and the drift-crash cases.

Fixes treeverse#11075
@github-project-automation github-project-automation Bot moved this to Backlog in DVC Aug 3, 2026
@CLAassistant

CLAassistant commented Aug 3, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@codecov

codecov Bot commented Aug 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.98%. Comparing base (2431ec6) to head (eabf9ac).
⚠️ Report is 211 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #11076      +/-   ##
==========================================
+ Coverage   90.68%   90.98%   +0.30%     
==========================================
  Files         504      505       +1     
  Lines       39795    41153    +1358     
  Branches     3141     3263     +122     
==========================================
+ Hits        36087    37443    +1356     
- Misses       3042     3071      +29     
+ Partials      666      639      -27     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pull: KeyError crash (or silently skipped target) when mixing .dvc-file targets with granular paths inside a tracked directory

2 participants