fix(overlays): expand overlay files after config resolution by liunan-ms · Pull Request #256 · microsoft/azure-linux-dev-tools

liunan-ms · 2026-06-26T23:46:42Z

Summary

This updates overlay-files expansion so per-file overlay documents are loaded after component config resolution instead of during raw config file loading.

That means overlay-files can now be declared in default-component-config at the project, distro, or component-group level and inherited by every applicable component. Relative patterns are interpreted for each concrete component, so a shared default such as:

[default-component-config]
overlay-files = ["overlays/*.overlay.toml"]

can discover component-local overlay files for each component.

Behavior Changes

Allows overlay-files in default component config.
Applies inherited overlay-files to every component that receives that default.
Resolves inherited relative overlay-file globs relative to the concrete component config file, or the matched spec directory for spec-discovered components.
Treats overlay-file glob patterns with no matches as no-op instead of an error.
Clears OverlayFiles after expansion so overlays are not appended more than once.
Preserves component-level override behavior: a non-empty component overlay-files list replaces the inherited list.

Fixed #253

Copilot

Pull request overview

This pull request moves overlay-files glob expansion from raw config-file loading to the component resolution phase, enabling overlay-files to be inherited via default-component-config (project/distro/component-group) and resolved relative to each concrete component (or spec directory for spec-discovered components).

Changes:

Expand and validate per-file overlay documents during component resolution (ExpandResolvedOverlayFiles), then clear OverlayFiles to prevent double expansion.
Adjust overlay-file glob semantics: patterns with no matches are treated as a no-op instead of an error, and invalid overlays are validated with clearer per-overlay context.
Update inheritance behavior so a non-empty overlay-files list overrides inherited values, plus update schema/snapshots/docs/tests to reflect the new behavior.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
schemas/azldev.schema.json	Updates schema description to reflect post-resolution overlay-file expansion and spec-dir-relative behavior.
scenario/snapshots/TestSnapshotsContainer_config_generate-schema_stdout_1.snap	Updates generated schema snapshot description text.
scenario/snapshots/TestSnapshots_config_generate-schema_stdout_1.snap	Updates generated schema snapshot description text.
internal/projectconfig/overlay_file.go	Replaces loader-time expansion with `ExpandResolvedOverlayFiles`; changes no-match glob handling; validates overlays during overlay-file parsing.
internal/projectconfig/overlay_file_test.go	Updates and expands unit tests for new no-op semantics, per-overlay validation, and post-resolution expansion behavior.
internal/projectconfig/loader.go	Removes overlay-file expansion from raw config loading.
internal/projectconfig/component.go	Updates `OverlayFiles` doc/schema tag and changes merge semantics so non-empty overlay-files override inherited values.
internal/projectconfig/component_test.go	Adds tests covering overlay-files override vs inheritance behavior in merges.
internal/app/azldev/core/components/resolver.go	Expands overlay-files after config inheritance is applied, using an appropriate reference directory (config dir or spec dir).
internal/app/azldev/core/components/resolver_test.go	Adds resolver test ensuring spec-discovered components inherit and expand overlay-files defaults correctly.
docs/user/reference/config/overlays.md	Updates user docs for new inheritance behavior, reference-dir rules, and no-op no-match semantics.
docs/user/reference/config/components.md	Updates component config docs to reflect post-resolution expansion and inheritance semantics.

Tonisal-byte

Good work! Just one small comment

Tonisal-byte · 2026-06-29T20:39:25Z

 			return nil, fmt.Errorf("failed to scan for overlay files matching %q:\n%w", pattern, err)
 		case len(matches) == 0:
-			return nil, fmt.Errorf("%w: %q", ErrOverlayFilesNoMatches, pattern)
+			continue


Should a no-match here really be silent? We dropped ErrOverlayFilesNoMatches, so a typo'd literal path (e.g. overlasy/foo.toml) now applies nothing with no error. Could we keep tolerating real globs but still error on literal paths that match nothing, like containsPattern does for includes in loader.go?

Agree here, we should at least emit a warning, but ideally there is some way to distinguish between globs that we expect to fail, vs. "real" entries.

Thanks, fixed. overlay-files now distinguishes literal paths from glob patterns after expansion: missing literal paths return an error, while glob patterns with no matches remain a no-op.

dmcilvaney · 2026-06-29T21:40:55Z


 Set `overlay-files` on the component to one or more globs (relative to the component config) and drop one overlay file per logical change into a directory of your choosing. The conventional layout uses a sibling `overlays/` directory and a `*.overlay.toml` filename suffix, but neither is required — `overlay-files` is just a glob, so any layout you can describe with `**`/`*` patterns works.

+`overlay-files` can also be inherited from `default-component-config` at the project, distro, or component-group level. Inherited relative patterns are still resolved for each concrete component: from its component config file when it has one, or from the matched spec file's directory when it is discovered by a component group's `specs` pattern. This makes defaults useful for component-local discovery patterns such as `overlay-files = ["overlays/*.overlay.toml"]`. If a component sets a non-empty `overlay-files` list, that list replaces the inherited list; include both patterns explicitly when you want to keep default discovery and add component-specific locations.


Not your bug but probably should be fixed either here or with a fast follow PR.

Tracing the flow, I think component groups may not work. GroupsByComponent is populated from a group's Componets list. GetComponentGroupByName finds glob'd specs, but aren't written back, and applyInheritedDefaultsToComponent uses the GroupsByComponent[name] map, so a glob'd spec won't get the new overlays.

Good catch. I fixed this in the resolver by carrying dynamic component-group membership for spec-discovered components into inherited default resolution. I also found and fixed the related direct SpecPaths filter case, where a component resolved by spec path could still miss group defaults.

dmcilvaney · 2026-06-29T21:43:25Z

 			return nil, fmt.Errorf("failed to scan for overlay files matching %q:\n%w", pattern, err)
 		case len(matches) == 0:
-			return nil, fmt.Errorf("%w: %q", ErrOverlayFilesNoMatches, pattern)
+			continue


Agree here, we should at least emit a warning, but ideally there is some way to distinguish between globs that we expect to fail, vs. "real" entries.

dmcilvaney · 2026-06-29T21:46:20Z

 		return fmt.Errorf("failed to merge project info:\n%w", err)
 	}

+	if len(otherOverlayFiles) > 0 {


Should we have a way to make overlay-files = [] in a component override the inherited values? ie we don't want them for some specific package.

Fixed. overlay-files = [] now explicitly clears inherited overlay-file patterns, while an omitted value still inherits lower-priority defaults and a non-empty value replaces them.

Resolve inherited overlay-file globs only after component defaults are merged so relative patterns can be interpreted for each concrete component. This also consumes the glob list after expansion to avoid duplicate overlay loading.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Bump the azldev pin from 4b5f72c to 35d8fae, the merged form of the overlay-files-after-resolution change (microsoft/azure-linux-dev-tools#256). The merged commit incorporates the review follow-ups — notably allowing an empty overlay-files list to disable inherited patterns and clearer literal-path handling. Regenerate external/schemas/azldev.schema.json to match (overlay-files description updated). No lock drift; config loads cleanly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Pin azldev to 35d8fae562980db7bda057d79322cc51ddad8d62, the merged form of the overlay-files-after-resolution change (microsoft/azure-linux-dev-tools#256). This version supports inheriting overlay-files from a project-wide [default-component-config] and allows an empty overlay-files list to disable inherited patterns. Regenerate external/schemas/azldev.schema.json against the pinned tool so the authoritative schema knows the new metadata / overlay-files / BugRef fields. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… BEFORE MERGE] TEMPORARY: this commit exists solely so the PR checks run against an azldev build that understands the new overlay-files / metadata / BugRef config. It MUST be reverted before the PR is merged. Pin azldev to 35d8fae562980db7bda057d79322cc51ddad8d62, the merged form of the overlay-files-after-resolution change (microsoft/azure-linux-dev-tools#256). This version supports inheriting overlay-files from a project-wide [default-component-config] and allows an empty overlay-files list to disable inherited patterns. Regenerate external/schemas/azldev.schema.json against the pinned tool so the authoritative schema knows the new metadata / overlay-files / BugRef fields. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 26, 2026 23:46

Copilot started reviewing on behalf of liunan-ms June 26, 2026 23:47 View session

Copilot AI reviewed Jun 26, 2026

View reviewed changes

Comment thread internal/app/azldev/core/components/resolver.go Outdated

liunan-ms force-pushed the liunan/overlay-files-post-resolution branch from 6924435 to 4b5f72c Compare June 29, 2026 19:52

Tonisal-byte reviewed Jun 29, 2026

View reviewed changes

dmcilvaney reviewed Jun 29, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings June 30, 2026 17:12

liunan-ms force-pushed the liunan/overlay-files-post-resolution branch from 4b5f72c to 49e2840 Compare June 30, 2026 17:12

Copilot started reviewing on behalf of liunan-ms June 30, 2026 17:13 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

dmcilvaney approved these changes Jun 30, 2026

View reviewed changes

liunan-ms merged commit 35d8fae into microsoft:main Jun 30, 2026
18 checks passed

liunan-ms mentioned this pull request Jun 30, 2026

chore(comps): annotate backport overlays with category metadata microsoft/azurelinux#17727

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(overlays): expand overlay files after config resolution#256

fix(overlays): expand overlay files after config resolution#256
liunan-ms merged 1 commit into
microsoft:mainfrom
liunan-ms:liunan/overlay-files-post-resolution

liunan-ms commented Jun 26, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Tonisal-byte left a comment

Uh oh!

Tonisal-byte Jun 29, 2026

Uh oh!

dmcilvaney Jun 29, 2026

Uh oh!

liunan-ms Jun 30, 2026

Uh oh!

dmcilvaney Jun 29, 2026

Uh oh!

liunan-ms Jun 30, 2026

Uh oh!

dmcilvaney Jun 29, 2026

Uh oh!

dmcilvaney Jun 29, 2026

Uh oh!

liunan-ms Jun 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		Set `overlay-files` on the component to one or more globs (relative to the component config) and drop one overlay file per logical change into a directory of your choosing. The conventional layout uses a sibling `overlays/` directory and a `.overlay.toml` filename suffix, but neither is required — `overlay-files` is just a glob, so any layout you can describe with ``/`` patterns works.

		`overlay-files` can also be inherited from `default-component-config` at the project, distro, or component-group level. Inherited relative patterns are still resolved for each concrete component: from its component config file when it has one, or from the matched spec file's directory when it is discovered by a component group's `specs` pattern. This makes defaults useful for component-local discovery patterns such as `overlay-files = ["overlays/*.overlay.toml"]`. If a component sets a non-empty `overlay-files` list, that list replaces the inherited list; include both patterns explicitly when you want to keep default discovery and add component-specific locations.

Uh oh!

Conversation

liunan-ms commented Jun 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Tonisal-byte left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants