Skip to content

NO-ISSUE: resolve symlinked graph root in containers-storage policy key - #6389

Closed
qxtaiba wants to merge 1 commit into
openshift:mainfrom
qxtaiba:fix-containers-storage-policy-key-symlinked-graphroot
Closed

NO-ISSUE: resolve symlinked graph root in containers-storage policy key#6389
qxtaiba wants to merge 1 commit into
openshift:mainfrom
qxtaiba:fix-containers-storage-policy-key-symlinked-graphroot

Conversation

@qxtaiba

@qxtaiba qxtaiba commented Aug 7, 2026

Copy link
Copy Markdown

- What I did

I'm running into a bug where the rpm-ostree rebase from local container storage fails with is rejected by policy when the container store's graph root is a symlink. I found that the MCD writes a temporary allow rule so rpm-ostree can rebase from local container storage under a restrictive policy. The rule is keyed on the container storage graph root, but the writer and the evaluator disagree about what that is.

generateTransportPolicyKeyForReference in pkg/daemon/rpm-ostree.go takes the graph root as the store reports it, unresolved:

$ podman system info --format '{{.Store.GraphRoot}}'
/var/lib/containers/storage

$ readlink -f /var/lib/containers/storage
/var/lib/kubelet/containers/storage

The symlink is deliberate here. The container store is relocated onto a separate disk and relocating the store is a supported configuration, so the daemon should not assume the default path.

The containers/storage path resolves the graph root with EvalSymlinks when it opens the store, so the key we write and the key PolicyConfigurationIdentity() returns differ, such that we write:

[overlay@/var/lib/containers/storage]<repo-digest>@<image-id>

Then what gets looked up:

[overlay@/var/lib/kubelet/containers/storage]<repo-digest>@<image-id>

The PolicyConfigurationNamespaces() fn derives its fallback scopes from the same resolved root, so nothing matches and evaluation reaches the policy default. The rebase fails with is rejected by policy.

This fix resolves the graph root before building the key, using the same EvalSymlinks-with-Clean fallback that containers/storage uses in expandEnvPath, so the written key matches what the evaluator looks up.

This PR fixes #6388

- How to verify it

go test ./pkg/daemon/ -run 'TestCanonicalizeGraphRoot|TestGenerateTransportPolicyKeyForReferenceResolvesGraphRoot'

- Description for the changelog
This PR resolves the container storage graph root symlink before building the transport policy key, fixing rpm-ostree rebase rejections from local storage.

Summary by CodeRabbit

  • Bug Fixes
    • Improved compatibility with symlinked container storage paths when generating transport policy keys.
    • Added a fallback for storage paths that cannot be resolved, preventing related operations from failing unexpectedly.
  • Tests
    • Added coverage for resolved and unresolved storage paths.
    • Expanded test support for configured container runtime information.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown

Walkthrough

The daemon resolves symlinked container storage graph roots before generating containers-storage policy keys. Tests cover resolved paths, fallback cleaning, and policy keys that omit the original symlink path.

Changes

Container storage policy handling

Layer / File(s) Summary
Canonicalize graph roots
pkg/daemon/rpm-ostree.go
generateTransportPolicyKeyForReference resolves graph-root symlinks and falls back to a cleaned path when resolution fails.
Validate canonical policy keys
pkg/daemon/rpm-ostree_test.go, pkg/daemon/update_test.go
Tests cover canonicalization, fallback behavior, and generated keys that use the resolved graph root. MockPodmanInterface can provide configured PodmanInfo values.

Estimated code review effort: 2 (Simple) | ~15 minutes

Suggested reviewers: dkhater-redhat, sergiordlr

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes resolving symlinked graph roots in the containers-storage policy key.
Linked Issues check ✅ Passed The changes resolve graph-root symlinks before policy-key generation and add regression tests required by issue #6388.
Out of Scope Changes check ✅ Passed All changes support graph-root canonicalization, policy-key generation, or the related regression tests.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Stable And Deterministic Test Names ✅ Passed The PR adds only static Go test names: TestCanonicalizeGraphRoot and TestGenerateTransportPolicyKeyForReferenceResolvesGraphRoot; no Ginkgo titles or run-dependent values appear.
Test Structure And Quality ✅ Passed The added tests are focused unit tests using testify, t.TempDir cleanup, and meaningful assertion messages; they perform no cluster operations or Eventually/Consistently waits.
Microshift Test Compatibility ✅ Passed The PR adds only Go unit tests (TestCanonicalizeGraphRoot and TestGenerateTransportPolicyKeyForReferenceResolvesGraphRoot); it adds no Ginkgo e2e tests or MicroShift-incompatible API references.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The PR adds only standard Go tests in pkg/daemon; no new Ginkgo e2e tests or multi-node/HA assumptions require SNO review.
Topology-Aware Scheduling Compatibility ✅ Passed The diff changes only rpm-ostree path handling and test mocks; it adds no manifests, workloads, replicas, affinity, topology spread, node selectors, tolerations, or PDBs.
Ote Binary Stdout Contract ✅ Passed The PR changes only pkg/daemon files. No OTE entry-point stdout writes were added; OTE output uses GinkgoWriter or os.Stderr, and main only formats errors.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The PR adds standard Go unit tests only; no new Ginkgo e2e tests, IPv4 assumptions, or external connectivity requirements are present.
No-Weak-Crypto ✅ Passed The diff adds path canonicalization, logging, tests, and mock data only. It introduces no weak crypto APIs, custom crypto, or secret/token comparisons; sha256 appears only in test data.
Container-Privileges ✅ Passed The patch modifies only three Go files. No added lines contain privileged, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation, or root security settings.
No-Sensitive-Data-In-Logs ✅ Passed The change logs only a storage path and symlink-resolution error at verbosity 2; no passwords, tokens, PII, session IDs, or customer data are introduced.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: qxtaiba
Once this PR has been reviewed and has the lgtm label, please assign yuqi-zhang for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 7, 2026
@openshift-ci

openshift-ci Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Hi @qxtaiba. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@qxtaiba

qxtaiba commented Aug 7, 2026

Copy link
Copy Markdown
Author

hey @sergiordlr / @dkhater-redhat – let me know what i can do to get this filed properly. not sure how/where to file an OCP issue for this, but put up a fix if you want to review/merge and/or reimplement.

i've added context in the linked issue and pr description, along with some tests to reproduce.

@qxtaiba
qxtaiba force-pushed the fix-containers-storage-policy-key-symlinked-graphroot branch from c8447ec to c9a7661 Compare August 7, 2026 20:41
@qxtaiba qxtaiba changed the title fix: resolve symlinked graph root in containers-storage policy key NOoISSUE: resolve symlinked graph root in containers-storage policy key Aug 7, 2026
@qxtaiba qxtaiba changed the title NOoISSUE: resolve symlinked graph root in containers-storage policy key NO-ISSUE: resolve symlinked graph root in containers-storage policy key Aug 7, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Aug 7, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@qxtaiba: This pull request explicitly references no jira issue.

Details

In response to this:

- What I did

generateTransportPolicyKeyForReference builds the containers-storage policy key from the graph root reported by podman system info. That value comes straight out of storage.conf and is not resolved.

The policy evaluator uses a different string. containers/storage puts the configured graphroot through filepath.EvalSymlinks when it opens the store, so PolicyConfigurationIdentity() hands back the resolved path.

If the store is a symlink the two don't match:

written:   [overlay@/var/lib/containers/storage]<repo-digest>@<image-id>
looked up: [overlay@/var/lib/kubelet/containers/storage+...]<repo-digest>@<image-id>

The fallback scopes from PolicyConfigurationNamespaces() are built off the same resolved path, so there's nothing the written rule can match against. Evaluation hits the policy default, and if that default is reject the rebase dies.

The fix in this PR resolves the graph root before building the key, using the same EvalSymlinks with a Clean fallback that containers/storage uses in expandEnvPath.

Fixes: #6388

- How to verify it

go test ./pkg/daemon/ -run 'TestCanonicalizeGraphRoot|TestGenerateTransportPolicyKeyForReferenceResolvesGraphRoot'

Both fail before the change.

On a cluster you need three things at once: /var/lib/containers/storage symlinked elsewhere, a policy.json with a restrictive default, and a PinnedImageSet on the pool so MCD takes the containers-storage rebase path rather than the registry one.

- Description for the changelog

Fix rpm-ostree rebase from local container storage failing with is rejected by policy when the container storage graph root is a symlink.

Summary by CodeRabbit

  • Bug Fixes
  • Improved compatibility with container storage paths that use symbolic links.
  • Ensured transport policy handling uses canonical storage paths when available.
  • Added fallback behavior for paths that cannot be resolved, preventing invalid path data from being included.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@qxtaiba
qxtaiba force-pushed the fix-containers-storage-policy-key-symlinked-graphroot branch from c9a7661 to 64f291a Compare August 7, 2026 22:44
@qxtaiba

qxtaiba commented Aug 8, 2026

Copy link
Copy Markdown
Author

@coderabbitai help

@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

CodeRabbit commands

These commands are invoked using PR/Issue comments.

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai rate limit to show your current review rate limit status.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai evaluate custom pre-merge check --instructions <custom-checks-instructions> --name <custom-checks-title> [--mode <error|warning>] to validate the custom pre-merge checks instructions. Defaults to error when --mode is omitted.
  • @coderabbitai ignore pre-merge checks to override pre-merge checks and get an approval on PR.
  • @coderabbitai run pre-merge checks to run pre-merge checks on the pull request.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai generate unit tests to generate unit tests for this PR.
  • @coderabbitai emit path instructions to emit generated path-specific review instructions for this repository.
  • @coderabbitai resolve merge conflict to automatically resolve merge conflicts.
  • @coderabbitai autofix to automatically fix issues identified in unresolved review comments.
  • @coderabbitai fix-ci to automatically fix failing CI checks in a stacked pull request.
  • @coderabbitai fix-ci commit to automatically fix failing CI checks by committing fixes to the current branch.
  • @coderabbitai resolve to resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai generate configuration to create a PR that adds the current resolved configuration as .coderabbit.yaml (or show it if already present).
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore or @coderabbit ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Status, support, documentation and community

  • Visit our status page to check the current availability of CodeRabbit.
  • Create a ticket on our support page for assistance with any issues or questions.
  • Visit our documentation site for detailed information on how to use CodeRabbit.
  • Join our Discord community to connect with other users and get help from the community.
  • Follow us on X/Twitter for updates and announcements.

@qxtaiba

qxtaiba commented Aug 8, 2026

Copy link
Copy Markdown
Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown
✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/daemon/rpm-ostree.go`:
- Line 334: Update the fallback warning in the graph-root resolution flow to
avoid logging raw graphRoot and err values, which may expose host-specific data.
Keep the diagnostic message generic or pass values through the project’s
approved redaction helper before calling klog.V(2).Infof.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 50c6e656-c213-4a45-9b49-8a933d14c491

📥 Commits

Reviewing files that changed from the base of the PR and between f6aefa6 and 64f291a.

📒 Files selected for processing (3)
  • pkg/daemon/rpm-ostree.go
  • pkg/daemon/rpm-ostree_test.go
  • pkg/daemon/update_test.go

Comment thread pkg/daemon/rpm-ostree.go
func canonicalizeGraphRoot(graphRoot string) string {
resolved, err := filepath.EvalSymlinks(graphRoot)
if err != nil {
klog.V(2).Infof("Could not resolve container storage graph root %q, falling back to a cleaned path: %v", graphRoot, err)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

Do not log the raw graph root or filesystem error.

graphRoot and err can contain user-specific storage paths, usernames, or other host data. Keep this diagnostic message generic or pass values through an approved redaction helper.

Proposed fix
-		klog.V(2).Infof("Could not resolve container storage graph root %q, falling back to a cleaned path: %v", graphRoot, err)
+		klog.V(2).Infof("Could not resolve container storage graph root; falling back to a cleaned path")

As per coding guidelines: **/*.{js,ts,go,java,py,rb,php,cs,cpp,c,sh}: Flag logging that may expose passwords, tokens, API keys, PII (email, SSN, credit card), session IDs, internal hostnames, or customer data.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
klog.V(2).Infof("Could not resolve container storage graph root %q, falling back to a cleaned path: %v", graphRoot, err)
klog.V(2).Infof("Could not resolve container storage graph root; falling back to a cleaned path")
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/daemon/rpm-ostree.go` at line 334, Update the fallback warning in the
graph-root resolution flow to avoid logging raw graphRoot and err values, which
may expose host-specific data. Keep the diagnostic message generic or pass
values through the project’s approved redaction helper before calling
klog.V(2).Infof.

Source: Coding guidelines

@qxtaiba qxtaiba closed this Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

containers-storage policy key uses the unresolved graph root, so rebase fails when the store is a symlink

2 participants