Skip to content

drm: transform plane damage to framebuffer-space for FB_DAMAGE_CLIPS#2048

Merged
cmeissl merged 1 commit into
Smithay:masterfrom
Ph4ntomas:1651-bad-drmcompositor-damage
Jun 23, 2026
Merged

drm: transform plane damage to framebuffer-space for FB_DAMAGE_CLIPS#2048
cmeissl merged 1 commit into
Smithay:masterfrom
Ph4ntomas:1651-bad-drmcompositor-damage

Conversation

@Ph4ntomas

@Ph4ntomas Ph4ntomas commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Description

Right after rendering, DrmCompositor build a PlaneDamageClips object, which is used to populate the FB_DAMAGE_CLIPS plane property. This property may be used by drivers to repaint only the damaged part of the plane, and is expected to be a bunch of rectangles in framebuffer coordinate-space. Current implementation uses the render_damage directly, which are transformed following the output_transform. If the device driver respect the FB_DAMAGE_CLIPS and only repaint those areas, this breaks rendering for any transform but normal since the actual damage and the reported one are misaligned.

Fix: apply the output inverted transform on the damage rectangle before passing them to PlaneDamageClips::from_damage.

Fixes: #1651

Note for review

There are two things I'm not 100% sure about this PR:

  1. I went to the change with the least possible side-effect, so damages are transformed to framebuffer-space at the last possible moment. I'm unsure if the transform should be applied to the Vec returned to calling code. My gut feeling is that it shouldn't however.
    Transforming the damage in damage_bag.render_output would fix winit backend in anvil for free, but also introduce a lot of side effects (I don't think I've seen any code re-transforming the damages returned by that function), but I'd like to have someone else input on this.
  2. Size for the transform area is output_geometry.size because that's what has been used in damage computation.
    I'm unsure if it's should be changed to config.properties.dst which is the un-transformed output size (and would need an additional transform). As far as I can tell, the two are equal anyway, so output_geometry.size felt like a good fit.

EDIT: There's a second call to PlaneDamageClips for direct scanout, which might require some changes too. I haven't managed to hit that code while testing yet. I'll add a commit if I do and it's buggy.
EDIT 2: Well, still haven't hit that code, but I think element_config() does the right transformation before building the PlaneDamageClips.

Checklist

@tux-linux

Copy link
Copy Markdown

This fixed the issue we had on Quill OS with portrait mode not working. It was previously described here: niri-wm/niri#2741
Since niri does not yet seem to support the latest smithay commits, I found it was easier just making a new commit with this commit's contents on our forked branch to make things work for now.

Kind request: if everything passes your tests and requirements, please merge the PR when it's possible for you. Thanks!

@Ph4ntomas

Copy link
Copy Markdown
Contributor Author

Ping @cmeissl, IIUC you're the one dealing with most of the DRM stuff.

Do you mind having a look whenever you have some time?

@cmeissl cmeissl left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this!

Comment thread src/backend/drm/compositor/mod.rs Outdated
@Ph4ntomas Ph4ntomas force-pushed the 1651-bad-drmcompositor-damage branch 2 times, most recently from ee1ba14 to c2dd531 Compare June 21, 2026 13:46
@Ph4ntomas Ph4ntomas requested a review from cmeissl June 21, 2026 13:46
@Ph4ntomas Ph4ntomas force-pushed the 1651-bad-drmcompositor-damage branch from c2dd531 to 4d859ad Compare June 21, 2026 13:56
@Ph4ntomas Ph4ntomas marked this pull request as draft June 21, 2026 14:08
@Ph4ntomas Ph4ntomas force-pushed the 1651-bad-drmcompositor-damage branch from 4d859ad to f09a88f Compare June 21, 2026 14:12
@Ph4ntomas Ph4ntomas marked this pull request as ready for review June 21, 2026 14:13
@Ph4ntomas Ph4ntomas force-pushed the 1651-bad-drmcompositor-damage branch from f09a88f to 2817914 Compare June 21, 2026 14:15
Comment thread src/backend/drm/surface/mod.rs Outdated
Right after rendering, DrmCompositor build a PlaneDamageClips object,
which is used to populate the FB_DAMAGE_CLIPS plane property. This
property may be used by drivers to repaint only the damaged part of the
plane, and is expected to be a bunch of rectangles in framebuffer
coordinate-space.

Current implementation uses the render_damage directly, which are
transformed following the output_transform. If the device driver respect
the FB_DAMAGE_CLIPS and only repaint those areas, this breaks rendering
for any transform but normal since the actual damage and the reported
one are misaligned.

For direct scan-out, we also need to perform a transform in case the
element's underlying buffer has a rotation.

Fix: apply the output inverted transform on the damage rectangle before
passing them to PlaneDamageClips::from_damage.

Fixes: Smithay#1651
@Ph4ntomas Ph4ntomas force-pushed the 1651-bad-drmcompositor-damage branch from 2817914 to 8deb957 Compare June 23, 2026 09:39

@cmeissl cmeissl left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@cmeissl cmeissl merged commit 298ebc9 into Smithay:master Jun 23, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Output transform not applied to damage by DrmCompositor

3 participants