wayland/dma-buf: Don't require modifiers support for scanout tranche

Dor Askayo requested to merge doraskayo/mutter:scanout-without-modifiers into main

From the commit message:

When building the list of formats to be sent as part of the scanout
tranche, avoid requiring modifier support by the DRM driver for
formats relying on implicit modifiers (DRM_FORMAT_MOD_INVALID).

Specifically, the previous check required the DRM driver to have
advertised some modifier support for the given format in its
IN_FORMATS KMS plane property, regardless of modifier it was. If it
hadn't, the format was left out of the list of formats to be sent
in the scanout tranche.

When no formats remained to be sent in the scanout tranche, the
tranche simply wasn't sent.

This resulted in the scanout tranche never being sent for GPUs where
modifiers aren't supported. In those cases, no formats are advertised
using the IN_FORMATS property, and thus the list of formats to be sent
in the scanout tranche remained empty.

Since Mesa doesn't use scanout-compatible buffers for native Wayland
clients unless specifically requested to do so using the "scanout"
tranche flag, it effectively means that direct scanout of native
Wayland clients wasn't supported for GPUs without modifiers support.

Sending a tranche with formats paired with the implicit modifier
(DRM_FORMAT_MOD_INVALID) is both allowed by the protocol and is
already done by default for GPUs with modifiers support, unless the
experimental support for explicit modifiers is enabled in Mutter.

So instead of requiring modifiers to be supported for each format
being evaluated for the scanout tranche, when processing formats
which rely on implicit modifiers, only check if the format in
question is supported by the DRM driver for scanout on the primary

I tested this on Firefox running with the Wayland backend, and confirmed that it enables direct scanout during video playback. I used an AMD Radeon RX 580, which doesn't have modifiers support.

CC: @jadahl, @swick, @daenzer, @rmader

Edited by Dor Askayo

Merge request reports