[RFC] WIP: Color management support in Mutter
Enabling color management and HDR tone mapping in the Mutter.
Feature summary
Color management & HDR tone mapping support requires color space conversion (sRGB->bt2020, bt709->2020 or other way around) and Tone Mapping for which, we need to execute the following steps in the Mutter:
- Degamma (to linearize the buffer)
- Color space conversion (BT709 -> BT2020...)
- Tone mapping (SDR -> HDR, HDR -> SDR, HDR -> HDR conversion)
- Gamma: Apply gamma/transfer function for the display (PQ/HLG using Display EDID info)
We are currently targeting the sRGB, BT609, BT709 and BT2020 color spaces conversion.
We would like to propose following solutions in Mutter Compositor: Implement color management & tone mapping policy in Mutter compositor (1. Display engine path for power savings, 2. GL shaders path)
-
Display engine color management & tone mapping path for power savings Using display engine capabilities for above color management & tone mapping support.
-
GL color management & tone mapping path a) Using Mesa driver for GL based implementation for above color management & tone mapping support and just create OpenGL/ES interface in the Clutter/Mutter. b) Or, GL shaders implementation in Clutter/Cogl inside Mutter for above color management & tone mapping support.
We would like to propose color management & HDR tone mapping in 2 parts:
- Have just the colorspace be sent as part of the Wayland color protocol; with this we can manage stuff like BT2020, but HDR will still remain unsupported.
- For HDR, we can have a separate Wayland protocol and pass the HDR metadata to the compositor from the client. Once we get the HDR metadata, Tone mapping and blending decision will be taken by compositor
For the LUT calculations and Tone mapping algorithms requiring complex mathematical functions and floating math. We would like to propose 2 different solutions:
- Proposal 1: To introduce a general kms color manager library (not limited to Intel platforms) which compositors can use for color operations.
- Proposal 2: To add these in the compositor itself or use existing gnome color library i.e. Gnome color manager. We would like to know your feedback on the above approach, which would be a better option to go with ?
Detailed description
Following steps will be carried out in order to achieve color management & HDR tone mapping using Mutter compositor:
- Using Wayland protocol extension API for colorspace & HDR tone mapping, Wayland client will attach or associate color spaces & HDR metadata with their surfaces i.e. wl_surface
- Mutter Compositor should convert all the planes or surfaces to the same color space (Mostly to Display colorspace) and blend them together. To do that compositor should have:
- A compositor policy algorithm to select the appropriate path (i.e. Display engine path or GL path) for Color management & tone mapping.
- The compositor needs to query the platform's color & HDR capabilities and sets the properties to send computed data.
We would like to understand more on the following components going ahead:
- We are seeing an on-going Wayland support for colorspace protocol API merge request in-progress and we will align with it to get merged and plan our changes accordingly: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/14
- We would need additional protocol extension for HDR so applications could attach their metadata to a buffer assuming above colorspace protocol MR hasn't incorporated this. We will work separately with the wayland-protocols project on our earlier proposal: https://patchwork.freedesktop.org/patch/291644/
- We are planning to use Gnome Videos or Photos application to write simple use cases. Do you see a better use case? Later we will want any application to be able to use HDR such as web browsers & games as well.
- What are the plans on the DRM Atomic support in Mutter compositor, should we go ahead and use existing non-atomic DRM APIs (Transactional KMS API). We can keep it hidden behind an experimental feature flag.
We would like to get feedback whether this is the correct approach to implement color management & subsequently HDR tone mapping support in Mutter. Any feedback is appreciated.
Attached RFC (PDF) contains detailed description about this feature.
How would you like it to work
- User applications (such as Wayland clients) send pixels buffers with a colorspace signature embedded (i.e. legacy sRGB or BT709 or BT2020) to wl_buffer/wl_surface.
- The Mutter Compositor receives the input buffer/surface and does on-demand color-correction from buffer colorspace to display colorspace. Example BT709 to BT2020 for HDR usecase.
Relevant links, screenshots, screencasts etc.
A new extensions for Wayland color management space protocol & HDR tone mapping will be required. With it, a Wayland client will attach or associate color spaces & HDR metadata to the surface.