Apply layer mask by comparing instead of multiplying
Submitted by Raphaël Quinet
Link to original bug (#128118)
Description
First, some introduction:
When editing an image with transparent areas, there are several indirect ways to modify the alpha channel: using the eraser, adding a mask and drawing in it, creating a selection (maybe using the quick mask) and then clearing its contents, etc. Using the mask as a way to edit the opacity of the image is useful because many operations can be applied to that mask, including plug-ins, and it is easy to toggle between viewing the image and viewing the mask. Viewing the mask can be very useful because it shows precisely what areas are made transparent by the mask, without being disturbed by the features present in the image.
However, this does not work as well after the mask has been applied: even if it is possible to re-create the mask by copying it from the alpha channel, applying it to the image later will not produce the expected result because the mask and the alpha channel will be multiplied. As a result, the areas that were partially transparent before will be made more transparent even if they were not edited. This multiplicative effect is useful in some cases, but usually not when one creates the layer mask from the alpha channel.
Current (but bad) solutions:
A known but tedious workaround to this problem is to use the eraser in "anti-erase" mode and "un-erase" the partially transparent areas of the picture before applying the mask, so that the multiplicative effect is avoided. This is not an elegant solution, because it relies on a hack in the eraser. Besides, "un-erasing" large areas of the picture takes time and is error-prone.
Another solution has been proposed in bug #127930 by Pedro Gimeno: add a new option for creating the layer mask. Allow the alpha channel to be "transferred" to the layer mask. So besides creating a mask from the alpha channel, it also resets the alpha channel to full opacity, so that the problems with the multiplication will be avoided when the mask is applied later. However, this presents some problems, because the user could also decide to discard the layer mask, or to make some fully transparent areas opaque.
What happens then is that some internal data (the "color" of fully transparent pixels) is revealed to the user. The image would then contain arbitrary data, depending on how it was created or from which file format it was loaded. This is bad because it reveals some random internal data, but also because some users could start relying on this and could complain if we ever try to optimize the handling of transparent pixels in a future version of the GIMP (maybe via GEGL).
A better solution (IMNSHO):
Instead of having a special case for creating the mask, let's change the way the mask is applied to the image: use a comparison instead of a multiplication. In pseudo-code: for each pixel: { if opacity(mask) < opacity(A) then opacity(A) = opacity(mask) } So any part of the mask that has been edited to reduce its opacity will be transfered to the alpha channel. Any part that has not been edited will be left unchanged. This avoids the drawbacks of the default way to apply the layer mask (which performs the unconditional operation: opacity(A) = opacity(A) * opacity(mask)). In fact, this comparison is similar to combining grayscale layers with the "Darken Only" mode.
Advantages:
- allows easy editing of the alpha channel via the layer mask
- does not require any hacks relying on "un-erase"
- no problems if the user discards the layer mask
The code: Well, I created a patch two days ago, but I think that I was a bit too ambitious because I tried to have several ways to apply the mask, like the different modes for combining layers. But after getting lost for a while trying to propagate the right modes to the paint core, I decided that it was the wrong way to go and I started on a simpler track involving only two ways to apply the layer mask: "Apply (multiply)" and "Apply (compare)". This also made the menus simpler and easier to understand. I worked on this yesterday, but I still have to finish the patch and test it. I will attach it to this bug report as soon as possible (hopefully this evening, maybe tomorrow) but I am opening this bug report already now, in order to have a good reason to ask for the patch in bug #127930 to be reverted (even if the patch is nice, I think that it solves the wrong problem and introduces several new ones).
Version: git master