app: move all special-case mode processing optimizations to GimpOperationLayerMode
Stuff like passing "input" directly if "aux"'s opacity is 0, etc. Used to be partly handled by normal mode, even though it applies to other modes too. Adjust the logic for the new compositing modes. Add a GimpLayerModeAffectMask enum, and a corresponding get_affect_mask() function to GimpOperationLayerMode, which specifies which of the op's inputs, if any, are affected by the mode, apart from the overlapping regions. Most modes affect only the overlapping regions, but dissolve and replace also affect the rest of the input. This information is used for determining if the optimizations are applicable.
Showing with 206 additions and 88 deletions