After measuring it, I realized the optimization never triggers for
Adwaita and rarely ever triggers for simple themes. So it is not
useful to keep it around.