With the previous approach we would spend most of the time waiting for
the swapchain to be filled again because it seems the compositor takes
care of 2 images at once from time to time.

This is not visible in profiles because waiting for a frame is a
read/poll/whatever operation that does not take CPU. It's only
noticeable because the app becomes less responsive.