Only add margins to direct children of toolbar items, this fixes the gaps between linked buttons inside custom toolbar items.
Example of the problem: