When going back to flags or any emoji category the icons keeps growing in the list. Flags is the easy one as only 4 flags, then 8 flags then 12 flags and so on.
Click flags then any category, then click back to flags.