We don't need to resort unless e.g. a file changed or was added.

Additionally, for weird reason this makes the keyboard focus handling
work when deleting a file (file after deleted gets focus). This was
broken due to a relayout happening due to a size_allocate due to
a style set due to a focus change when going to/from the "are you sure"
dialog.