Be prepared for the eventuality that somebody
might reinsert the same action group without
holding any extra references on it.

This was observed as causing crashes in
gnome-builder.