That may happen separately from grab-notify, and also due to external
reasons, so ensure all sequences are cancelled if a grab is taken
in some GdkWindows that would obscure events on the controller.