In case we have a plugin that is spinning (looking at you
rustup), ensure that we actually exit in a reasonable amount
of time.

We still need to fix rustup, since it will spin at first use,
but this at least ensures the app will exit.