mouse: Reinstate keyboard navigation in the panel

Fixes #246 (closed)

Please cherry-pick this one for gnome-3-32 (to be included in the 3.32.1 release).