swipe-tracker: Guard all swipeable access with NULL

This avoids a

GLib-GObject-FATAL-CRITICAL: g_object_set_data: assertion 'G_IS_OBJECT (object)' failed

when the weak ref is already gone.