Since we are using g_*_source_new() API to attach the source
to non-default main contexts, we cannot use g_source_remove()
to remove its ID, but must g_source_destroy() the GSource to
detach it from the main loop.