backends/x11: Fix use after free on device removal

The devices_by_id hash table is responsible for managing the reference to the devices. In remove_device however, for non-core devices there are additional calls to dispose/unref, after the last reference has already been dropped by the hash table.