Otherwise, the WM cannot take a grab on the pointer device, since we'll still have the implicit grab.