This device may go away, which means we have a pointer to a possibly
stale object. Keep a reference here, and let future events bring this
accounting up-to-date when the mouse pointer is next updated via other
device.

Fixes: #855