Displays can be opened multiple times, which caused the device manager
to try to add their devices multiple times (which gets prevented with
warnings), and then remove the devices prematurely when the
multiple-opened display gets closed the first time (which is even
worse).

Add a simple hash that keeps track of how often displays are open, and
only add/remove their devices on first open and last close.

This actually happened with gtk-inspector on the gtk3-port branch, but
there is no reason this can't also happen in stable.