We really don't want to be in the game of adding/removing weak pointers
when we can't reliably clear them (due to some design issues in GObject)
when second-degree (or more) objects are used.

So instead try to avoid using them altogether.

This might be related to #531