imwayland: destroy objects only if it matches our manager object

Should fix one of the crash from #129.
1 job for gtk-3-22 in 16 minutes and 44 seconds (queued for 5 seconds)
Status Job ID Name Coverage
  Build
passed #19586
build

00:16:44