`gdk_surface_get_device_position: assertion 'GDK_IS_SURFACE (surface)' failed` on popover close

Steps to reproduce

1. Create a GtkPopover that's opened on left or right click, and is unparented on close
2. Close the popover with popover.popdown()

Current behavior

(gnome-sudoku:51782): Gdk-CRITICAL **: 09:49:08.899: gdk_surface_get_device_position: assertion 'GDK_IS_SURFACE (surface)' failed

Expected outcome

The popover would close without any critical errors

Version information

Gtk 4.11.3, Wayland backend, on Fedora Silverblue 38

Additional information

A backtrace can be found at gnome-sudoku!54 (comment 1749189)