GDM force-disables Wayland for virtual machines
It is currently impossible to use Gnome Wayland with a virtual display. The culprit appears to be the GDM udev rules file at data/61-gdm.rules.in from this repository.
The rules in there appear to be broken and always detect "gdm-machine-has-hardware-gpu" and "gdm-machine-has-hybrid-graphics", even when the machine does not have a hardware GPU or laptop hybrid graphics.
To reproduce this boot up recent a Gnome desktop, for example Fedora 37 Workstation, by any means (VM or hardware), and run "ls /run/udev". The files "gdm-machine-has-hardware-gpu" and "gdm-machine-has-hybrid-graphics" will always be there, even for example booting a QEMU virtual machine with a QXL virtual display.
Another set of rules in that file force disabled Wayland when hardware graphics coexists with virtual graphics, commented "Disable wayland in situation where we're in a guest with a virtual gpu and host passthrough gpu".
As hardware graphics is always falsely detected, this results in Wayland being force disabled just if virtual graphics is detected, preventing Wayland being used in virtual machines.