Mutter does not notice failed mode set
Scenario: GNOME Shell is in use inside a VirtualBox virtual machine with too little video RAM allocated. The user switches the virtual machine window to full-screen on the host. For GNOME Shell inside the virtual machine, this looks like a new monitor has been hotplugged into the virtual machine graphics device with a preferred resolution the size of the host screen. GNOME Shell tries to switch to this resolution, but the mode set fails for lack of video memory. As far as I can see, Mutter does not notice that the mode set failed and tries to use the new mode anyway.
Desired behaviour: Mutter notices the mode set failure and continues to use the old mode. Caveats: I am seeing this from the perspective of the driver in the kernel and of VirtualBox. I am assuming that this makes sense in user-space too, and that it would make sense on non-virtual hardware. I am also assuming that I have understood correctly what Mutter is doing.