OBS cannot capture fullscreen Xwayland applications
Fedora 32, Wayland session, mutter-3.36.4-1.fc32.x86_64
OBS cannot Window Capture fullscreen Xwayland applications, all it gets is a frozen frame of the last time the application wasn't fullscreen.
Works on GNOME Xorg, works on Sway, I think it has also worked on GNOME Wayland 3.34 but I don't recall. The issue affects SDL2 test apps and SDL2 games running with Xwayland backend. I was also able to reproduce it with
glxgears using the "Toggle fullscreen mode" Shell shortcut.
Steps to reproduce
It's easy to reproduce with SDL2 test apps.
- Build SDL2 with test apps.
test/testdraw2(make sure it's using Xwayland rather than SDL2's Wayland backend), set up OBS window capture to capture it.
- Press Ctrl+1 to make it fullscreen, OBS starts showing a freeze frame.
- Alternatively, just launch
Very occasionally it inexplicably works, but most of the time it causes a freeze frame. I also got Xwayland to crash doing this once.
What did you expect to happen
OBS continues capturing as normal.