Broken screen/window capture with OBS and NVIDIA+Wayland
Apparenty OBS does not currently support NVIDIA on Wayland. They claim they cannot do anything since it only depends on other projects:
to our knowledge no xdg-desktop-portal implementation actually supports proper buffer negotiation (some pending changes for xdg-desktop-portal-gnome and mutter resulted in proper capture, but i have not followed their acceptance or rejection from those projects). So there is nothing we can track here. If you feel things should work please file a bug against your compositor's implementation of the portal.
Since we are not aware of anyone supporting the required infrastructure (xdg-desktop-portal implementations), we cannot make any claims to support nvidia.
If anyone wants to open a bug relating to nvidia on wayland please provide a link to a bug opened against your compositor or xdg-desktop-portal implementation where they claim the issue is in OBS first.
I asked in the OBS discord channel and they said:
every compositor (the 3 main ones: GNOME, KDE, wlroots) would need to officially support nvidia on wayland for us to consider officially supporting it.
and of course officially support xdg-desktop-portal on wayland on nvidia as well.
And that support would need to be for gpu buffer capture as well.
I opened an issue in xdg-desktop-portal
without knowing if it could be related to them, but they said:
xdg-desktop-portal itself is not involved in any buffer transfer between applications and screen cast sources / compositors, it only sets up the PipeWire stream itself. Actual buffer allocation/transfer/.. happen elsewhere.
With nvidia supporting gbm for buffer allocation with recent drivers, ideally nothing should be needed in the PipeWire source nor the application, but if it isn't I suggest opening a bug on the component that handles buffer allocation, and see if the issue is there, or if it is in the driver itself. What component that is depends on what desktop environment you're using.
So since I am using GNOME/Mutter, I guess this is the right place to open this issue? Would this be fixed by supporting buffer negotiation / gpu buffer capture?