GDK does not respect XDG_SESSION_TYPE - causing issues
GDK does not respect the XDG_SESSION_TYPE environmental variable when choosing its backend. The choice can be controlled using the GDK-specific variable GDK_BACKEND, but if that is not present, it tries them all and picks the first one that manages to open a display. The problem is that the wayland backend will always succeed if a "wayland-0" display exists, even if it is completely unrelated to the current session. Even if WAYLAND_DISPLAY is not set to anything, because "wayland-0" is the default.
Steps to reproduce (option one)
- Start any X11-based session. For example Gnome on Xorg.
- As the same user, start any wayland compositor. For example a windowed weston.
- In the X11 session attempt to start any GDK application.
- Surprise, the application shows window inside the Wayland session, not in the X11 session where it was started.
Steps to reproduce (option two)
This is real downstream bug: https://bugzilla.opensuse.org/show_bug.cgi?id=1091245
- Configure GDM to enable Wayland and XDMCP.
- Start GDM, observe that it uses Wayland and you can see its greeter on monitor.
- Attempt to use the XDCMP, for example by running "Xvnc -query localhost ...".
- Observe that the screen is black, no greeter is shown.
Logs show that XDMCP client was connected and X11-based greeter was started for it but gnome-session-check-accelerated crashed. Coredump analysis shows that GDK opened the Wayland display instead of X11 display, but tried to call gdk_x11_* function on it. See the above-mentioned bug for more details.
GDK applications ignore XDG_SESSION_TYPE and use the first backend that succeeds in opening a display, which is always wayland backend when "wayland-0" display exists.
If XDG_SESSION_TYPE is "x11", GDK applications should attempt to use the X backend first. If that fails or when XDG_SESSION_TYPE is not set, then perhaps the current logic could be used.
I have observed this behavior in OpenSUSE Tumbleweed with GTK 3.22.30, but based on reading the code, it seems to be the same in current master.
Qt respects the XDG_SESSION_TYPE (added fairly recently), see here: https://code.qt.io/cgit/qt/qtbase.git/tree/src/gui/kernel/qguiapplication.cpp?id=74305ba470f48da8b4c4e806fc714fe9f7649156#n1298