GDK does not respect XDG_SESSION_TYPE - causing issues
Description
GDK does not respect the XDG_SESSION_TYPE environmental variable when choosing its backend. The choice can be controlled using the GDK-specific variable GDK_BACKEND, but if that is not present, it tries them all and picks the first one that manages to open a display. The problem is that the wayland backend will always succeed if a "wayland-0" display exists, even if it is completely unrelated to the current session. Even if WAYLAND_DISPLAY is not set to anything, because "wayland-0" is the default.
Steps to reproduce (option one)
- Start any X11-based session. For example Gnome on Xorg.
- As the same user, start any wayland compositor. For example a windowed weston.
- In the X11 session attempt to start any GDK application.
- Surprise, the application shows window inside the Wayland session, not in the X11 session where it was started.
Steps to reproduce (option two)
This is real downstream bug: https://bugzilla.opensuse.org/show_bug.cgi?id=1091245
- Configure GDM to enable Wayland and XDMCP.
- Start GDM, observe that it uses Wayland and you can see its greeter on monitor.
- Attempt to use the XDCMP, for example by running "Xvnc -query localhost ...".
- Observe that the screen is black, no greeter is shown.
Logs show that XDMCP client was connected and X11-based greeter was started for it but gnome-session-check-accelerated crashed. Coredump analysis shows that GDK opened the Wayland display instead of X11 display, but tried to call gdk_x11_* function on it. See the above-mentioned bug for more details.
Current behavior
GDK applications ignore XDG_SESSION_TYPE and use the first backend that succeeds in opening a display, which is always wayland backend when "wayland-0" display exists.
Expected outcome
If XDG_SESSION_TYPE is "x11", GDK applications should attempt to use the X backend first. If that fails or when XDG_SESSION_TYPE is not set, then perhaps the current logic could be used.
Version information
I have observed this behavior in OpenSUSE Tumbleweed with GTK 3.22.30, but based on reading the code, it seems to be the same in current master.
Additional information
Qt respects the XDG_SESSION_TYPE (added fairly recently), see here: https://code.qt.io/cgit/qt/qtbase.git/tree/src/gui/kernel/qguiapplication.cpp?id=74305ba470f48da8b4c4e806fc714fe9f7649156#n1298