cleanup: Make GDK/GTK dependency specific to X11

This way, the dependencies on GTK/GDK could be completely dropped if built with Wayland only.

Note this only makes sense once !2354 (merged) is merged

Part of #2272