daemon: Don't try Xorg as a last ditch fallback if there is no X11 support

This commit gets rid of some last ditch fallback code that is only useful when X11 support is enabled and can get in the way if X11 support is disabled.