Avoid enabling extra repos we don't need, and try to coalesce dnf
install commands down into fewer to avoid extra overhead.

Also install libxmlb-devel so that we don't need to build libxmlb in CI,
making it pass faster.