Integrate with https://openqa.gnome.org/ to automatically test the iso-installer image using OpenQA and KVM.
Testing is driven by Gitlab CI, tests execute on the usual gnome-build-meta runner. (This is an unconventional OpenQA setup, see below).
Test scripts are stored in this repo. Needles (screenshot data used by tests) are in a separate repo. (OpenQA has limited flexibility about this, see below for details.)
Tests must be manually started. We would like to run on every commit, #396 (slow iso-installer builds) is the blocker.
Tests are expected to run against 'master', primarily to guard against regressions.
If you're a maintainer of gnome-build-meta and want admin access to https://openqa.gnome.org, first log in to the OpenQA server, then ask in this MR and someone can set up your account permissions.
Connecting Gitlab CI to OpenQA
OpenQA is designed to work with a bank of dedicated test machines.
Gnome-build-meta uses transient Gitlab CI runners. To attach the transient
CI runner to the OpenQA server as a worker, the
does the following:
- Define an OpenQA WORKER_CLASS which is unique to the current job
- Create a machine in openqa.gnome.org with this class.
/run_openqa_worker.sh, which connects as this machine.
- Trigger the OpenQA test job and set the same WORKER_CLASS, so the server must now assign the job to our local worker.
- When the test finishes, delete the OpenQA machine.
Limitations of current setup
OpenQA server expects to read tests and needles from the root of two Git repositories. The URL and ref is configured server-side and is shared for all branches and versions.
We have (ab)used
CASEDIR setting so the worker reads the test cases
from the local gnome-build-meta.git checkout. The OpenQA server cannot
see these, meaning links in the web UI to the testcase source time
out with 'Bad gateway' error. Upstream changes would be needed to
The same trick cannot be used for needles so these live in a separate repo.
openqa_worker Docker image
Docker images are provided by the project. We found an issue with the image at registry.opensuse.org/devel/openqa/containers15.2/openqa_worker:latest (missing qemu-img) so are using a locally built version. The Fedora-based images linked here are 4 years out of date. Upstream issue: https://github.com/os-autoinst/openQA/issues/4102