Skip to content

Test GNOME OS using OpenQA

Sam Thursfield requested to merge sthursfield/james/openqa-master into master

Integrate with https://openqa.gnome.org/ to automatically test the iso-installer image using OpenQA and KVM.

Issue: #206

Documentation: https://gitlab.gnome.org/GNOME/gnome-build-meta/-/wikis/openqa/OpenQA-for-GNOME-developers

Summary

  • Testing is driven by Gitlab CI, tests execute on the usual gnome-build-meta runner. (This is an unconventional OpenQA setup, see below).

  • Test scripts are stored in this repo. Needles (screenshot data used by tests) are in a separate repo. (OpenQA has limited flexibility about this, see below for details.)

  • Tests must be manually started. We would like to run on every commit, #396 (slow iso-installer builds) is the blocker.

  • Tests are expected to run against 'master', primarily to guard against regressions.

  • If you're a maintainer of gnome-build-meta and want admin access to https://openqa.gnome.org, first log in to the OpenQA server, then ask in this MR and someone can set up your account permissions.

Connecting Gitlab CI to OpenQA

OpenQA is designed to work with a bank of dedicated test machines. Gnome-build-meta uses transient Gitlab CI runners. To attach the transient CI runner to the OpenQA server as a worker, the test-iso-installer-x86_64 job does the following:

  • Define an OpenQA WORKER_CLASS which is unique to the current job
  • Create a machine in openqa.gnome.org with this class.
  • Start /run_openqa_worker.sh, which connects as this machine.
  • Trigger the OpenQA test job and set the same WORKER_CLASS, so the server must now assign the job to our local worker.
  • When the test finishes, delete the OpenQA machine.

Limitations of current setup

OpenQA server expects to read tests and needles from the root of two Git repositories. The URL and ref is configured server-side and is shared for all branches and versions.

We have (ab)used CASEDIR setting so the worker reads the test cases from the local gnome-build-meta.git checkout. The OpenQA server cannot see these, meaning links in the web UI to the testcase source time out with 'Bad gateway' error. Upstream changes would be needed to improve this.

The same trick cannot be used for needles so these live in a separate repo.

openqa_worker Docker image

Docker images are provided by the project. We found an issue with the image at registry.opensuse.org/devel/openqa/containers15.2/openqa_worker:latest (missing qemu-img) so are using a locally built version. The Fedora-based images linked here are 4 years out of date. Upstream issue: https://github.com/os-autoinst/openQA/issues/4102

Screenshot

image

Edited by Sam Thursfield

Merge request reports