GitLab

Using OpenQA we can install the nightly GNOME OS images in a virtual machine, and simulate user interactions to test that it boots to a working graphical desktop. We can then do more integration testing of GNOME as a whole via the Shell and the core applications.

OpenQA has some useful online documentation. Be sure to look at the starter guide, tests developer guide and users guide.

Deployment of openqa.gnome.org is documented here.

Running the OpenQA tests

The OpenQA tests run as part of Gitlab CI and can be started manually as follows.

On an open merge request:

1. Wait for 'track' and 'build-x86_64' jobs to complete
2. Start 'iso-installer-x86_64' job.
3. Start 'test-iso-installer-x86_64' job.
4. Visit https://openqa.gnome.org/ to see progress (a link will be printed in Gitlab CI log)

To run the tests against 'master' or a branch with no open merge request, first start a pipeline and then follow the steps above.

We would like to run the tests for every MR, which requires us to first speed up the iso-installer job.

When tests fail

Tests can fail for several reasons. You will see the failure in the Web UI at https://openqa.gnome.org/, and the corresponding Gitlab CI job will also fail.

First check the "Logs & Assets" tab. Most useful is autoinst-log.txt, containing the log of installing the ISO and running the testcases - this should log what went wrong. The video.ogv file can also be useful to see what went wrong.

If you see "no candidate needle matched" or "assert_screen failure", this indicates a graphical change meaning the screenshot no longer matches. If the UI change is intentional then we will need to update the needle.

The OpenQA Web UI lets you compare the actual screenshot with the expected ones. Select a needle in 'Candidate needles and tags', then use the orange slider to see differences. The 'landscape' and 'eye' icons in the list switch between area-only or full diff. In the screenshot below, we have the expected image (installation completed) on the left, and the actual screenshot (installation still in progress) on the right.

If you think the test failure represents a bug, open an issue against the relevant component. Otherwise, or if in doubt, open a gnome-build-meta issue.

How to update a test

Test scripts live in gnome-build-meta in the openqa/ folder. If you need to modify the test itself, create a branch of gnome-build-meta as normal with your change. The test pipeline will run with your version of the tests, and once it passes we can merge this to master.

Needles live in openqa-needles folder, and the master branch is used by all tests. Multiple needles can have the same name, and OpenQA will pass a test if any one needle matches. If you need to modify a needle, the workflow is to create a new one, and the preferred way is to use OpenQA's developer mode as follows.

1. Ensure you have 'operator' permissions on OpenQA - ask an admin or open an issue if you don't have this.
2. Open the failed test in the Web UI.
3. Click the 'Create new needle' button (pin with a green plus) to open the developer mode page.
4. Use the 'Screenshot and Areas' section to create a new needle
   • Usually you 'Take image from' the screenshot and 'Copy areas from' the most recent needle.
   • Click and drag on the screenshot to create, move and resize match areas
   • Click an area once to select it, enabling the "Selected area" controls
   • Click on area again to change its type - 'exclude' matches can be very useful when updating needles for UI changes.
5. Now use 'Basics of Needle' to commit your change to openqa-needles.git:
   • Write a short commit message describing why you made the change
   • Click 'Save' to commit and push from the OpenQA webUI.

Now rerun the test which hopefully passes.

Adding more tests

OpenQA can be used to test anything that gnome-build-meta builds. We must be able to maintain the tests with volunteer effort, so try to write tests only around parts of the UI that won't change frequently.

The how to write tests documentation is your starting point for adding a new test.

Some distributions are already testing GNOME with OpenQA, and you may find you can adapt an existing test. Here are some we know of:

• OpenSuSE: web UI, tests, needles
• Fedora: web UI, tests

Running the test suite locally

When adding or debugging tests you probably don't want to wait for a full Gitlab CI pipeline. You can run the test suite locally if you don't mind doing some preparations. You will need Docker / Podman to run the tests, and jq.

1. Clone gnome-build-meta repo.

2. Clone openqa-needles repo, link or rename it to be inside gnome-build-meta folder at ./openqa/needles.

3. Get the installer.iso file you want to test. The simplest way is to build the 'iso-installer-x86_64` job in Gitlab CI, then use the 'Job artifacts' menu to download it.

4. Get the vars.json file from a successful OpenQA run. Visit https://openqa.gnome.org/tests to find the latest successful run, then visit the 'Logs & Assets' tab and download vars.json.

5. Run this command inside the gnome-build-meta clone to start the openqa_worker image:

   podman run --name openqa --privileged -it \
       -v ./openqa:/tests -v ./installer.iso:/installer.iso \
       --entrypoint isotovideo quay.io/ssssam/openqa_worker  \
       $(jq -jr 'keys[] as $k | "\($k)=\(.[$k]) "'  < vars.json) \
       CASEDIR=/tests

   The test log will be written to stdout, the other data is still inside the container.

6. Copy the test results and video out of the container:

   podman cp openqa:/testresults ./testresults 
   podman cp openqa:/video.ogv .
   # See all files in the container by running: podman export 12345678abc | tar -v -t | less

7. Remove the container with podman rm openqa when you are done.

It's common to use save_screenshot when writing testcases, which gives you a .png which you can then convert into a Needle.

Future work

See open issues matching 'openqa'