... | ... | @@ -30,24 +30,17 @@ We would like to run the tests for every MR, which requires us to first [speed u |
|
|
|
|
|
## When tests fail
|
|
|
|
|
|
The test failure will be shown in the Web UI at https://openqa.gnome.org/.
|
|
|
Tests can fail for several reasons. You will see the failure in the Web UI at https://openqa.gnome.org/, and the corresponding Gitlab CI job will also fail.
|
|
|
|
|
|
If the machine fails to install or boot this is likely a regression somewhere
|
|
|
in GNOME or Freedesktop SDK.
|
|
|
First check the "Logs & Assets" tab. Most useful is `autoinst-log.txt`, containing the log of installing the ISO and running the testcases - this should log what went wrong. The `video.ogv` file can also be useful to see what went wrong.
|
|
|
|
|
|
If the test fails because GNOME's UI has changed, we might need to update the
|
|
|
test in question.
|
|
|
If you see "no candidate needle matched" or "`assert_screen` failure", this indicates a graphical change meaning the screenshot no longer matches. If the UI change is intentional then we will need to update the needle. See below.
|
|
|
|
|
|
In either case, the first step is to
|
|
|
[open a gnome-build-meta issue](https://gitlab.gnome.org/GNOME/gnome-build-meta/-/issues).
|
|
|
If in doubt about a test failure, [open a gnome-build-meta issue](https://gitlab.gnome.org/GNOME/gnome-build-meta/-/issues).
|
|
|
|
|
|
|
|
|
## How to update a test
|
|
|
|
|
|
A well-written test is robust against minor changes, but major changes to GNOME's
|
|
|
UI will result in a test failure at the `assert_screen` step. In this case the
|
|
|
[needle](http://open.qa/docs/#_needles) needs updating for a UI change.
|
|
|
|
|
|
OpenQA's [developer mode](http://open.qa/docs/#_developer_mode) allows you to
|
|
|
update the needle from the web UI.
|
|
|
|
... | ... | |