... | ... | @@ -34,17 +34,33 @@ Tests can fail for several reasons. You will see the failure in the Web UI at ht |
|
|
|
|
|
First check the "Logs & Assets" tab. Most useful is `autoinst-log.txt`, containing the log of installing the ISO and running the testcases - this should log what went wrong. The `video.ogv` file can also be useful to see what went wrong.
|
|
|
|
|
|
If you see "no candidate needle matched" or "`assert_screen` failure", this indicates a graphical change meaning the screenshot no longer matches. If the UI change is intentional then we will need to update the needle. See below.
|
|
|
If you see "no candidate needle matched" or "`assert_screen` failure", this indicates a graphical change meaning the screenshot no longer matches. If the UI change is intentional then we will need to update the needle.
|
|
|
|
|
|
If in doubt about a test failure, [open a gnome-build-meta issue](https://gitlab.gnome.org/GNOME/gnome-build-meta/-/issues).
|
|
|
The OpenQA Web UI lets you compare the actual screenshot with the expected ones. Select a needle in 'Candidate needles and tags', then use the orange slider to see differences. The 'landscape' and 'eye' icons in the list switch between area-only or full diff. In the screenshot below, we have the expected image (installation completed) on the left, and the actual screenshot (installation still in progress) on the right.
|
|
|
|
|
|
![image](uploads/e1b3415068ed01db4ce68aada5972b0f/image.png)
|
|
|
|
|
|
If you think the test failure represents a bug, open an issue against the relevant component. Otherwise, or if in doubt, [open a gnome-build-meta issue](https://gitlab.gnome.org/GNOME/gnome-build-meta/-/issues).
|
|
|
|
|
|
## How to update a test
|
|
|
|
|
|
OpenQA's [developer mode](http://open.qa/docs/#_developer_mode) allows you to
|
|
|
update the needle from the web UI.
|
|
|
Test scripts live in [gnome-build-meta](https://gitlab.gnome.org/GNOME/gnome-build-meta/) in the `openqa/` folder. If you need to modify the test itself, create a branch of gnome-build-meta as normal with your change. The test pipeline will run with your version of the tests, and once it passes we can merge this to master.
|
|
|
|
|
|
Needles live in [openqa-needles](https://gitlab.gnome.org/GNOME/openqa-needles) folder, and the `master` branch is used by all tests. Multiple needles can have the same name, and OpenQA will pass a test if any one needle matches. If you need to modify a needle, the workflow is to create a new one, and the preferred way is to use OpenQA's [developer mode](http://open.qa/docs/#_developer_mode) as follows.
|
|
|
|
|
|
1. Ensure you have 'operator' permissions on OpenQA - ask an admin or open an issue if you don't have this.
|
|
|
2. Open the failed test in the Web UI.
|
|
|
3. Click the 'Create new needle' button to open the developer mode page
|
|
|
4. Use the 'Screenshot and Areas' section to create a new needle
|
|
|
* Usually you 'Take image from' the screenshot and 'Copy areas from' the most recent needle.
|
|
|
* Click and drag on the screenshot to create, move and resize match areas
|
|
|
* Click an area once to select it, enabling the "Selected area" controls
|
|
|
* Click on area again to change its [type](http://open.qa/docs/#_areas) - 'exclude' matches can be very useful when updating needles for UI changes.
|
|
|
5. Now use 'Basics of Needle' to commit your change to [openqa-needles.git](https://gitlab.gnome.org/GNOME/openqa-needles):
|
|
|
* Write a short commit message describing why you made the change
|
|
|
* Click 'Save' to commit and push from the OpenQA webUI.
|
|
|
|
|
|
**FIXME: do you need to be an admin to use 'developer mode'? Write more here.**
|
|
|
Now rerun the test which hopefully passes.
|
|
|
|
|
|
## Adding more tests
|
|
|
|
... | ... | |