Skip to content

Tests failing to start due to load on CI runners

Latest gnome-build-meta pipeline on master branch is running these two jobs:

These have managed to start openQA jobs, but they have not been picked up by any worker. In our case, the Gitlab CI job should be acting as a worker, so this is an infra problem of some sort.

Looking at an older job which already finished (aka was cancelled):

This also started some stuck openQA jobs such as https://openqa.gnome.org/tests/6849. Lets look at the config.

We use WORKER_CLASS to ensure the job goes to the correct Gitlab CI worker, this job has:

WORKER_CLASS qemu_x86_64-4752175

We can look at the worker.log of the failed job to see what's happening on the worker. Link to artifacts.

Worker log shows this:

[warn] [pid:91] The average load (60.43 40.02 27.48) is exceeding the configured threshold of 40. The worker will temporarily not accept new jobs until the load is lower again. - checking again for web UI 'https://openqa.gnome.org' in 125.61 s
[warn] [pid:91] The average load (61.52 47.34 31.82) is exceeding the configured threshold of 40. The worker will temporarily not accept new jobs until the load is lower again. - checking again for web UI 'https://openqa.gnome.org' in 276.68 s
[warn] [pid:91] The average load (73.88 75.42 48.78) is exceeding the configured threshold of 40. The worker will temporarily not accept new jobs until the load is lower again. - checking again for web UI 'https://openqa.gnome.org' in 94.39 s
[warn] [pid:91] The average load (90.40 78.77 52.45) is exceeding the configured threshold of 40. The worker will temporarily not accept new jobs until the load is lower again. - checking again for web UI 'https://openqa.gnome.org' in 186.42 s

So, either there's a behaviour change in recent openQA worker images, or the load on the runners is higher than ever before. Either way it's causing the QA tests to completely fail to start.

Edited by Sam Thursfield