Skip to content

meson, ci: Support tests that can fail under certain conditions

Marco Trevisan requested to merge wip/3v1n0/support-can-fail-tests into main

We have tests that are failing in some environments, but it's difficult to handle them because:

  • for some environments we just allow all the tests to fail: DANGEROUS
  • when we don't allow failures we have flacky tests: A CI pain

So, to avoid this and ensure that:

  • New failing tests are tracked in all platforms
  • gitlab integration on tests reports is working
  • coverage is reported also for failing tests (until they don't abort)

Add support for can_fail keyword on tests that would mark the test as part of the failing test suite.
Not adding the suite directly when defining the tests as this is definitely simpler and allows to define conditions more clearly (see next commits).

Now, add a default test setup that does not run the failing and flaky tests by default (not to bother distributors with testing well-known issues) and eventually run all the tests in CI:

  • Non-flaky tests cannot fail in all platforms
  • Failing and Flaky tests can fail

In both cases we save the test reports so that gitlab integration is preserved.

Initially I was thinking to use another approach, but it was indeed more hackish and didn't make the tests to appear as failed in gitlab MRs. So the only advantage was only to run the tests in default builds too.

Merge request reports