Meson build of GTK 3 fails test suite if one of the "somehow_broken" reftests passes
The Meson build has 7 tests which are marked as expected failures:
# Depending on the environment these fail, feel free to fix them
somehow_broken = [
'button-wrapping.ui',
'cellrenderer-pixbuf-stock-rtl.ui',
'label-sizing.ui',
'label-wrap-justify.ui',
'quit-mnemonic.ui',
'symbolic-icon-translucent-color.ui',
'window-height-for-width.ui',
]
...
test('reftest ' + testname, gtk_reftest,
...
should_fail: somehow_broken.contains(testname),
On Debian testing/unstable as of today, it seems the stars have aligned such that all of these except symbolic-icon-translucent-color.ui
actually pass (exit 0). Meson's interpretation of should_fail
is that if a test with should_fail: true
passes, then that is an error that causes the overall test suite to fail (this is not the same as Autotools, where an unexpected pass shows a warning but doesn't fail the overall test suite). This doesn't seem like what was intended for tests that can unpredictably pass or fail according to the environment.
I think it would be better if these 7 tests were handled in the same way as GLib's suite: "flaky"
tests, which aren't run unless specifically requested. I'll prepare a MR.