Orientation tests frequently fail on ARM (race condition?)

While trying to update from mutter 40 to 41 in Debian, we've seen test failures on armel, armhf and arm64 autobuilders (32-bit ARMv5 EABI softfloat, 32-bit ARMv7 EABI hardfloat and 64-bit aarch64 ARMv8 hardfloat, respectively). Some of the test failures do not happen every time, which makes me suspect a race condition.

The tests I'm looking at are the ones for screen orientation/rotation, in src/tests/monitor-unit-tests.c.

After adding more debug messages (!2048 (closed)) it seems that the problem is that wait_for_orientation_changes() does not always wait long enough. From an unsuccessful armel build:

ok 69 /backends/monitor/orientation/initial-stored-rotated-no-touch
# reusing existing orientation manager 0x1ed2920
# mutter-DEBUG: Applying mapping 0 to input device 'test-touchscreen', capabilities 1
# mutter-DEBUG: Output candidate 'Built-in display', score 0
# mutter-DEBUG: Matched input 'test-touchscreen' with output 'Built-in display'
# wait_for_orientation_changes: Waiting for orientation to change from 0: (undefined)...
# mutter-DEBUG: ConfigureNotify[200005] (1024, 768)
# mutter-DEBUG: expose for stage: win:0x200005 - redrawing area (x: 0, y: 0, width: 1024, height: 768)
# wait_for_orientation_changes: Timed out waiting for orientation change
# wait_for_orientation_changes: Orientation is now 0: (undefined)
**
mutter:ERROR:../src/tests/monitor-unit-tests.c:4410:meta_test_monitor_orientation_changes: assertion failed (meta_orientation_manager_get_orientation (orientation_manager) == i): (0 == 4)

The same test-case from a successful retry, also on armel for comparison:

ok 69 /backends/monitor/orientation/initial-stored-rotated-no-touch
# reusing existing orientation manager 0xdaa720
# mutter-DEBUG: Applying mapping 0 to input device 'test-touchscreen', capabilities 1
# mutter-DEBUG: Output candidate 'Built-in display', score 0
# mutter-DEBUG: Matched input 'test-touchscreen' with output 'Built-in display'
# wait_for_orientation_changes: Waiting for orientation to change from 0: (undefined)...
# mutter-DEBUG: ConfigureNotify[200005] (1024, 768)
# mutter-DEBUG: expose for stage: win:0x200005 - redrawing area (x: 0, y: 0, width: 1024, height: 768)
# mutter-DEBUG: Applying mapping 0 to input device 'test-touchscreen', capabilities 1
# mutter-DEBUG: Output candidate 'Built-in display', score 0
# mutter-DEBUG: Matched input 'test-touchscreen' with output 'Built-in display'
# 0xdaa720: Orientation changed to 4: right-up
# wait_for_orientation_changes: Orientation changed to 4: right-up
# wait_for_orientation_changes: Orientation is now 4: right-up
# mutter-DEBUG: meta_test_monitor_orientation_changes: ../src/tests/monitor-unit-tests.c:4414: Checking configuration per orientation
# mutter-DEBUG: Got monitor 0x0 : 768x1024
# mutter-DEBUG: Checking monitor 0
# mutter-DEBUG: Checking mode 1024x768 @ 60.000496
# mutter-DEBUG: Checking CRTC 0
# mutter-DEBUG: Checking CRTC Output 0
# mutter-DEBUG: ConfigureNotify[200005] (768, 1024)
# mutter-DEBUG: expose for stage: win:0x200005 - redrawing area (x: 0, y: 0, width: 768, height: 1024)
# mutter-DEBUG: Applying mapping 0 to input device 'test-touchscreen', capabilities 1
# mutter-DEBUG: Output candidate 'Built-in display', score 0
# mutter-DEBUG: Matched input 'test-touchscreen' with output 'Built-in display'
# 0xdaa720: Orientation changed to 3: left-up
# wait_for_orientation_changes: Waiting for orientation to change from 3: left-up...
# wait_for_orientation_changes: Timed out waiting for orientation change
# wait_for_orientation_changes: Orientation is now 3: left-up
# mutter-DEBUG: meta_test_monitor_orientation_changes: ../src/tests/monitor-unit-tests.c:4414: Checking configuration per orientation
# mutter-DEBUG: Got monitor 0x0 : 768x1024
# mutter-DEBUG: Checking monitor 0
# mutter-DEBUG: Checking mode 1024x768 @ 60.000496
# mutter-DEBUG: Checking CRTC 0
# mutter-DEBUG: Checking CRTC Output 0
# mutter-DEBUG: Applying mapping 0 to input device 'test-touchscreen', capabilities 1
# mutter-DEBUG: Output candidate 'Built-in display', score 0
# mutter-DEBUG: Matched input 'test-touchscreen' with output 'Built-in display'
# 0xdaa720: Orientation changed to 2: bottom-up
# wait_for_orientation_changes: Waiting for orientation to change from 2: bottom-up...
# mutter-DEBUG: ConfigureNotify[200005] (1024, 768)
# mutter-DEBUG: expose for stage: win:0x200005 - redrawing area (x: 0, y: 0, width: 1024, height: 768)
# wait_for_orientation_changes: Timed out waiting for orientation change
# wait_for_orientation_changes: Orientation is now 2: bottom-up
# mutter-DEBUG: meta_test_monitor_orientation_changes: ../src/tests/monitor-unit-tests.c:4414: Checking configuration per orientation
# mutter-DEBUG: Got monitor 0x0 : 1024x768
# mutter-DEBUG: Checking monitor 0
# mutter-DEBUG: Checking mode 1024x768 @ 60.000496
# mutter-DEBUG: Checking CRTC 0
# mutter-DEBUG: Checking CRTC Output 0
# mutter-DEBUG: Applying mapping 0 to input device 'test-touchscreen', capabilities 1
# mutter-DEBUG: Output candidate 'Built-in display', score 0
# mutter-DEBUG: Matched input 'test-touchscreen' with output 'Built-in display'
# 0xdaa720: Orientation changed to 1: normal
# wait_for_orientation_changes: Waiting for orientation to change from 1: normal...
# wait_for_orientation_changes: Timed out waiting for orientation change
# wait_for_orientation_changes: Orientation is now 1: normal
# mutter-DEBUG: meta_test_monitor_orientation_changes: ../src/tests/monitor-unit-tests.c:4414: Checking configuration per orientation
# mutter-DEBUG: Got monitor 0x0 : 1024x768
# mutter-DEBUG: Checking monitor 0
# mutter-DEBUG: Checking mode 1024x768 @ 60.000496
# mutter-DEBUG: Checking CRTC 0
# mutter-DEBUG: Checking CRTC Output 0
# wait_for_orientation_changes: Waiting for orientation to change from 1: normal...
# wait_for_orientation_changes: Timed out waiting for orientation change
# wait_for_orientation_changes: Orientation is now 1: normal
# mutter-DEBUG: meta_test_monitor_orientation_changes: ../src/tests/monitor-unit-tests.c:4439: Checking configuration per orientation
# mutter-DEBUG: Got monitor 0x0 : 1024x768
# mutter-DEBUG: Checking monitor 0
# mutter-DEBUG: Checking mode 1024x768 @ 60.000496
# mutter-DEBUG: Checking CRTC 0
# mutter-DEBUG: Checking CRTC Output 0
# 0xdaa720: Orientation changed to 4: right-up
# wait_for_orientation_changes: Waiting for orientation to change from 4: right-up...
# wait_for_orientation_changes: Timed out waiting for orientation change
# wait_for_orientation_changes: Orientation is now 4: right-up
# mutter-DEBUG: meta_test_monitor_orientation_changes: ../src/tests/monitor-unit-tests.c:4461: Checking configuration per orientation
# mutter-DEBUG: Got monitor 0x0 : 1024x768
# mutter-DEBUG: Checking monitor 0
# mutter-DEBUG: Checking mode 1024x768 @ 60.000496
# mutter-DEBUG: Checking CRTC 0
# mutter-DEBUG: Checking CRTC Output 0
# 0xdaa720: Orientation changed to 3: left-up
# wait_for_orientation_changes: Waiting for orientation to change from 3: left-up...
# wait_for_orientation_changes: Timed out waiting for orientation change
# wait_for_orientation_changes: Orientation is now 3: left-up
# mutter-DEBUG: meta_test_monitor_orientation_changes: ../src/tests/monitor-unit-tests.c:4461: Checking configuration per orientation
# mutter-DEBUG: Got monitor 0x0 : 1024x768
# mutter-DEBUG: Checking monitor 0
# mutter-DEBUG: Checking mode 1024x768 @ 60.000496
# mutter-DEBUG: Checking CRTC 0
# mutter-DEBUG: Checking CRTC Output 0
# 0xdaa720: Orientation changed to 2: bottom-up
# wait_for_orientation_changes: Waiting for orientation to change from 2: bottom-up...
# wait_for_orientation_changes: Timed out waiting for orientation change
# wait_for_orientation_changes: Orientation is now 2: bottom-up
# mutter-DEBUG: meta_test_monitor_orientation_changes: ../src/tests/monitor-unit-tests.c:4461: Checking configuration per orientation
# mutter-DEBUG: Got monitor 0x0 : 1024x768
# mutter-DEBUG: Checking monitor 0
# mutter-DEBUG: Checking mode 1024x768 @ 60.000496
# mutter-DEBUG: Checking CRTC 0
# mutter-DEBUG: Checking CRTC Output 0
# 0xdaa720: Orientation changed to 1: normal
# wait_for_orientation_changes: Waiting for orientation to change from 1: normal...
# wait_for_orientation_changes: Timed out waiting for orientation change
# wait_for_orientation_changes: Orientation is now 1: normal
# mutter-DEBUG: meta_test_monitor_orientation_changes: ../src/tests/monitor-unit-tests.c:4461: Checking configuration per orientation
# mutter-DEBUG: Got monitor 0x0 : 1024x768
# mutter-DEBUG: Checking monitor 0
# mutter-DEBUG: Checking mode 1024x768 @ 60.000496
# mutter-DEBUG: Checking CRTC 0
# mutter-DEBUG: Checking CRTC Output 0
# wait_for_orientation_changes: Waiting for orientation to change from 1: normal...
# wait_for_orientation_changes: Timed out waiting for orientation change
# wait_for_orientation_changes: Orientation is now 1: normal
ok 70 /backends/monitor/orientation/changes