Skip to content

Visual regression tests

Sam Day requested to merge samcday/phosh:visual-regression-tests into main

Unfortunately this MR is quite large. I initially wanted to open an MR that introduces the 140 screenshots of the baseline, and then target the actual changes against that branch so that there's a cleaner diff to discuss. I'm unable to do this however as that would require the baseline branch to be pushed to the upstream repo, and I lack the permissions to do that.

I'm open to suggestions on how to tame this. Perhaps you could just pull the visual-regression-tests-baseline branch from my fork into the main repo? Then the merge-base for this MR can be updated and sanity can be restored.

This is the initial stab at #1051. It introduces a new tool and CI job in the main build, which continuously compares the screenshots generated in CI to a baseline reference. The results are made available through a JUnit report which provides convenient links to failed diffs.

An assortment of tweaks/changes to the screenshot test suite was required to tame the results into something that is more deterministic across multiple runs. There's still a lot of noise though, which you'll see soon enough after a few runs of the new pipeline in this MR.

Edited by Sam Day

Merge request reports