Skip to content

Plan for a new unit-testing framework

Current situation

While I was trying to fix some unit broken during the !837 (merged) long MR, I had to look closely at several of our unit testing. It looks to me that some of the many issues with how we test right now are:

  • We are kinda reimplementing the initialization of the main application, then raising dialogs, or clicking with the pointer, etc. Instead of actually working on the existing GUI code. It means that we may have bugs which only appear in our unit test, while we may also miss actual bugs in the real application. It's not optimal at all.
  • many of the interactions, like pointer clicks, etc. probably don't work in Wayland so a major part of our unit test framework might end up dead in a few years.
  • Creating new unit tests is boringly complicated and time-consuming, so none does it.
  • Very often, the tests break, even though what they test still work. It's simply the test code itself which doesn't go well with changes in actual GIMP core.

Ideal situation

The ideal situation for GUI testing would be to have either a generic GUI-testing framework, but these have their own issues as well. Also most FLOSS tools I find are really web-oriented (or "mobile app" oriented).

A GTK-targetting test framework would be nice too, but after searching for this for years, it looks like we should not keep our hopes too high here.

Proposed changes

For libgimp* or any other libraries GIMP might create in the future

We could just build small binaries using these, or simply create test plug-ins which we would run from the built GIMP.

In any case, we should start testing exhaustively any function in our libraries.

Plug-ins

We should also create small scripts to test all the core plug-ins. These can simply be called from GIMP CLI, then we check results or whatnot (depending on type of plug-in).

Core and GUI

I am thinking that we could easily create a test framework within GIMP, so that tests are run by the real GIMP itself (and not some code partially initializing GIMP main objects like right now). Also the tests could be defined as very simple scripts which even non-developer could write. A good example of this is the release notes demo feature which I implemented (see demo property on tags in desktop/org.gimp.GIMP.appdata.xml.in.in). For instance, to show the "Fill by line art detection" settings, I created this demo:

          <li demo="toolbox:bucket-fill,
                    tool-options:fill-area=2,
                    tool-options:line-art-settings">

It tells GIMP to select the bucket-fill tool from the toolbox, then show the tool-options dockable, then the second value in the "fill-area" widget, then the "line-art-settings" widget. This relies on adding IDs to widgets we want to be able to "pick".

The same logic could be extended. We should also add the ability to pick from menus, or run actions, or create images, click and drag in the canvas, etc. This could all be defined in very simple lines. For instance, I see we have some test where we test the crop tool. We could write tests this way:

action:image-new
keyboard:Enter
toolbox:crop
canvas:click(10,10)
canvas:drag(30,40)
keyboard:Enter
test(image.width == 20 and image.height == 30)

Note that this is not a spec, just a quick'n dirty example of the kind of ideas I have on how our test syntax could look like. Advantages:

  • It would be very easy to write new tests.
  • The test framework would use actual GIMP code, making results more consistent (as much as we can at least; some interaction may still end up different, this is why generic framework able to actually produce events at system levels are ideal as these could be indistinguishable from real user interaction).

Note: for security, GIMP display should be obvious in stating it is in "test mode" (maybe the no-image canvas should write prominently this fact?) as we don't want people to be able to make GIMP run weird instructions without this being obvious it is running the test framework.

Color Management and Rendering

A very complicated part in a graphic software is color management and rendering, and bugs can often go invisible for a long time until someone actually run the maths on pixel data. We need to test this too.

This is actually part of the various other sections, i.e. it can be in libgimp testing (functions doing color work), in plug-ins (do we export colors correctly in this or that format) or in core code (do we show colors correctly? Do we simulate/soft-proof for printing correctly? And so on). I still add this section because someone recently raised this point to me by email and it's true that such checks are important enough that we need to give it particular care.

So here it is for a plan about how our test framework should evolve, because I am growing frustrated of our very crappy tests and constantly debugging them (not because of real bugs, but because the tests themselves break) so far.

I'm adding a 3.0 milestone, just in case. Not sure we'll be able to make this happen for GIMP 3.0.

Edited by Jehan