2.99 ScriptFu enhance: need test framework for ScriptFu in the repo
We need a test framework, written in Scheme, to test ScriptFu. Along with a test suite and/or comments/annotations that are testable.
The principle is: good software has a test suite to prove it works.
This issue is for discussing:
- good choice of test framework
- organization of the repo i.e. where the test framework and tests are kept.
Candidate frameworks
- The framework at the top of test9
- re-test, attached to #9660 (closed)
- others not proposed yet
(#9660 (closed) is about a bug in ScriptFu but has a side track discussing test frameworks. All discussion of test frameworks should now be in this new issue.)
It might make sense to use multiple frameworks.
Test categories
The tests can be divided into these categories:
- tests of TinyScheme, the embedded interpreter. Tests of basic Scheme functions such as (number? ) predicate function.
- tests of ScriptFu, any Scheme functions that ScriptFu has added to the language, such as (script-fu-register).
- tests of ScriptFu binding to the PDB, such as (gimp-image-new).
- test plug-ins (not unit-test, but functional test)
Differences from existing test suites
Since TinyScheme implements a subset of Scheme, some existing test suites for Scheme can test TinyScheme, with tests outside the subset expected to fail.
Since the ScriptFu version of TinyScheme has diverged from upstream TinyScheme repo,we need additional tests for the divergence:
-
ScriptFu TinyScheme supports Unicode but upstream TinyScheme does not (#9660 (closed) concerns this)
-
ScriptFu TinyScheme supports byte IO but upstream TinyScheme does not (test9.scm)
-
Other changes/fixes to the embedded TinyScheme not adopted upstream
Since ScriptFu binds to the Gimp PDB, we need tests of the bound functions. This serves two purposes: to test ScriptFu and to test the PDB, which otherwise has no test suite.
Suggested organization of the repo
plug-ins/scrip-fu/test
frameworks
tests
tinyscheme
script-fu
pdb
plug-ins
test-sphere.scm tests ScriptFu GUI
test-sphere-v3.scm
ts-helloworld.scm
test9.scm renamed as test-byte-IO.scm
meson.build (optional install test framework, plugins and scripts)
Any tests now in plug-ins/scrip-fu/scripts/test will be moved. These plug-ins are ordinarily not installed.
Installation of the test framework
Test framework(s) can be installed, say in GIMP/3.0/scripts/testframework.scm. Then users can load it in their plug-ins or in the ScriptFu console. #9640
Alternatively, the test framework can be defined in GIMP/3.0/scripts/scriptfu.init (the Scheme file that ScriptFu reads at startup) so that the test framework is accessible to plugins and scripts without explicitly loading it.
Testing process
Some test suites will be wrapped as plugins, for convenience. They won't be usually installed. When installed, you run a test suite by choosing the plugin's menu item. You monitor the results in the Gimp Error Console or the console where Gimp was started.
Other test suites will just be Scheme scripts. You can invoke them from the ScriptFu Console. You monitor the result in the SF Console, or in the Gimp Error Console and the console where Gimp was started.
For debugging, you can start Gimp in a console where G_MESSAGES_DEBUG=scriptfu (or =all) is in the environment.