gtester needs support for repeating random tests
Submitted by Allison (desrt)
Assigned to Tim Janik
Link to original bug (#544244)
Description
gtester has reasonable random support, with the aim of reproduction of test failures. it lacks, however, a decent framework for running the same testcase repeatedly with different random seeds.
this forces the test writter to simply provide a single long-running test case that loops over itself trying many different random cases. it also means that, if the tests fail, you probably have to run this loop for many iterations (using the random seed given by gtester) to fast-forward the generator to the case that actually failed (in order to debug).
far better would be if gtester had some concept of repeating test cases, reseeding the generator back to some predictable place each time. some nice features could be done:
-
no need to fast-forward the generator -- the failing instance of the test could be immediately reproduced from the seed output by the tester
-
number of iterations could be chosen based on the thoroughness of testing
-
some nice progress bar code could be implemented by gtester for repeated tests