From fb2ef5fbf61b170475b0518bd889406cd4a7e662 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tor=20Arne=20Vestb=C3=B8?= Date: Mon, 20 Jul 2020 20:12:05 +0200 Subject: testlib selftest: Add README explaining how to run and debug tests Change-Id: Ica08f7013933e9e6a7678c0ba1f5827efa6eff42 Reviewed-by: Friedemann Kleint --- tests/auto/testlib/selftests/README | 71 +++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 tests/auto/testlib/selftests/README (limited to 'tests') diff --git a/tests/auto/testlib/selftests/README b/tests/auto/testlib/selftests/README new file mode 100644 index 0000000000..821bed45df --- /dev/null +++ b/tests/auto/testlib/selftests/README @@ -0,0 +1,71 @@ + +Running the QtTestLib selftests with Catch2 +=========================================== + +Catch2 [1] is a header only test framework that we use to allow +testing QtTestLib without relying on any part of testlib itself. + +To run the test suite, execute 'make check' or './tst_selftests' +as normal. This should print: + + =================================================================== + All tests passed (2453 assertions in 5 test cases) + +To run specific tests, first lists the available tests: + + ❯ ./tst_selftests -l + All available test cases: + Loggers support both old and new style arguments + Loggers can output to both file and stdout + Logging to file and stdout at the same time + All loggers can be enabled at the same time + Scenario: Test output of the loggers is as expected + 5 test cases + +Then pass the name of the test in quotes as the first argument: + + ❯ ./tst_selftests "Loggers support both old and new style arguments" + Filters: Loggers support both old and new style arguments + ================================================================== + All tests passed (96 assertions in 1 test case) + +You can find the tests in the sources as individual TEST_CASE +entries. Note that each of these tests run the tests once per +logger, and in the case of the test log check also all sub tests, +so the amount of actual test assertions is much higher than the +five tests listed above. + +To see what the tests is actually doing, pass the -s option. +This will result in very verbose output. Each leaf test is +prefixed with a heading: + + --------------------------------------------------------------- + Given: The QTestLog::TAP logger + When: Passing arguments with new style + --------------------------------------------------------------- + +You can choose a specific subtest by passing the -c option: + + ❯ ./tst_selftests "Scenario: Test output of the loggers is as expected" \ + -c "Given: The QTestLog::Plain logger" \ + -c 'And given: The "skip" subtest' + +It's possible to pass only the first -c options, to e.g. run all +tests with the Plain logger, but it's unfortunately not possible +to pass only the last -c option, to run the 'skip' subtest with +all loggers. + +If a test fails it will print the expected, actual, and difference. +The test results are also left in a temporary directory for closer +inspection. + +Add new tests by modifying selftest.pri and CMakeLists.txt, adding +a new subprogram. + +Generating new test expectations is done using the python script +in this directory (generate_expected_output.py). In the future this +will be done with the --rebase option to ./tst_selftest, but this +is not fleshed out yet. + +[1] https://github.com/catchorg/Catch2 + -- cgit v1.2.3