summaryrefslogtreecommitdiffstats
path: root/tests/auto/testlib/selftests
diff options
context:
space:
mode:
authorTor Arne Vestbø <tor.arne.vestbo@qt.io>2020-07-20 20:12:05 +0200
committerTor Arne Vestbø <tor.arne.vestbo@qt.io>2020-07-21 13:43:51 +0200
commitfb2ef5fbf61b170475b0518bd889406cd4a7e662 (patch)
treee0c09213c06cb1e0f947f76c480a6fef8ee00bc0 /tests/auto/testlib/selftests
parent85ce556443e84271549f73e94f7f5b3a03b7925a (diff)
testlib selftest: Add README explaining how to run and debug tests
Change-Id: Ica08f7013933e9e6a7678c0ba1f5827efa6eff42 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
Diffstat (limited to 'tests/auto/testlib/selftests')
-rw-r--r--tests/auto/testlib/selftests/README71
1 files changed, 71 insertions, 0 deletions
diff --git a/tests/auto/testlib/selftests/README b/tests/auto/testlib/selftests/README
new file mode 100644
index 0000000000..821bed45df
--- /dev/null
+++ b/tests/auto/testlib/selftests/README
@@ -0,0 +1,71 @@
+
+Running the QtTestLib selftests with Catch2
+===========================================
+
+Catch2 [1] is a header only test framework that we use to allow
+testing QtTestLib without relying on any part of testlib itself.
+
+To run the test suite, execute 'make check' or './tst_selftests'
+as normal. This should print:
+
+ ===================================================================
+ All tests passed (2453 assertions in 5 test cases)
+
+To run specific tests, first lists the available tests:
+
+ ❯ ./tst_selftests -l
+ All available test cases:
+ Loggers support both old and new style arguments
+ Loggers can output to both file and stdout
+ Logging to file and stdout at the same time
+ All loggers can be enabled at the same time
+ Scenario: Test output of the loggers is as expected
+ 5 test cases
+
+Then pass the name of the test in quotes as the first argument:
+
+ ❯ ./tst_selftests "Loggers support both old and new style arguments"
+ Filters: Loggers support both old and new style arguments
+ ==================================================================
+ All tests passed (96 assertions in 1 test case)
+
+You can find the tests in the sources as individual TEST_CASE
+entries. Note that each of these tests run the tests once per
+logger, and in the case of the test log check also all sub tests,
+so the amount of actual test assertions is much higher than the
+five tests listed above.
+
+To see what the tests is actually doing, pass the -s option.
+This will result in very verbose output. Each leaf test is
+prefixed with a heading:
+
+ ---------------------------------------------------------------
+ Given: The QTestLog::TAP logger
+ When: Passing arguments with new style
+ ---------------------------------------------------------------
+
+You can choose a specific subtest by passing the -c option:
+
+ ❯ ./tst_selftests "Scenario: Test output of the loggers is as expected" \
+ -c "Given: The QTestLog::Plain logger" \
+ -c 'And given: The "skip" subtest'
+
+It's possible to pass only the first -c options, to e.g. run all
+tests with the Plain logger, but it's unfortunately not possible
+to pass only the last -c option, to run the 'skip' subtest with
+all loggers.
+
+If a test fails it will print the expected, actual, and difference.
+The test results are also left in a temporary directory for closer
+inspection.
+
+Add new tests by modifying selftest.pri and CMakeLists.txt, adding
+a new subprogram.
+
+Generating new test expectations is done using the python script
+in this directory (generate_expected_output.py). In the future this
+will be done with the --rebase option to ./tst_selftest, but this
+is not fleshed out yet.
+
+[1] https://github.com/catchorg/Catch2
+