testlib selftest: Add README explaining how to run and debug tests

Change-Id: Ica08f7013933e9e6a7678c0ba1f5827efa6eff42
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>

1 files changed, 71 insertions, 0 deletions

diff --git a/tests/auto/testlib/selftests/README b/tests/auto/testlib/selftests/README
new file mode 100644
index 0000000000..821bed45df
--- /dev/null
+++ b/tests/auto/testlib/selftests/README
@@ -0,0 +1,71 @@
+
+Running the QtTestLib selftests with Catch2
+===========================================
+
+Catch2 [1] is a header only test framework that we use to allow
+testing QtTestLib without relying on any part of testlib itself.
+
+To run the test suite, execute 'make check' or './tst_selftests'
+as normal. This should print:
+
+    ===================================================================
+    All tests passed (2453 assertions in 5 test cases)
+
+To run specific tests, first lists the available tests:
+
+    ❯ ./tst_selftests -l
+    All available test cases:
+      Loggers support both old and new style arguments
+      Loggers can output to both file and stdout
+      Logging to file and stdout at the same time
+      All loggers can be enabled at the same time
+      Scenario: Test output of the loggers is as expected
+    5 test cases
+
+Then pass the name of the test in quotes as the first argument:
+
+    ❯ ./tst_selftests "Loggers support both old and new style arguments"
+    Filters: Loggers support both old and new style arguments
+    ==================================================================
+    All tests passed (96 assertions in 1 test case)
+
+You can find the tests in the sources as individual TEST_CASE
+entries. Note that each of these tests run the tests once per
+logger, and in the case of the test log check also all sub tests,
+so the amount of actual test assertions is much higher than the
+five tests listed above.
+
+To see what the tests is actually doing, pass the -s option.
+This will result in very verbose output. Each leaf test is
+prefixed with a heading:
+
+    ---------------------------------------------------------------
+          Given: The QTestLog::TAP logger
+           When: Passing arguments with new style
+    ---------------------------------------------------------------
+
+You can choose a specific subtest by passing the -c option:
+
+    ❯ ./tst_selftests "Scenario: Test output of the loggers is as expected" \
+        -c "Given: The QTestLog::Plain logger" \
+        -c 'And given: The "skip" subtest'
+
+It's possible to pass only the first -c options, to e.g. run all
+tests with the Plain logger, but it's unfortunately not possible
+to pass only the last -c option, to run the 'skip' subtest with
+all loggers.
+
+If a test fails it will print the expected, actual, and difference.
+The test results are also left in a temporary directory for closer
+inspection.
+
+Add new tests by modifying selftest.pri and CMakeLists.txt, adding
+a new subprogram.
+
+Generating new test expectations is done using the python script
+in this directory (generate_expected_output.py). In the future this
+will be done with the --rebase option to ./tst_selftest, but this
+is not fleshed out yet.
+
+[1] https://github.com/catchorg/Catch2
+