testlib: Add Test Anything Protocol (TAP) reporter

The Test Anything Protocol (TAP), was originally Perl's simple text-based
interface between testing modules and test harnesses, but has since been
adopted by a large number of producers and consumers in many different
languages, which allows colorizing and summarizing test results.

The format is very simple:

TAP version 13
ok 1 - test description
not ok 2 - test description
  ---
  message: 'Failure message'
  severity: fail
  expected: 123
  actual: 456
  ...
ok 3 - test description # SKIP
1..3

The specification [1] is very brief, so the implementation has been
based on how typical consumers behave, especially when it comes to
the undefined diagnostics block.

[1] http://testanything.org/tap-version-13-specification.html

Change-Id: I616e802ea380165c678510e940ddc6607d39c92d
Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>

1 files changed, 183 insertions, 0 deletions

diff --git a/tests/auto/testlib/selftests/expected_datatable.tap b/tests/auto/testlib/selftests/expected_datatable.tap
new file mode 100644
index 0000000000..8a3d473b6c
--- /dev/null
+++ b/tests/auto/testlib/selftests/expected_datatable.tap
@@ -0,0 +1,183 @@
+TAP version 13
+# tst_DataTable
+ok 1 - initTestCase()
+ok 2 - singleTestFunction1()
+ok 3 - singleTestFunction2()
+ok 4 - fiveTablePasses(fiveTablePasses_data 1)
+ok 5 - fiveTablePasses(fiveTablePasses_data 2)
+ok 6 - fiveTablePasses(fiveTablePasses_data 3)
+ok 7 - fiveTablePasses(fiveTablePasses_data 4)
+ok 8 - fiveTablePasses(fiveTablePasses_data 5)
+not ok 9 - fiveTableFailures(fiveTableFailures_data 1)
+  ---
+  type: QVERIFY
+  message: Verification failed
+  wanted: true (test)
+  found: false (test)
+  expected: true (test)
+  actual: false (test)
+  at: tst_DataTable::fiveTableFailures() (qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp:78)
+  file: qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp
+  line: 78
+  ...
+not ok 10 - fiveTableFailures(fiveTableFailures_data 2)
+  ---
+  type: QVERIFY
+  message: Verification failed
+  wanted: true (test)
+  found: false (test)
+  expected: true (test)
+  actual: false (test)
+  at: tst_DataTable::fiveTableFailures() (qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp:78)
+  file: qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp
+  line: 78
+  ...
+not ok 11 - fiveTableFailures(fiveTableFailures_data 3)
+  ---
+  type: QVERIFY
+  message: Verification failed
+  wanted: true (test)
+  found: false (test)
+  expected: true (test)
+  actual: false (test)
+  at: tst_DataTable::fiveTableFailures() (qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp:78)
+  file: qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp
+  line: 78
+  ...
+not ok 12 - fiveTableFailures(fiveTableFailures_data 4)
+  ---
+  type: QVERIFY
+  message: Verification failed
+  wanted: true (test)
+  found: false (test)
+  expected: true (test)
+  actual: false (test)
+  at: tst_DataTable::fiveTableFailures() (qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp:78)
+  file: qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp
+  line: 78
+  ...
+not ok 13 - fiveTableFailures(fiveTableFailures_data 5)
+  ---
+  type: QVERIFY
+  message: Verification failed
+  wanted: true (test)
+  found: false (test)
+  expected: true (test)
+  actual: false (test)
+  at: tst_DataTable::fiveTableFailures() (qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp:78)
+  file: qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp
+  line: 78
+  ...
+not ok 14 - startsWithFailure(startsWithFailure_data 1)
+  ---
+  type: QVERIFY
+  message: Verification failed
+  wanted: true (test)
+  found: false (test)
+  expected: true (test)
+  actual: false (test)
+  at: tst_DataTable::startsWithFailure() (qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp:78)
+  file: qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp
+  line: 78
+  ...
+ok 15 - startsWithFailure(startsWithFailure_data 2)
+ok 16 - startsWithFailure(startsWithFailure_data 3)
+ok 17 - startsWithFailure(startsWithFailure_data 4)
+ok 18 - startsWithFailure(startsWithFailure_data 5)
+ok 19 - endsWithFailure(endsWithFailure 1)
+ok 20 - endsWithFailure(endsWithFailure 2)
+ok 21 - endsWithFailure(endsWithFailure 3)
+ok 22 - endsWithFailure(endsWithFailure 4)
+not ok 23 - endsWithFailure(endsWithFailure 5)
+  ---
+  type: QVERIFY
+  message: Verification failed
+  wanted: true (test)
+  found: false (test)
+  expected: true (test)
+  actual: false (test)
+  at: tst_DataTable::endsWithFailure() (qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp:78)
+  file: qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp
+  line: 78
+  ...
+ok 24 - failureInMiddle(failureInMiddle_data 1)
+ok 25 - failureInMiddle(failureInMiddle_data 2)
+not ok 26 - failureInMiddle(failureInMiddle_data 3)
+  ---
+  type: QVERIFY
+  message: Verification failed
+  wanted: true (test)
+  found: false (test)
+  expected: true (test)
+  actual: false (test)
+  at: tst_DataTable::failureInMiddle() (qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp:78)
+  file: qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp
+  line: 78
+  ...
+ok 27 - failureInMiddle(failureInMiddle_data 4)
+ok 28 - failureInMiddle(failureInMiddle_data 5)
+not ok 29 - fiveIsolatedFailures(fiveIsolatedFailures_data 1)
+  ---
+  type: QVERIFY
+  message: Verification failed
+  wanted: true (!test)
+  found: false (!test)
+  expected: true (!test)
+  actual: false (!test)
+  at: tst_DataTable::fiveIsolatedFailures() (qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp:160)
+  file: qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp
+  line: 160
+  ...
+not ok 30 - fiveIsolatedFailures(fiveIsolatedFailures_data 2)
+  ---
+  type: QVERIFY
+  message: Verification failed
+  wanted: true (!test)
+  found: false (!test)
+  expected: true (!test)
+  actual: false (!test)
+  at: tst_DataTable::fiveIsolatedFailures() (qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp:160)
+  file: qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp
+  line: 160
+  ...
+not ok 31 - fiveIsolatedFailures(fiveIsolatedFailures_data 3)
+  ---
+  type: QVERIFY
+  message: Verification failed
+  wanted: true (!test)
+  found: false (!test)
+  expected: true (!test)
+  actual: false (!test)
+  at: tst_DataTable::fiveIsolatedFailures() (qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp:160)
+  file: qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp
+  line: 160
+  ...
+not ok 32 - fiveIsolatedFailures(fiveIsolatedFailures_data 4)
+  ---
+  type: QVERIFY
+  message: Verification failed
+  wanted: true (!test)
+  found: false (!test)
+  expected: true (!test)
+  actual: false (!test)
+  at: tst_DataTable::fiveIsolatedFailures() (qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp:160)
+  file: qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp
+  line: 160
+  ...
+not ok 33 - fiveIsolatedFailures(fiveIsolatedFailures_data 5)
+  ---
+  type: QVERIFY
+  message: Verification failed
+  wanted: true (!test)
+  found: false (!test)
+  expected: true (!test)
+  actual: false (!test)
+  at: tst_DataTable::fiveIsolatedFailures() (qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp:160)
+  file: qtbase/tests/auto/testlib/selftests/datatable/tst_datatable.cpp
+  line: 160
+  ...
+ok 34 - cleanupTestCase()
+1..34
+# tests 34
+# pass 21
+# fail 13