TAP test logger: report B?XFAIL (mostly) as a message

Previously, only the first B?XFAIL would be reported, all others would
be discarded. Furthermore, if a B?XFAIL had happened, B?PASS was also
not reported, since the B?XFAIL served as test line. However, if the
B?XFAIL was followed by a SKIP, B?XPASS or B?FAIL, these were reported
as normal, producing exactly the kind of duplicated test line that the
skipping of B?PASS was meant to supply.

So change B?XFAIL to be reported among the messages, but retain the
TODO annotation of the first on the test line of a subsequent B?PASS,
if nothing more drastic happens in the mean time. So now more than one
B?XFAIL can be reported, the test is still marked as a TODO and we
don't get duplicate test lines for a subsequent non-passing result.

This replaces the bool m_wasExpectedFail member with a QTestCharBuffer
m_firstExpectedFail that records the first XFAIL's TODO line (so its
isEmpty() fully replaces m_wasExpectedFail).

Previously, the at/file/line information for a B?XFail would be
supplied as top-level keys in the YAML block for a "Pass" reported as
not ok due to the XFail, as this location information is now part of
the B?XFail's message in the extensions/messages block. Duplicating
the first B?XFail's location at top level would add complexity and is
arguably misleading, as the test result is really a pass (after
ignoring known issues), and the location of the pass is indeterminate
(nominally the end of the test function, but actually also after the
cleanup() call for this test, when relevant), which is why a Pass has
no location information.

Task-number: QTBUG-96844
Change-Id: Ib3f24f56266ff49bf3bc9759ac8263ac69a62130
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>

3 files changed, 141 insertions, 61 deletions

diff --git a/tests/auto/testlib/selftests/expected_blacklisted.tap b/tests/auto/testlib/selftests/expected_blacklisted.tap
index 3217b42a4f..83def09e34 100644
--- a/tests/auto/testlib/selftests/expected_blacklisted.tap
+++ b/tests/auto/testlib/selftests/expected_blacklisted.tap
@@ -23,9 +23,13 @@ not ok 4 - fail() # TODO 'false' returned FALSE. (This test should BFAIL)
   ...
 not ok 5 - xfail() # TODO This test should BXFAIL then BPASS
   ---
-  at: tst_Blacklisted::xfail() (qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp:0)
-  file: qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp
-  line: 0
+  extensions:
+    messages:
+    - severity: xfail
+      message: This test should BXFAIL then BPASS
+      at: tst_Blacklisted::xfail() (qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp
+      line: 0
   ...
 ok 6 - multiSkip() # SKIP This skip should be repeated ten times
 ok 6 - multiSkip() # SKIP This skip should be repeated ten times
@@ -104,24 +108,31 @@ not ok 7 - multiFail() # TODO But this test should only contribute one to the bl
   file: qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp
   line: 0
   ...
-not ok 8 - xfailContinueSkip() # TODO This test should BXFAIL then SKIP
+not ok 8 - xfailContinueSkip() # SKIP This skip should be seen and counted
   ---
   at: tst_Blacklisted::xfailContinueSkip() (qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp:0)
   file: qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp
   line: 0
-  ...
-ok 8 - xfailContinueSkip() # SKIP This skip should be seen and counted
-not ok 9 - xfailContinueFail() # TODO This test should BXFAIL then BFAIL
-  ---
-  at: tst_Blacklisted::xfailContinueFail() (qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp:0)
-  file: qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp
-  line: 0
+  extensions:
+    messages:
+    - severity: xfail
+      message: This test should BXFAIL then SKIP
+      at: tst_Blacklisted::xfailContinueSkip() (qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp
+      line: 0
   ...
 not ok 9 - xfailContinueFail() # TODO This fail should be seen and counted as blacklisted
   ---
   at: tst_Blacklisted::xfailContinueFail() (qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp:0)
   file: qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp
   line: 0
+  extensions:
+    messages:
+    - severity: xfail
+      message: This test should BXFAIL then BFAIL
+      at: tst_Blacklisted::xfailContinueFail() (qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp
+      line: 0
   ...
 ok 10 - xpass() # TODO 'true' returned TRUE unexpectedly. (This test should BXPASS)
 ok 11 - xpassContinueSkip() # TODO 'true' returned TRUE unexpectedly. (This test should BXPASS then SKIP)
diff --git a/tests/auto/testlib/selftests/expected_expectfail.tap b/tests/auto/testlib/selftests/expected_expectfail.tap
index c61f47fd48..8ad34c5f0a 100644
--- a/tests/auto/testlib/selftests/expected_expectfail.tap
+++ b/tests/auto/testlib/selftests/expected_expectfail.tap
@@ -3,37 +3,52 @@ TAP version 13
 ok 1 - initTestCase()
 not ok 2 - xfailAndContinue() # TODO This should xfail
   ---
-  at: tst_ExpectFail::xfailAndContinue() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
-  file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
-  line: 0
   extensions:
     messages:
     - severity: debug
       message: begin
+    - severity: xfail
+      message: This should xfail
+      at: tst_ExpectFail::xfailAndContinue() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
+      line: 0
+    - severity: debug
+      message: after
   ...
-# debug: after
 not ok 3 - xfailAndAbort() # TODO This should xfail
   ---
-  at: tst_ExpectFail::xfailAndAbort() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
-  file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
-  line: 0
   extensions:
     messages:
     - severity: debug
       message: begin
+    - severity: xfail
+      message: This should xfail
+      at: tst_ExpectFail::xfailAndAbort() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
+      line: 0
   ...
-not ok 4 - xfailContinueSkip() # TODO This should xfail then skip
+not ok 4 - xfailContinueSkip() # SKIP This skip should be reported and counted
   ---
   at: tst_ExpectFail::xfailContinueSkip() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
   file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
   line: 0
+  extensions:
+    messages:
+    - severity: xfail
+      message: This should xfail then skip
+      at: tst_ExpectFail::xfailContinueSkip() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
+      line: 0
   ...
-ok 4 - xfailContinueSkip() # SKIP This skip should be reported and counted
 not ok 5 - xfailAbortSkip() # TODO This should xfail
   ---
-  at: tst_ExpectFail::xfailAbortSkip() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
-  file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
-  line: 0
+  extensions:
+    messages:
+    - severity: xfail
+      message: This should xfail
+      at: tst_ExpectFail::xfailAbortSkip() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
+      line: 0
   ...
 not ok 6 - xfailTwice()
   ---
@@ -72,52 +87,92 @@ not ok 10 - xfailDataDrivenTwice(Fail Continue)
   ...
 not ok 11 - xfailWithQString() # TODO A string
   ---
-  at: tst_ExpectFail::xfailWithQString() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
-  file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
-  line: 0
+  extensions:
+    messages:
+    - severity: xfail
+      message: A string
+      at: tst_ExpectFail::xfailWithQString() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
+      line: 0
+    - severity: xfail
+      message: Bug 5 (The message)
+      at: tst_ExpectFail::xfailWithQString() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
+      line: 0
   ...
 ok 12 - xfailDataDrivenWithQString(Pass Abort) # SKIP Each Continue or Pass reports this and increments skip-count
 ok 13 - xfailDataDrivenWithQString(Pass Continue) # SKIP Each Continue or Pass reports this and increments skip-count
 not ok 14 - xfailDataDrivenWithQString(Fail Abort) # TODO A string
   ---
-  at: tst_ExpectFail::xfailDataDrivenWithQString() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
-  file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
-  line: 0
+  extensions:
+    messages:
+    - severity: xfail
+      message: A string
+      at: tst_ExpectFail::xfailDataDrivenWithQString() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
+      line: 0
   ...
-not ok 15 - xfailDataDrivenWithQString(Fail Continue) # TODO A string
+not ok 15 - xfailDataDrivenWithQString(Fail Continue) # SKIP Each Continue or Pass reports this and increments skip-count
   ---
   at: tst_ExpectFail::xfailDataDrivenWithQString() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
   file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
   line: 0
+  extensions:
+    messages:
+    - severity: xfail
+      message: A string
+      at: tst_ExpectFail::xfailDataDrivenWithQString() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
+      line: 0
+    - severity: xfail
+      message: Bug 5 (The message)
+      at: tst_ExpectFail::xfailDataDrivenWithQString() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
+      line: 0
   ...
-ok 15 - xfailDataDrivenWithQString(Fail Continue) # SKIP Each Continue or Pass reports this and increments skip-count
 ok 16 - xfailDataDrivenWithQVerify(Pass Abort)
 ok 17 - xfailDataDrivenWithQVerify(Pass Continue)
 not ok 18 - xfailDataDrivenWithQVerify(Fail Abort) # TODO This test should xfail
   ---
-  at: tst_ExpectFail::xfailDataDrivenWithQVerify() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
-  file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
-  line: 0
+  extensions:
+    messages:
+    - severity: xfail
+      message: This test should xfail
+      at: tst_ExpectFail::xfailDataDrivenWithQVerify() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
+      line: 0
   ...
 not ok 19 - xfailDataDrivenWithQVerify(Fail Continue) # TODO This test should xfail
   ---
-  at: tst_ExpectFail::xfailDataDrivenWithQVerify() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
-  file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
-  line: 0
+  extensions:
+    messages:
+    - severity: xfail
+      message: This test should xfail
+      at: tst_ExpectFail::xfailDataDrivenWithQVerify() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
+      line: 0
   ...
 ok 20 - xfailDataDrivenWithQCompare(Pass Abort)
 ok 21 - xfailDataDrivenWithQCompare(Pass Continue)
 not ok 22 - xfailDataDrivenWithQCompare(Fail Abort) # TODO This test should xfail
   ---
-  at: tst_ExpectFail::xfailDataDrivenWithQCompare() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
-  file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
-  line: 0
+  extensions:
+    messages:
+    - severity: xfail
+      message: This test should xfail
+      at: tst_ExpectFail::xfailDataDrivenWithQCompare() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
+      line: 0
   ...
 not ok 23 - xfailDataDrivenWithQCompare(Fail Continue) # TODO This test should xfail
   ---
-  at: tst_ExpectFail::xfailDataDrivenWithQCompare() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
-  file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
-  line: 0
+  extensions:
+    messages:
+    - severity: xfail
+      message: This test should xfail
+      at: tst_ExpectFail::xfailDataDrivenWithQCompare() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
+      line: 0
   ...
 ok 24 - xfailOnWrongRow(Fail Abort)
   ---
@@ -135,15 +190,23 @@ ok 25 - xfailOnWrongRow(Fail Continue)
   ...
 not ok 26 - xfailOnAnyRow(Fail Abort) # TODO This test should xfail
   ---
-  at: tst_ExpectFail::xfailOnAnyRow() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
-  file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
-  line: 0
+  extensions:
+    messages:
+    - severity: xfail
+      message: This test should xfail
+      at: tst_ExpectFail::xfailOnAnyRow() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
+      line: 0
   ...
 not ok 27 - xfailOnAnyRow(Fail Continue) # TODO This test should xfail
   ---
-  at: tst_ExpectFail::xfailOnAnyRow() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
-  file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
-  line: 0
+  extensions:
+    messages:
+    - severity: xfail
+      message: This test should xfail
+      at: tst_ExpectFail::xfailOnAnyRow() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
+      line: 0
   ...
 not ok 28 - xfailWithoutCheck(Fail Abort)
   ---
@@ -169,12 +232,7 @@ ok 33 - xpassContinue() # TODO 'true' returned TRUE unexpectedly. ()
 ok 34 - xpassContinueSkip() # TODO 'true' returned TRUE unexpectedly. ()
 ok 34 - xpassContinueSkip() # SKIP This should be reached but not increment skip-count
 ok 35 - xpassContinueXfailAbort() # TODO 'true' returned TRUE unexpectedly. ()
-not ok 36 - xpassContinueXfailAbort() # TODO This test should xfail but not add to totals
-  ---
-  at: tst_ExpectFail::xpassContinueXfailAbort() (qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp:0)
-  file: qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp
-  line: 0
-  ...
+  # xfail: This test should xfail but not add to totals
 ok 36 - xpassAbortDataDrivenWithQVerify(XPass) # TODO 'true' returned TRUE unexpectedly. ()
 ok 37 - xpassAbortDataDrivenWithQVerify(Pass)
 ok 38 - xpassContinueDataDrivenWithQVerify(XPass) # TODO 'true' returned TRUE unexpectedly. ()
diff --git a/tests/auto/testlib/selftests/expected_strcmp.tap b/tests/auto/testlib/selftests/expected_strcmp.tap
index 0ff00d9b67..8f58ec8d24 100644
--- a/tests/auto/testlib/selftests/expected_strcmp.tap
+++ b/tests/auto/testlib/selftests/expected_strcmp.tap
@@ -2,12 +2,6 @@ TAP version 13
 # tst_StrCmp
 ok 1 - initTestCase()
 ok 2 - compareCharStars()
-not ok 3 - compareByteArray() # TODO Next test should fail
-  ---
-  at: tst_StrCmp::compareByteArray() (qtbase/tests/auto/testlib/selftests/strcmp/tst_strcmp.cpp:0)
-  file: qtbase/tests/auto/testlib/selftests/strcmp/tst_strcmp.cpp
-  line: 0
-  ...
 not ok 3 - compareByteArray()
   ---
   type: QCOMPARE
@@ -19,6 +13,23 @@ not ok 3 - compareByteArray()
   at: tst_StrCmp::compareByteArray() (qtbase/tests/auto/testlib/selftests/strcmp/tst_strcmp.cpp:0)
   file: qtbase/tests/auto/testlib/selftests/strcmp/tst_strcmp.cpp
   line: 0
+  extensions:
+    messages:
+    - severity: xfail
+      message: Next test should fail
+      at: tst_StrCmp::compareByteArray() (qtbase/tests/auto/testlib/selftests/strcmp/tst_strcmp.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/strcmp/tst_strcmp.cpp
+      line: 0
+    - severity: xfail
+      message: Next test should fail
+      at: tst_StrCmp::compareByteArray() (qtbase/tests/auto/testlib/selftests/strcmp/tst_strcmp.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/strcmp/tst_strcmp.cpp
+      line: 0
+    - severity: xfail
+      message: Next test should fail
+      at: tst_StrCmp::compareByteArray() (qtbase/tests/auto/testlib/selftests/strcmp/tst_strcmp.cpp:0)
+      file: qtbase/tests/auto/testlib/selftests/strcmp/tst_strcmp.cpp
+      line: 0
   ...
 not ok 4 - failByteArray()
   ---