Expand testing of QEXPECT_FAIL()

Test both Abort and Continue cases. Test more with successive marked
as expected failure. Test cases with a QSKIP after the marked check.
Unify data functions where practical.

Change-Id: I2eade5e4dd0907d23e37137ce3d93cd5ca79f802
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>

1 files changed, 83 insertions, 25 deletions

diff --git a/tests/auto/testlib/selftests/expected_expectfail.txt b/tests/auto/testlib/selftests/expected_expectfail.txt
index 75d41e5733..83f673f2f3 100644
--- a/tests/auto/testlib/selftests/expected_expectfail.txt
+++ b/tests/auto/testlib/selftests/expected_expectfail.txt
@@ -10,46 +10,104 @@ QDEBUG : tst_ExpectFail::xfailAndAbort() begin
 XFAIL  : tst_ExpectFail::xfailAndAbort() This should xfail
    Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
 PASS   : tst_ExpectFail::xfailAndAbort()
+XFAIL  : tst_ExpectFail::xfailContinueSkip() This should xfail then skip
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+SKIP   : tst_ExpectFail::xfailContinueSkip() This skip should be reported and counted
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+XFAIL  : tst_ExpectFail::xfailAbortSkip() This should xfail
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+PASS   : tst_ExpectFail::xfailAbortSkip()
 FAIL!  : tst_ExpectFail::xfailTwice() Already expecting a fail
    Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+FAIL!  : tst_ExpectFail::xfailDataDrivenTwice(Pass Abort) Already expecting a fail
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+FAIL!  : tst_ExpectFail::xfailDataDrivenTwice(Pass Continue) Already expecting a fail
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+FAIL!  : tst_ExpectFail::xfailDataDrivenTwice(Fail Abort) Already expecting a fail
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+FAIL!  : tst_ExpectFail::xfailDataDrivenTwice(Fail Continue) Already expecting a fail
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
 XFAIL  : tst_ExpectFail::xfailWithQString() A string
    Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
 XFAIL  : tst_ExpectFail::xfailWithQString() Bug 5 (The message)
    Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
 PASS   : tst_ExpectFail::xfailWithQString()
-PASS   : tst_ExpectFail::xfailDataDrivenWithQVerify(Pass 1)
-PASS   : tst_ExpectFail::xfailDataDrivenWithQVerify(Pass 2)
-XFAIL  : tst_ExpectFail::xfailDataDrivenWithQVerify(Abort) This test should xfail
+SKIP   : tst_ExpectFail::xfailDataDrivenWithQString(Pass Abort) Each Continue or Pass reports this and increments skip-count
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+SKIP   : tst_ExpectFail::xfailDataDrivenWithQString(Pass Continue) Each Continue or Pass reports this and increments skip-count
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+XFAIL  : tst_ExpectFail::xfailDataDrivenWithQString(Fail Abort) A string
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+PASS   : tst_ExpectFail::xfailDataDrivenWithQString(Fail Abort)
+XFAIL  : tst_ExpectFail::xfailDataDrivenWithQString(Fail Continue) A string
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+XFAIL  : tst_ExpectFail::xfailDataDrivenWithQString(Fail Continue) Bug 5 (The message)
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+SKIP   : tst_ExpectFail::xfailDataDrivenWithQString(Fail Continue) Each Continue or Pass reports this and increments skip-count
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+PASS   : tst_ExpectFail::xfailDataDrivenWithQVerify(Pass Abort)
+PASS   : tst_ExpectFail::xfailDataDrivenWithQVerify(Pass Continue)
+XFAIL  : tst_ExpectFail::xfailDataDrivenWithQVerify(Fail Abort) This test should xfail
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+PASS   : tst_ExpectFail::xfailDataDrivenWithQVerify(Fail Abort)
+XFAIL  : tst_ExpectFail::xfailDataDrivenWithQVerify(Fail Continue) This test should xfail
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+PASS   : tst_ExpectFail::xfailDataDrivenWithQVerify(Fail Continue)
+PASS   : tst_ExpectFail::xfailDataDrivenWithQCompare(Pass Abort)
+PASS   : tst_ExpectFail::xfailDataDrivenWithQCompare(Pass Continue)
+XFAIL  : tst_ExpectFail::xfailDataDrivenWithQCompare(Fail Abort) This test should xfail
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+PASS   : tst_ExpectFail::xfailDataDrivenWithQCompare(Fail Abort)
+XFAIL  : tst_ExpectFail::xfailDataDrivenWithQCompare(Fail Continue) This test should xfail
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+PASS   : tst_ExpectFail::xfailDataDrivenWithQCompare(Fail Continue)
+QDEBUG : tst_ExpectFail::xfailOnWrongRow(Fail Abort) Should pass (*not* xpass), despite test-case name
+PASS   : tst_ExpectFail::xfailOnWrongRow(Fail Abort)
+QDEBUG : tst_ExpectFail::xfailOnWrongRow(Fail Continue) Should pass (*not* xpass), despite test-case name
+PASS   : tst_ExpectFail::xfailOnWrongRow(Fail Continue)
+XFAIL  : tst_ExpectFail::xfailOnAnyRow(Fail Abort) This test should xfail
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+PASS   : tst_ExpectFail::xfailOnAnyRow(Fail Abort)
+XFAIL  : tst_ExpectFail::xfailOnAnyRow(Fail Continue) This test should xfail
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+PASS   : tst_ExpectFail::xfailOnAnyRow(Fail Continue)
+QDEBUG : tst_ExpectFail::xfailWithoutCheck(Fail Abort) Should fail (*not* xfail), despite test-case name
+FAIL!  : tst_ExpectFail::xfailWithoutCheck(Fail Abort) QEXPECT_FAIL was called without any subsequent verification statements
+QDEBUG : tst_ExpectFail::xfailWithoutCheck(Fail Continue) Should fail (*not* xfail), despite test-case name
+FAIL!  : tst_ExpectFail::xfailWithoutCheck(Fail Continue) QEXPECT_FAIL was called without any subsequent verification statements
+XPASS  : tst_ExpectFail::xpassAbort() 'true' returned TRUE unexpectedly. ()
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+XPASS  : tst_ExpectFail::xpassAbortSkip() 'true' returned TRUE unexpectedly. ()
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+XPASS  : tst_ExpectFail::xpassAbortXfailContinue() 'true' returned TRUE unexpectedly. ()
+   Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
+XPASS  : tst_ExpectFail::xpassContinue() 'true' returned TRUE unexpectedly. ()
    Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
-PASS   : tst_ExpectFail::xfailDataDrivenWithQVerify(Abort)
-XFAIL  : tst_ExpectFail::xfailDataDrivenWithQVerify(Continue) This test should xfail
+QDEBUG : tst_ExpectFail::xpassContinue() This should be reached
+XPASS  : tst_ExpectFail::xpassContinueSkip() 'true' returned TRUE unexpectedly. ()
    Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
-PASS   : tst_ExpectFail::xfailDataDrivenWithQVerify(Continue)
-PASS   : tst_ExpectFail::xfailDataDrivenWithQCompare(Pass 1)
-PASS   : tst_ExpectFail::xfailDataDrivenWithQCompare(Pass 2)
-XFAIL  : tst_ExpectFail::xfailDataDrivenWithQCompare(Abort) This test should xfail
+SKIP   : tst_ExpectFail::xpassContinueSkip() This should be reached but not increment skip-count
    Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
-PASS   : tst_ExpectFail::xfailDataDrivenWithQCompare(Abort)
-XFAIL  : tst_ExpectFail::xfailDataDrivenWithQCompare(Continue) This test should xfail
+XPASS  : tst_ExpectFail::xpassContinueXfailAbort() 'true' returned TRUE unexpectedly. ()
    Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
-PASS   : tst_ExpectFail::xfailDataDrivenWithQCompare(Continue)
-PASS   : tst_ExpectFail::xfailOnWrongRow(right row)
-XFAIL  : tst_ExpectFail::xfailOnAnyRow(first row) This test should xfail
+XFAIL  : tst_ExpectFail::xpassContinueXfailAbort() This test should xfail but not add to totals
    Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
-PASS   : tst_ExpectFail::xfailOnAnyRow(first row)
-XFAIL  : tst_ExpectFail::xfailOnAnyRow(second row) This test should xfail
+XPASS  : tst_ExpectFail::xpassAbortDataDrivenWithQVerify(XPass) 'true' returned TRUE unexpectedly. ()
    Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
-PASS   : tst_ExpectFail::xfailOnAnyRow(second row)
-FAIL!  : tst_ExpectFail::xfailWithoutVerify(first row) QEXPECT_FAIL was called without any subsequent verification statements
-FAIL!  : tst_ExpectFail::xfailWithoutVerify(second row) QEXPECT_FAIL was called without any subsequent verification statements
-XPASS  : tst_ExpectFail::xpass() 'true' returned TRUE unexpectedly. ()
+PASS   : tst_ExpectFail::xpassAbortDataDrivenWithQVerify(Pass)
+XPASS  : tst_ExpectFail::xpassContinueDataDrivenWithQVerify(XPass) 'true' returned TRUE unexpectedly. ()
    Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
-XPASS  : tst_ExpectFail::xpassDataDrivenWithQVerify(XPass) 'true' returned TRUE unexpectedly. ()
+QDEBUG : tst_ExpectFail::xpassContinueDataDrivenWithQVerify(XPass) Test should Continue past XPASS
+QDEBUG : tst_ExpectFail::xpassContinueDataDrivenWithQVerify(Pass) Test should simply PASS
+PASS   : tst_ExpectFail::xpassContinueDataDrivenWithQVerify(Pass)
+XPASS  : tst_ExpectFail::xpassAbortDataDrivenWithQCompare(XPass) QCOMPARE(1, 1) returned TRUE unexpectedly.
    Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
-PASS   : tst_ExpectFail::xpassDataDrivenWithQVerify(Pass)
-XPASS  : tst_ExpectFail::xpassDataDrivenWithQCompare(XPass) QCOMPARE(1, 1) returned TRUE unexpectedly.
+PASS   : tst_ExpectFail::xpassAbortDataDrivenWithQCompare(Pass)
+XPASS  : tst_ExpectFail::xpassContinueDataDrivenWithQCompare(XPass) QCOMPARE(1, 1) returned TRUE unexpectedly.
    Loc: [qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(0)]
-PASS   : tst_ExpectFail::xpassDataDrivenWithQCompare(Pass)
+QDEBUG : tst_ExpectFail::xpassContinueDataDrivenWithQCompare(XPass) Test should Continue past XPASS
+QDEBUG : tst_ExpectFail::xpassContinueDataDrivenWithQCompare(Pass) Test should simply PASS
+PASS   : tst_ExpectFail::xpassContinueDataDrivenWithQCompare(Pass)
 PASS   : tst_ExpectFail::cleanupTestCase()
-Totals: 18 passed, 6 failed, 0 skipped, 0 blacklisted, 0ms
+Totals: 23 passed, 17 failed, 5 skipped, 0 blacklisted, 0ms
 ********* Finished testing of tst_ExpectFail *********