testlib: Improve verbose and XPASS output

Previously, verbose (-v2) and XPASS test output showed all QCOMPAREs as
"COMPARE()", making it impossible to see what was compared and difficult
to match the output to the source of a test containing many calls to
QCOMPARE.

This commit changes testlib's internal compare_helper API so that string
representations of the compared expressions are always passed to
QTestResult::compare() when available, and can thus be shown in the
verbose and XPASS output.  The XPASS output has also been changed to
state explicitly that the comparison succeeded unexpectedly, bringing it
in line with the XPASS output resulting from a call to QVERIFY.

This commit also changes all calls to compare_helper() to call the
eight-argument version of the function, which simplifies much of the
calling code.  The now obsolete four-argument version of
compare_helper() has been changed to output a warning that it is
obsolete.  It will be removed once other modules have had some time to
catch up.

The improved XPASS and verbose output is demonstrated by the expectfail
and verbose2 selftests.

Change-Id: I8baa46d5dd30e6c43b26f366c34dc5b64aab5f7c
Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>

9 files changed, 35 insertions, 35 deletions

diff --git a/tests/auto/testlib/selftests/expected_expectfail.lightxml b/tests/auto/testlib/selftests/expected_expectfail.lightxml
index 34f4f1e70b..55bd9578a8 100644
--- a/tests/auto/testlib/selftests/expected_expectfail.lightxml
+++ b/tests/auto/testlib/selftests/expected_expectfail.lightxml
@@ -132,7 +132,7 @@
 <TestFunction name="xpassDataDrivenWithQCompare">
 <Incident type="xpass" file="/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp" line="271">
     <DataTag><![CDATA[XPass]]></DataTag>
-    <Description><![CDATA[COMPARE()]]></Description>
+    <Description><![CDATA[QCOMPARE(1, 1) returned TRUE unexpectedly.]]></Description>
 </Incident>
 <Incident type="pass" file="" line="0">
     <DataTag><![CDATA[Pass]]></DataTag>
diff --git a/tests/auto/testlib/selftests/expected_expectfail.txt b/tests/auto/testlib/selftests/expected_expectfail.txt
index bd4e2172d6..6028b5ddfe 100644
--- a/tests/auto/testlib/selftests/expected_expectfail.txt
+++ b/tests/auto/testlib/selftests/expected_expectfail.txt
@@ -47,7 +47,7 @@ XPASS  : tst_ExpectFail::xpass() 'true' returned TRUE unexpectedly. ()
 XPASS  : tst_ExpectFail::xpassDataDrivenWithQVerify(XPass) 'true' returned TRUE unexpectedly. ()
    Loc: [/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(250)]
 PASS   : tst_ExpectFail::xpassDataDrivenWithQVerify(Pass)
-XPASS  : tst_ExpectFail::xpassDataDrivenWithQCompare(XPass) COMPARE()
+XPASS  : tst_ExpectFail::xpassDataDrivenWithQCompare(XPass) QCOMPARE(1, 1) returned TRUE unexpectedly.
    Loc: [/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp(271)]
 PASS   : tst_ExpectFail::xpassDataDrivenWithQCompare(Pass)
 PASS   : tst_ExpectFail::cleanupTestCase()
diff --git a/tests/auto/testlib/selftests/expected_expectfail.xml b/tests/auto/testlib/selftests/expected_expectfail.xml
index ff870a6a68..e3200b00d6 100644
--- a/tests/auto/testlib/selftests/expected_expectfail.xml
+++ b/tests/auto/testlib/selftests/expected_expectfail.xml
@@ -134,7 +134,7 @@
 <TestFunction name="xpassDataDrivenWithQCompare">
 <Incident type="xpass" file="/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/expectfail/tst_expectfail.cpp" line="271">
     <DataTag><![CDATA[XPass]]></DataTag>
-    <Description><![CDATA[COMPARE()]]></Description>
+    <Description><![CDATA[QCOMPARE(1, 1) returned TRUE unexpectedly.]]></Description>
 </Incident>
 <Incident type="pass" file="" line="0">
     <DataTag><![CDATA[Pass]]></DataTag>
diff --git a/tests/auto/testlib/selftests/expected_expectfail.xunitxml b/tests/auto/testlib/selftests/expected_expectfail.xunitxml
index 6040af41ff..45c260c77f 100644
--- a/tests/auto/testlib/selftests/expected_expectfail.xunitxml
+++ b/tests/auto/testlib/selftests/expected_expectfail.xunitxml
@@ -45,7 +45,7 @@
     <failure tag="XPass" message="&apos;true&apos; returned TRUE unexpectedly. ()" result="xpass"/>
   </testcase>
   <testcase result="xpass" name="xpassDataDrivenWithQCompare">
-    <failure tag="XPass" message="COMPARE()" result="xpass"/>
+    <failure tag="XPass" message="QCOMPARE(1, 1) returned TRUE unexpectedly." result="xpass"/>
   </testcase>
   <testcase result="pass" name="cleanupTestCase"/>
   <system-err>
diff --git a/tests/auto/testlib/selftests/expected_verbose2.lightxml b/tests/auto/testlib/selftests/expected_verbose2.lightxml
index 1310f2bb09..2937adbe01 100644
--- a/tests/auto/testlib/selftests/expected_verbose2.lightxml
+++ b/tests/auto/testlib/selftests/expected_verbose2.lightxml
@@ -12,7 +12,7 @@
 </Message>
 <Message type="info" file="/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp" line="112">
     <DataTag><![CDATA[row 1]]></DataTag>
-    <Description><![CDATA[COMPARE()]]></Description>
+    <Description><![CDATA[QCOMPARE(2 + 1, 3)]]></Description>
 </Message>
 <Incident type="pass" file="" line="0">
     <DataTag><![CDATA[row 1]]></DataTag>
@@ -23,7 +23,7 @@
 </Message>
 <Message type="info" file="/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp" line="112">
     <DataTag><![CDATA[row 2]]></DataTag>
-    <Description><![CDATA[COMPARE()]]></Description>
+    <Description><![CDATA[QCOMPARE(2 + 1, 3)]]></Description>
 </Message>
 <Incident type="pass" file="" line="0">
     <DataTag><![CDATA[row 2]]></DataTag>
@@ -36,7 +36,7 @@
 </Message>
 <Message type="info" file="/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp" line="112">
     <DataTag><![CDATA[row 1]]></DataTag>
-    <Description><![CDATA[COMPARE()]]></Description>
+    <Description><![CDATA[QCOMPARE(2 + 1, 3)]]></Description>
 </Message>
 <Incident type="pass" file="" line="0">
     <DataTag><![CDATA[row 1]]></DataTag>
@@ -53,7 +53,7 @@
 </Message>
 <Message type="info" file="/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp" line="112">
     <DataTag><![CDATA[row 1]]></DataTag>
-    <Description><![CDATA[COMPARE()]]></Description>
+    <Description><![CDATA[QCOMPARE(2 + 1, 3)]]></Description>
 </Message>
 <Incident type="pass" file="" line="0">
     <DataTag><![CDATA[row 1]]></DataTag>
@@ -78,7 +78,7 @@
 </Message>
 <Message type="info" file="/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp" line="112">
     <DataTag><![CDATA[row 2]]></DataTag>
-    <Description><![CDATA[COMPARE()]]></Description>
+    <Description><![CDATA[QCOMPARE(2 + 1, 3)]]></Description>
 </Message>
 <Incident type="pass" file="" line="0">
     <DataTag><![CDATA[row 2]]></DataTag>
@@ -123,7 +123,7 @@
 </Message>
 <Message type="info" file="/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp" line="112">
     <DataTag><![CDATA[row 2]]></DataTag>
-    <Description><![CDATA[COMPARE()]]></Description>
+    <Description><![CDATA[QCOMPARE(2 + 1, 3)]]></Description>
 </Message>
 <Incident type="pass" file="" line="0">
     <DataTag><![CDATA[row 2]]></DataTag>
diff --git a/tests/auto/testlib/selftests/expected_verbose2.txt b/tests/auto/testlib/selftests/expected_verbose2.txt
index 9012a7c569..34957f4ae7 100644
--- a/tests/auto/testlib/selftests/expected_verbose2.txt
+++ b/tests/auto/testlib/selftests/expected_verbose2.txt
@@ -5,18 +5,18 @@ PASS   : tst_Counting::initTestCase()
 INFO   : tst_Counting::testPassPass() entering
 INFO   : tst_Counting::testPassPass(row 1) QVERIFY(true)
    Loc: [/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp(111)]
-INFO   : tst_Counting::testPassPass(row 1) COMPARE()
+INFO   : tst_Counting::testPassPass(row 1) QCOMPARE(2 + 1, 3)
    Loc: [/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp(112)]
 PASS   : tst_Counting::testPassPass(row 1)
 INFO   : tst_Counting::testPassPass(row 2) QVERIFY(true)
    Loc: [/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp(111)]
-INFO   : tst_Counting::testPassPass(row 2) COMPARE()
+INFO   : tst_Counting::testPassPass(row 2) QCOMPARE(2 + 1, 3)
    Loc: [/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp(112)]
 PASS   : tst_Counting::testPassPass(row 2)
 INFO   : tst_Counting::testPassSkip() entering
 INFO   : tst_Counting::testPassSkip(row 1) QVERIFY(true)
    Loc: [/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp(111)]
-INFO   : tst_Counting::testPassSkip(row 1) COMPARE()
+INFO   : tst_Counting::testPassSkip(row 1) QCOMPARE(2 + 1, 3)
    Loc: [/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp(112)]
 PASS   : tst_Counting::testPassSkip(row 1)
 SKIP   : tst_Counting::testPassSkip(row 2) Skipping
@@ -24,7 +24,7 @@ SKIP   : tst_Counting::testPassSkip(row 2) Skipping
 INFO   : tst_Counting::testPassFail() entering
 INFO   : tst_Counting::testPassFail(row 1) QVERIFY(true)
    Loc: [/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp(111)]
-INFO   : tst_Counting::testPassFail(row 1) COMPARE()
+INFO   : tst_Counting::testPassFail(row 1) QCOMPARE(2 + 1, 3)
    Loc: [/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp(112)]
 PASS   : tst_Counting::testPassFail(row 1)
 INFO   : tst_Counting::testPassFail(row 2) QVERIFY(false)
@@ -36,7 +36,7 @@ SKIP   : tst_Counting::testSkipPass(row 1) Skipping
    Loc: [/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp(118)]
 INFO   : tst_Counting::testSkipPass(row 2) QVERIFY(true)
    Loc: [/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp(111)]
-INFO   : tst_Counting::testSkipPass(row 2) COMPARE()
+INFO   : tst_Counting::testSkipPass(row 2) QCOMPARE(2 + 1, 3)
    Loc: [/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp(112)]
 PASS   : tst_Counting::testSkipPass(row 2)
 INFO   : tst_Counting::testSkipSkip() entering
@@ -58,7 +58,7 @@ FAIL!  : tst_Counting::testFailPass(row 1) 'false' returned FALSE. ()
    Loc: [/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp(115)]
 INFO   : tst_Counting::testFailPass(row 2) QVERIFY(true)
    Loc: [/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp(111)]
-INFO   : tst_Counting::testFailPass(row 2) COMPARE()
+INFO   : tst_Counting::testFailPass(row 2) QCOMPARE(2 + 1, 3)
    Loc: [/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp(112)]
 PASS   : tst_Counting::testFailPass(row 2)
 INFO   : tst_Counting::testFailSkip() entering
diff --git a/tests/auto/testlib/selftests/expected_verbose2.xml b/tests/auto/testlib/selftests/expected_verbose2.xml
index 693ef2b187..d181c6d215 100644
--- a/tests/auto/testlib/selftests/expected_verbose2.xml
+++ b/tests/auto/testlib/selftests/expected_verbose2.xml
@@ -14,7 +14,7 @@
 </Message>
 <Message type="info" file="/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp" line="112">
     <DataTag><![CDATA[row 1]]></DataTag>
-    <Description><![CDATA[COMPARE()]]></Description>
+    <Description><![CDATA[QCOMPARE(2 + 1, 3)]]></Description>
 </Message>
 <Incident type="pass" file="" line="0">
     <DataTag><![CDATA[row 1]]></DataTag>
@@ -25,7 +25,7 @@
 </Message>
 <Message type="info" file="/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp" line="112">
     <DataTag><![CDATA[row 2]]></DataTag>
-    <Description><![CDATA[COMPARE()]]></Description>
+    <Description><![CDATA[QCOMPARE(2 + 1, 3)]]></Description>
 </Message>
 <Incident type="pass" file="" line="0">
     <DataTag><![CDATA[row 2]]></DataTag>
@@ -38,7 +38,7 @@
 </Message>
 <Message type="info" file="/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp" line="112">
     <DataTag><![CDATA[row 1]]></DataTag>
-    <Description><![CDATA[COMPARE()]]></Description>
+    <Description><![CDATA[QCOMPARE(2 + 1, 3)]]></Description>
 </Message>
 <Incident type="pass" file="" line="0">
     <DataTag><![CDATA[row 1]]></DataTag>
@@ -55,7 +55,7 @@
 </Message>
 <Message type="info" file="/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp" line="112">
     <DataTag><![CDATA[row 1]]></DataTag>
-    <Description><![CDATA[COMPARE()]]></Description>
+    <Description><![CDATA[QCOMPARE(2 + 1, 3)]]></Description>
 </Message>
 <Incident type="pass" file="" line="0">
     <DataTag><![CDATA[row 1]]></DataTag>
@@ -80,7 +80,7 @@
 </Message>
 <Message type="info" file="/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp" line="112">
     <DataTag><![CDATA[row 2]]></DataTag>
-    <Description><![CDATA[COMPARE()]]></Description>
+    <Description><![CDATA[QCOMPARE(2 + 1, 3)]]></Description>
 </Message>
 <Incident type="pass" file="" line="0">
     <DataTag><![CDATA[row 2]]></DataTag>
@@ -125,7 +125,7 @@
 </Message>
 <Message type="info" file="/home/user/dev/qt5/qtbase/tests/auto/testlib/selftests/counting/tst_counting.cpp" line="112">
     <DataTag><![CDATA[row 2]]></DataTag>
-    <Description><![CDATA[COMPARE()]]></Description>
+    <Description><![CDATA[QCOMPARE(2 + 1, 3)]]></Description>
 </Message>
 <Incident type="pass" file="" line="0">
     <DataTag><![CDATA[row 2]]></DataTag>
diff --git a/tests/auto/testlib/selftests/expected_verbose2.xunitxml b/tests/auto/testlib/selftests/expected_verbose2.xunitxml
index 8b9ed5257d..a774cb9d9f 100644
--- a/tests/auto/testlib/selftests/expected_verbose2.xunitxml
+++ b/tests/auto/testlib/selftests/expected_verbose2.xunitxml
@@ -7,25 +7,25 @@
   <testcase result="pass" name="initTestCase"/>
   <testcase result="pass" name="testPassPass">
     <!-- tag="row 1" message="QVERIFY(true)" type="info" -->
-    <!-- tag="row 1" message="COMPARE()" type="info" -->
+    <!-- tag="row 1" message="QCOMPARE(2 + 1, 3)" type="info" -->
     <!-- tag="row 2" message="QVERIFY(true)" type="info" -->
-    <!-- tag="row 2" message="COMPARE()" type="info" -->
+    <!-- tag="row 2" message="QCOMPARE(2 + 1, 3)" type="info" -->
   </testcase>
   <testcase result="pass" name="testPassSkip">
     <!-- tag="row 1" message="QVERIFY(true)" type="info" -->
-    <!-- tag="row 1" message="COMPARE()" type="info" -->
+    <!-- tag="row 1" message="QCOMPARE(2 + 1, 3)" type="info" -->
     <!-- tag="row 2" message="Skipping" type="skip" -->
   </testcase>
   <testcase result="fail" name="testPassFail">
     <!-- tag="row 1" message="QVERIFY(true)" type="info" -->
-    <!-- tag="row 1" message="COMPARE()" type="info" -->
+    <!-- tag="row 1" message="QCOMPARE(2 + 1, 3)" type="info" -->
     <!-- tag="row 2" message="QVERIFY(false)" type="info" -->
     <failure tag="row 2" message="&apos;false&apos; returned FALSE. ()" result="fail"/>
   </testcase>
   <testcase result="pass" name="testSkipPass">
     <!-- tag="row 1" message="Skipping" type="skip" -->
     <!-- tag="row 2" message="QVERIFY(true)" type="info" -->
-    <!-- tag="row 2" message="COMPARE()" type="info" -->
+    <!-- tag="row 2" message="QCOMPARE(2 + 1, 3)" type="info" -->
   </testcase>
   <testcase name="testSkipSkip">
     <!-- tag="row 1" message="Skipping" type="skip" -->
@@ -40,7 +40,7 @@
     <!-- tag="row 1" message="QVERIFY(false)" type="info" -->
     <failure tag="row 1" message="&apos;false&apos; returned FALSE. ()" result="fail"/>
     <!-- tag="row 2" message="QVERIFY(true)" type="info" -->
-    <!-- tag="row 2" message="COMPARE()" type="info" -->
+    <!-- tag="row 2" message="QCOMPARE(2 + 1, 3)" type="info" -->
   </testcase>
   <testcase result="fail" name="testFailSkip">
     <!-- tag="row 1" message="QVERIFY(false)" type="info" -->
@@ -70,25 +70,25 @@
   <testcase result="pass" name="cleanupTestCase"/>
   <system-err>
 <![CDATA[QVERIFY(true)]]>
-<![CDATA[COMPARE()]]>
+<![CDATA[QCOMPARE(2 + 1, 3)]]>
 <![CDATA[QVERIFY(true)]]>
-<![CDATA[COMPARE()]]>
+<![CDATA[QCOMPARE(2 + 1, 3)]]>
 <![CDATA[QVERIFY(true)]]>
-<![CDATA[COMPARE()]]>
+<![CDATA[QCOMPARE(2 + 1, 3)]]>
 <![CDATA[Skipping]]>
 <![CDATA[QVERIFY(true)]]>
-<![CDATA[COMPARE()]]>
+<![CDATA[QCOMPARE(2 + 1, 3)]]>
 <![CDATA[QVERIFY(false)]]>
 <![CDATA[Skipping]]>
 <![CDATA[QVERIFY(true)]]>
-<![CDATA[COMPARE()]]>
+<![CDATA[QCOMPARE(2 + 1, 3)]]>
 <![CDATA[Skipping]]>
 <![CDATA[Skipping]]>
 <![CDATA[Skipping]]>
 <![CDATA[QVERIFY(false)]]>
 <![CDATA[QVERIFY(false)]]>
 <![CDATA[QVERIFY(true)]]>
-<![CDATA[COMPARE()]]>
+<![CDATA[QCOMPARE(2 + 1, 3)]]>
 <![CDATA[QVERIFY(false)]]>
 <![CDATA[Skipping]]>
 <![CDATA[QVERIFY(false)]]>
diff --git a/tests/auto/testlib/selftests/tst_selftests.cpp b/tests/auto/testlib/selftests/tst_selftests.cpp
index 5d216992d7..a9ec4e31f2 100644
--- a/tests/auto/testlib/selftests/tst_selftests.cpp
+++ b/tests/auto/testlib/selftests/tst_selftests.cpp
@@ -104,7 +104,7 @@ inline bool qCompare
     }
 
     if (qAbs(qreal(r1.total) - qreal(r2.total)) <= qreal(r1.total)*variance) {
-        return compare_helper(true, "COMPARE()", file, line);
+        return compare_helper(true, 0, 0, 0, actual, expected, file, line);
     }
 
     // Whoops, didn't match.  Compare the whole string for the most useful failure message.