QTestLib: rework QTest::compare_helper()

[ChangeLog][QTestLib] QCOMPARE now evaluates toString() on its
arguments lazily, speeding up the general case where the comparison
doesn't fail. This is true for the QCOMPARE functionality provided
by Qt. If you specialized qCompare() for your own types, then you
need to change its implementation in line with Qt's own qCompare()
specializations in order to enable this feature.

[ChangeLog][QTestLib] QCOMPARE calls with nullptr argument(s) will
now print the actual and expected values upon failure.

Previously it was not like that because of the compareHelper()
overload in qtestresult.cpp that treated the presence of
nullptr-arguments as a reason to ignore formatFailMessage() call.
New implementation does not have this check, and correctly
executes formatFailMessage() for all arguments.

Note that the qCompare() overloads that call QTestResult::compare()
internally were not affected by this patch, because they already
defer toString() invocation until the comparison fails.

Some numbers, collected against shared release developer build.
I checked how this change affects the test execution. The idea was
to pick some tests for types that do not have a specific
QTestResult::compare overload, so I picked a couple of QByteArray
tests.
The comparison is done by running a test 10 times and taking the
average execution duration, as reported in the log.

tst_qbytearrayapisymmetry:
 Before: 15.6 ms
 After:  14.2 ms

tst_qbytearray:
 Before: 41 ms
 After:  36 ms

The benefit is around 9% and 12% respectively.

Fixes: QTBUG-98874
Change-Id: I7d59ddc760168b15974e7720930f629fb34efa13
Reviewed-by: Marc Mutz <marc.mutz@qt.io>

7 files changed, 32 insertions, 6 deletions

diff --git a/tests/auto/testlib/selftests/expected_testlib.junitxml b/tests/auto/testlib/selftests/expected_testlib.junitxml
index 8e5bd0e37e..4a8923cff9 100644
--- a/tests/auto/testlib/selftests/expected_testlib.junitxml
+++ b/tests/auto/testlib/selftests/expected_testlib.junitxml
@@ -7,7 +7,10 @@
   </properties>
   <testcase name="initTestCase" classname="tst_TestLib" time="@TEST_DURATION@"/>
   <testcase name="basics" classname="tst_TestLib" time="@TEST_DURATION@">
-    <failure type="fail" message="Compared QObject pointers are not the same"/>
+    <failure type="fail" message="Compared QObject pointers are not the same">
+      <![CDATA[   Actual   (QTest::testObject()): tst_TestLib/"TestObject"
+   Expected (nullptr)            : <null>]]>
+    </failure>
   </testcase>
   <testcase name="delays" classname="tst_TestLib" time="@TEST_DURATION@"/>
   <testcase name="reals(zero)" classname="tst_TestLib" time="@TEST_DURATION@"/>
diff --git a/tests/auto/testlib/selftests/expected_testlib.lightxml b/tests/auto/testlib/selftests/expected_testlib.lightxml
index c4b7924e38..7d0c1f66c6 100644
--- a/tests/auto/testlib/selftests/expected_testlib.lightxml
+++ b/tests/auto/testlib/selftests/expected_testlib.lightxml
@@ -9,7 +9,9 @@
   </TestFunction>
   <TestFunction name="basics">
     <Incident type="fail" file="qtbase/tests/auto/testlib/selftests/testlib/tst_testlib.cpp" line="0">
-      <Description><![CDATA[Compared QObject pointers are not the same]]></Description>
+      <Description><![CDATA[Compared QObject pointers are not the same
+   Actual   (QTest::testObject()): tst_TestLib/"TestObject"
+   Expected (nullptr)            : <null>]]></Description>
     </Incident>
     <Duration msecs="0"/>
   </TestFunction>
diff --git a/tests/auto/testlib/selftests/expected_testlib.tap b/tests/auto/testlib/selftests/expected_testlib.tap
index 020006904f..ed421f1241 100644
--- a/tests/auto/testlib/selftests/expected_testlib.tap
+++ b/tests/auto/testlib/selftests/expected_testlib.tap
@@ -3,7 +3,12 @@ TAP version 13
 ok 1 - initTestCase()
 not ok 2 - basics()
   ---
-  # Compared QObject pointers are not the same
+  type: QCOMPARE
+  message: Compared QObject pointers are not the same
+  wanted: <null> (nullptr)
+  found: tst_TestLib/"TestObject" (QTest::testObject())
+  expected: <null> (nullptr)
+  actual: tst_TestLib/"TestObject" (QTest::testObject())
   at: tst_TestLib::basics() (qtbase/tests/auto/testlib/selftests/testlib/tst_testlib.cpp:0)
   file: qtbase/tests/auto/testlib/selftests/testlib/tst_testlib.cpp
   line: 0
diff --git a/tests/auto/testlib/selftests/expected_testlib.teamcity b/tests/auto/testlib/selftests/expected_testlib.teamcity
index b72dfde420..4455800826 100644
--- a/tests/auto/testlib/selftests/expected_testlib.teamcity
+++ b/tests/auto/testlib/selftests/expected_testlib.teamcity
@@ -2,7 +2,7 @@
 ##teamcity[testStarted name='initTestCase()' flowId='tst_TestLib']
 ##teamcity[testFinished name='initTestCase()' flowId='tst_TestLib']
 ##teamcity[testStarted name='basics()' flowId='tst_TestLib']
-##teamcity[testFailed name='basics()' message='Failure! |[Loc: qtbase/tests/auto/testlib/selftests/testlib/tst_testlib.cpp(0)|]' details='Compared QObject pointers are not the same' flowId='tst_TestLib']
+##teamcity[testFailed name='basics()' message='Failure! |[Loc: qtbase/tests/auto/testlib/selftests/testlib/tst_testlib.cpp(0)|]' details='Compared QObject pointers are not the same|n Actual (QTest::testObject()): tst_TestLib/"TestObject"|n Expected (nullptr) : <null>' flowId='tst_TestLib']
 ##teamcity[testFinished name='basics()' flowId='tst_TestLib']
 ##teamcity[testStarted name='delays()' flowId='tst_TestLib']
 ##teamcity[testFinished name='delays()' flowId='tst_TestLib']
diff --git a/tests/auto/testlib/selftests/expected_testlib.txt b/tests/auto/testlib/selftests/expected_testlib.txt
index b8437fa8b4..98ca6ecf11 100644
--- a/tests/auto/testlib/selftests/expected_testlib.txt
+++ b/tests/auto/testlib/selftests/expected_testlib.txt
@@ -2,6 +2,8 @@
 Config: Using QtTest library
 PASS   : tst_TestLib::initTestCase()
 FAIL!  : tst_TestLib::basics() Compared QObject pointers are not the same
+   Actual   (QTest::testObject()): tst_TestLib/"TestObject"
+   Expected (nullptr)            : <null>
    Loc: [qtbase/tests/auto/testlib/selftests/testlib/tst_testlib.cpp(0)]
 PASS   : tst_TestLib::delays()
 PASS   : tst_TestLib::reals(zero)
diff --git a/tests/auto/testlib/selftests/expected_testlib.xml b/tests/auto/testlib/selftests/expected_testlib.xml
index b5e1c078f1..a31259a9e2 100644
--- a/tests/auto/testlib/selftests/expected_testlib.xml
+++ b/tests/auto/testlib/selftests/expected_testlib.xml
@@ -11,7 +11,9 @@
   </TestFunction>
   <TestFunction name="basics">
     <Incident type="fail" file="qtbase/tests/auto/testlib/selftests/testlib/tst_testlib.cpp" line="0">
-      <Description><![CDATA[Compared QObject pointers are not the same]]></Description>
+      <Description><![CDATA[Compared QObject pointers are not the same
+   Actual   (QTest::testObject()): tst_TestLib/"TestObject"
+   Expected (nullptr)            : <null>]]></Description>
     </Incident>
     <Duration msecs="0"/>
   </TestFunction>
diff --git a/tests/auto/testlib/selftests/testlib/tst_testlib.cpp b/tests/auto/testlib/selftests/testlib/tst_testlib.cpp
index 598049efdd..435c72d7ff 100644
--- a/tests/auto/testlib/selftests/testlib/tst_testlib.cpp
+++ b/tests/auto/testlib/selftests/testlib/tst_testlib.cpp
@@ -14,7 +14,11 @@
 
 class tst_TestLib : public QObject
 {
-Q_OBJECT
+    Q_OBJECT
+
+public:
+    tst_TestLib();
+
 private slots:
     void basics() const;
     void delays() const;
@@ -22,6 +26,14 @@ private slots:
     void reals() const;
 };
 
+tst_TestLib::tst_TestLib()
+{
+    // Set object name, so that it's printed out when some comparison fails.
+    // Othewise object address will be printed, which will not allow
+    // tst_sefltest to compare the output with expected.
+    setObjectName("TestObject");
+}
+
 void tst_TestLib::basics() const
 {
     QVERIFY(QByteArray(QTest::currentAppName()).contains("testlib"));