Exit test executable with failure if specified test functions are not found

If QtQuickTest test executable was given filters that didn't match in any of the
qml files run, it exited successfully.

In combination with a Coin bug when the test function name contains a space,
this allowed failing tests to pass the CI. If the test function fails, the
repeat attempts would pass two arguments to the executable. Neither of those
hit, but that wasn't considered a problem.

Check that all of the test functions named on the command line are actually
executed during the whole run and otherwise exit with a non-zero exit code.

I assume there's no duplicates in the whole names of test functions scoped with
testcase names.

Task-number: QTBUG-68197
Change-Id: Icf7fe263945403f02920522dfd187aeb76b7cb3c
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Kalle Viironen <kalle.viironen@qt.io>
Reviewed-by: Ville Voutilainen <ville.voutilainen@qt.io>

1 files changed, 14 insertions, 3 deletions

diff --git a/src/qmltest/quicktest.cpp b/src/qmltest/quicktest.cpp
index 8d3f5ffb23..a54b93f72b 100644
--- a/src/qmltest/quicktest.cpp
+++ b/src/qmltest/quicktest.cpp
@@ -476,6 +476,9 @@ int quick_test_main_with_setup(int argc, char **argv, const char *name, const ch
     // Register the test object
     qmlRegisterSingletonType<QTestRootObject>("Qt.test.qtestroot", 1, 0, "QTestRootObject", testRootObject);
 
+    QSet<QString> commandLineTestFunctions = QTest::testFunctions.toSet();
+    const bool filteringTestFunctions = !commandLineTestFunctions.isEmpty();
+
     // Scan through all of the "tst_*.qml" files and run each of them
     // in turn with a separate QQuickView (for test isolation).
     for (const QString &file : qAsConst(files)) {
@@ -508,10 +511,10 @@ int quick_test_main_with_setup(int argc, char **argv, const char *name, const ch
             continue;
         }
 
-        static const QSet<QString> commandLineTestFunctions = QTest::testFunctions.toSet();
-        if (!commandLineTestFunctions.isEmpty() &&
-            !availableTestFunctions.toSet().intersects(commandLineTestFunctions))
+        const QSet<QString> availableTestSet = availableTestFunctions.toSet();
+        if (filteringTestFunctions && !availableTestSet.intersects(commandLineTestFunctions))
             continue;
+        commandLineTestFunctions.subtract(availableTestSet);
 
         QQuickView view(&engine, nullptr);
         view.setFlags(Qt::Window | Qt::WindowSystemMenuHint
@@ -584,6 +587,14 @@ int quick_test_main_with_setup(int argc, char **argv, const char *name, const ch
     QuickTestResult::setProgramName(nullptr);
     delete app;
 
+    // Check that all test functions passed on the command line were found
+    if (!commandLineTestFunctions.isEmpty()) {
+        qWarning() << "Could not find the following test functions:";
+        for (const QString &functionName : qAsConst(commandLineTestFunctions))
+            qWarning("    %s()", qUtf8Printable(functionName));
+        return commandLineTestFunctions.count();
+    }
+
     // Return the number of failures as the exit code.
     return QuickTestResult::exitCode();
 }