summaryrefslogtreecommitdiffstats
path: root/src/testlib/qtestresult.cpp
Commit message (Collapse)AuthorAgeFilesLines
* QTest: rip out qxp::function_ref from reportResult()Thiago Macieira11 days1-4/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is causing huge code bloat because everything is a local lambda. Instead, pass direct type-erased function and data pointers to the replacement function. Testing with tst_qcborvalue, this reduces the compilation time and the output binary size significantly: Before After Compiler Time Size Time Size GCC 13.2 136.99 s 202.3 MB 13.88 s 14.3 MB GCC 14.0 131.49 s 202.7 MB 14.69 s 14.4 MB Clang 17 77.2 s 146.7 MB 13.62 s 12.2 MB Clang 18 141.9 s 187.1 MB 13.62 s 12.4 MB This causes a difference in how toString() overloads are found. Previously it would match far more overloads because the toString() calls were expanded by the macro. Now, we depend on Argument-Dependent Lookup and associated namespaces, so toString() overloads should not be in the QTest namespace any more. With this patch applied, the testlib testcase of tst_selftest started failing, because nullptr is now handled differently. However, I consider it as a bugfix, because previously it was falling back to a default implementation, and now it is using the QTest::toString(std::nullptr_t) overload, which is a desired behavior. Update the reference files for tst_selftest with the new expected output. Task-number: QTBUG-124272 Change-Id: Ie28eadac333c4bcd8c08fffd17c5484186accdf6 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
* QTest: make failOnWarning() functional on temp objects destructionVladimir Belyavsky2023-11-221-7/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | We need to be able to handle warnings that may occur when temporary objects, that were created in a test function, are destroyed. For example, now we miss all warnings that might be triggered from the object destructor, if the object's deletion was delayed (e.g. via deleteLater()). Also we miss all the warnings that were triggered on the test's cleanup() call. To fix this we need simply move QTestLog::clearFailOnWarnings() from QTestResult::finishedCurrentTestData() to the later stage, i.e. into QTestLog::clearCurrentTestState() which is actually called in appropriate time from QTestResult::finishedCurrentTestDataCleanup(). Same for QTestLog::clearIgnoreMessages(), since they are interrelated, so we need to clear them at the same time. We need this change for QML tests in particularly, to be able fail on warnings that might be triggered from Component.onDestruction() of some temporary test object. Pick-to: 6.6 6.5 Change-Id: I58a57691f20761619f56bd1bea3a862f2c26f569 Reviewed-by: Mitch Curtis <mitch.curtis@qt.io> Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
* Increase QTest failure message limitMitch Curtis2023-10-191-5/+9
| | | | | | | | | | | | | | QPalette specifically has quite a large amount of output (1363 characters) when toString is called on it. We should make sure that we can fit that in our failure messages. This patch does that by increasing the limit from 1024 characters to 4096. Fixes: QTBUG-5903 Fixes: QTBUG-87039 Pick-to: 6.5 6.6 Change-Id: I1dc5078ad05858bb6542c3a06c6b84711af79e4f Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
* QTestEventLoop: don't stop the test fails outside the main threadThiago Macieira2023-05-051-1/+2
| | | | | | | | | | | | | | | | | | | | | This amends commit 115f828ae406d2805869fccda9269209af646a84 "QTestEventLoop: stop when the test fails" from Qt 6.3 to not attempt to stop the QTestEventLoop instance if the failure happened outside the main thread. This is to prevent the QTEL from being created outside the main thread and thus cause warnings about QObject in addition to and logged before the real test failure. [ChangeLog][QtTest][QTestEventLoop] The QTestEventLoop no longer attempts to exit its event loop if the failure was detected outside the main thread. QWARN : tst_QMutex::tryLock_non_recursive() QObject: Cannot create children for a parent that is in a different thread. (Parent is QCoreApplication(0x7fffffffe710), parent's thread is QThread(0x802e29000), current thread is QThread(0x7fffffffd5a8) FAIL! : tst_QMutex::tryLock_non_recursive() 'timer.elapsed() >= waitTime - systemTimersResolution' returned FALSE. () Change-Id: Ieab617d69f3b4b54ab30fffd175b139322a1f02c Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* QTest: Rename QCOMPARE_XX parameters to add meaningMårten Nordheim2023-03-081-8/+8
| | | | | | | | | | | | | | | The parameter names were previously "lhs" and "rhs", which is incredibly abstract for a testing framework. One of the parameters will tend to be a baseline value to compare against while the other is the result of some action we want to test. Thus I suggest they be renamed "computed" and "baseline". This way we can, hopefully, retain the semantic that the 'left'/first argument is the computed ('actual' in QCOMPARE) value while the 'right'/second argument is the baseline ('expected' in QCOMPARE.) Change-Id: I3e0fdce2a3f1faca06fdf7184ef6e0eb9724d990 Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
* Long live Q_UNREACHABLE_RETURN()!Marc Mutz2022-10-151-4/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a combination of Q_UNREACHABLE() with a return statement. ATM, the return statement is unconditionally included. If we notice that some compilers warn about return after __builtin_unreachable(), then we can map Q_UNREACHABLE_RETURN(...) to Q_UNREACHABLE() without having to touch all the code that uses explicit Q_UNREACHABLE() + return. The fact that Boost has BOOST_UNREACHABLE_RETURN() indicates that there are compilers that complain about a lack of return after Q_UNREACHABLE (we know that MSVC, ICC, and GHS are among them), as well as compilers that complained about a return being present (Coverity). Take this opportunity to properly adapt to Coverity, by leaving out the return statement on this compiler. Apply the macro around the code base, using a clang-tidy transformer rule: const std::string unr = "unr", val = "val", ret = "ret"; auto makeUnreachableReturn = cat("Q_UNREACHABLE_RETURN(", ifBound(val, cat(node(val)), cat("")), ")"); auto ignoringSwitchCases = [](auto stmt) { return anyOf(stmt, switchCase(subStmt(stmt))); }; makeRule( stmt(ignoringSwitchCases(stmt(isExpandedFromMacro("Q_UNREACHABLE")).bind(unr)), nextStmt(returnStmt(optionally(hasReturnValue(expr().bind(val)))).bind(ret))), {changeTo(node(unr), cat(makeUnreachableReturn, ";")), // TODO: why is the ; lost w/o this? changeTo(node(ret), cat(""))}, cat("use ", makeUnreachableReturn)) ); where nextStmt() is copied from some upstream clang-tidy check's private implementation and subStmt() is a private matcher that gives access to SwitchCase's SubStmt. A.k.a. qt-use-unreachable-return. There were some false positives, suppressed them with NOLINTNEXTLINE. They're not really false positiives, it's just that Clang sees the world in one way and if conditonal compilation (#if) differs for other compilers, Clang doesn't know better. This is an artifact of matching two consecutive statements. I haven't figured out how to remove the empty line left by the deletion of the return statement, if it, indeed, was on a separate line, so post-processed the patch to remove all the lines matching ^\+ *$ from the diff: git commit -am meep git reset --hard HEAD^ git diff HEAD..HEAD@{1} | sed '/^\+ *$/d' | recountdiff - | patch -p1 [ChangeLog][QtCore][QtAssert] Added Q_UNREACHABLE_RETURN() macro. Change-Id: I9782939f16091c964f25b7826e1c0dbd13a71305 Reviewed-by: Marc Mutz <marc.mutz@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* testlib: Don't print QCOMPARE values if they lack string representationVolker Hilsheimer2022-07-101-23/+27
| | | | | | | | | | | | | | | | | | | | | | | Before 0681a2dd5a8095baddb5905fb21a58ce19b958c5, QCOMPARE'ing types for which no QTest::toString specialization exists did not output Actual and Expected lines on failure, as that would only print <null> for both values (which then look like the same value, confusingly). Commit 0681a2dd5a8095baddb5905fb21a58ce19b958c5 changed that behavior, and started printing the confusing <null> values. Take care of the logic in the formatFailMessage function: if both values are nullptr, then print only the variable names, but not the confusing <null> text representation of the values. Remove dead and duplicated code related to the formatting logic, add a self-test function, and update the expected_cmptest files. Fixes: QTBUG-104867 Pick-to: 6.4 Change-Id: I4be98e79f91196b14690a2cc0a68ffd50b431a45 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* QCOMPARE/QVERIFY: fix huge pessimisation in QTestResult::reportResultMarc Mutz2022-06-031-1/+2
| | | | | | | | | | | | | | | | The old code allocated a stack buffer, but asked the runtime to zero-initialize it. That's 1KiB of writes to the stack on every QCOMPARE and QVERIFY before any actual work is done. Fixing this little laissez-faire to just initialize the first character in the buffer results in nice little speedups of ~40%. This fixes the issue in reportResult(), a Cut'n'paste from compare() and verify(), which have since been fixed by a previous commit. Change-Id: I5cad57299490925b88e768dc751304699274db2e Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* QTestLib: rework QTest::compare_helper()Ivan Solovev2022-06-031-2/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | [ChangeLog][QTestLib] QCOMPARE now evaluates toString() on its arguments lazily, speeding up the general case where the comparison doesn't fail. This is true for the QCOMPARE functionality provided by Qt. If you specialized qCompare() for your own types, then you need to change its implementation in line with Qt's own qCompare() specializations in order to enable this feature. [ChangeLog][QTestLib] QCOMPARE calls with nullptr argument(s) will now print the actual and expected values upon failure. Previously it was not like that because of the compareHelper() overload in qtestresult.cpp that treated the presence of nullptr-arguments as a reason to ignore formatFailMessage() call. New implementation does not have this check, and correctly executes formatFailMessage() for all arguments. Note that the qCompare() overloads that call QTestResult::compare() internally were not affected by this patch, because they already defer toString() invocation until the comparison fails. Some numbers, collected against shared release developer build. I checked how this change affects the test execution. The idea was to pick some tests for types that do not have a specific QTestResult::compare overload, so I picked a couple of QByteArray tests. The comparison is done by running a test 10 times and taking the average execution duration, as reported in the log. tst_qbytearrayapisymmetry: Before: 15.6 ms After: 14.2 ms tst_qbytearray: Before: 41 ms After: 36 ms The benefit is around 9% and 12% respectively. Fixes: QTBUG-98874 Change-Id: I7d59ddc760168b15974e7720930f629fb34efa13 Reviewed-by: Marc Mutz <marc.mutz@qt.io>
* Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}()Ivan Solovev2022-06-031-11/+105
| | | | | | | | | | | | | | | | | | | | | [ChangeLog][QTestLib] Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}() macros. These new macros behave similarly to QVERIFY(a op b), where 'op' is ==, !=, <, <=, >, >= respectively, but print a formatted error message with argument values in case of failure. The formatting is done lazily, which means that the strings will be generated only when the comparison fails. Also add a new test for tst_selftest and generate expected output for it. Fixes: QTBUG-98873 Task-number: QTBUG-98874 Change-Id: Ic8074798901d7a469b1f58d5cd28bbf49a3da1db Reviewed-by: Marc Mutz <marc.mutz@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* QCOMPARE/QVERIFY: fix huge pessimisation in QTestResultMarc Mutz2022-06-031-2/+4
| | | | | | | | | | | | | | | | | | | | | | | The old code allocated a stack buffer, but asked the runtime to zero-initialize it. That's 1KiB of writes to the stack on every QCOMPARE and QVERIFY before any actual work is done. Fixing this little laissez-faire to just initialize the first character in the buffer results in nice little speedups of up to or exceeding 2x. Amends d946507727b363326d05f48da93c2af04bdda76d. [ChangeLog][QtTestLib] Optimized successful QCOMPARE and QVERIFY for an up to 2x speedup. This has the potential to meaningfully reduce the load on the CI, so picking all the way to 5.15. Pick-to: 6.3 6.2 5.15 Change-Id: Ib93d69282ec87cbd26a60e4ac14413e8cef8ff78 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Use SPDX license identifiersLucie Gérard2022-05-161-38/+2
| | | | | | | | | | | | | Replace the current license disclaimer in files by a SPDX-License-Identifier. Files that have to be modified by hand are modified. License files are organized under LICENSES directory. Task-number: QTBUG-67283 Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
* QtTestLib: replace remaining uses of QL1String with QL1StringViewSona Kurazyan2022-05-021-2/+2
| | | | | | | | | Remove unneeded \fn qdoc lines as a drive-by. Task-number: QTBUG-98434 Change-Id: Id93ddbb38b97a8f5a6734bfbc82686ccb3a87aa6 Reviewed-by: Marc Mutz <marc.mutz@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Make counts of various types of test result add up correctlyEdward Welbourne2022-01-131-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | Added tests for repeated skips and failures (from within void lambdas, to simulate skips and failures from within event handlers). These exhibit yet more ways to count more than one outcome for a test. The new QTest::failOnWarning() can also provoke more than one failure from a single test, and several existing selftests exhibited various ways for the Totals line's counts to add up to more than the number of actual tests run. Fixed counting so that only the first decisive incident is counted. Tests can still report later failure or skipping, but only the first is counted. Added a currentTestState in qtestlog.cpp, by which it keeps track of whether the test has resolved to a result, and clearCurrentTestState() by which other code can reset that at the end of each test. This brought to light various places where test-end clean-up was not being handled - due to failure or skipping in a *_data() method or init, or a skip in cleanup. Fixes: QTBUG-95661 Change-Id: I5d24a37a53d3db225fa602649d8aad8f5ed6c1ad Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Add QTest::failOnWarningMitch Curtis2021-11-261-0/+1
| | | | | | | | | | | | | | This solves the long-standing problem of not being able to easily fail a test when a certain warning is output. [ChangeLog][QtTest] Added QTest::failOnWarning. When called in a test function, any warning that matches the given pattern will cause a test failure. The test will continue execution when a failure is added. All patterns are cleared at the end of each test function. Fixes: QTBUG-70029 Change-Id: I5763f8d4acf1cee8178be43a503619fbfb0f4f36 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Let loggers know which test function is being leftTor Arne Vestbø2021-08-101-2/+2
| | | | | | | | | | | None of the loggers rely on this at the moment, but one could imagine one that does. Plus, doing any sort of debug logging in leaveTestFunction will at the moment be attributed to UnknownTestFunc(). Pick-to: 6.2 Change-Id: I284b2785a276e028b9f57c26357679fd9e045ca7 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Document order of QTestResult completion functionsTor Arne Vestbø2021-08-101-0/+39
| | | | | | | | Pick-to: 6.2 Change-Id: I34adbcc2bd02887a93cd1e86692401e840afe8bd Reviewed-by: Paul Wicking <paul.wicking@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* QTestEventLoop: stop when the test failsEdward Welbourne2021-08-051-0/+2
| | | | | | | | | | | | | | | | | | | | | | It makes no sense for the event loop of a test to keep running after a test has failed. This lets test code simply use the usual testlib macros to compare and verify values in asynchronous tests that would otherwise need to hand-test values and send a signal on failure (that the main test can connect to an event-loops quit() or equivalent). For example, QLocalSocket's benchmark simply uses the usual macros, without doing anything to stop its event loop if they fail, with the sad result that, when a test fails, it does so repeatedly and then times out, causing the test program to be killed without running later tests. With this change, that test code (once converted to use QTestEventLoop) is able to exit gracefully on the first failure. [ChangeLog][QtTest][QTestEventLoop] The QTestEventLoop new exits its event loop as soon as the test is known to be failing. Task-number: QTBUG-91713 Change-Id: If0d455741668722034906763025dda496d2afbb4 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* testlib: Pass on file location on failure, but don't assume we have oneTor Arne Vestbø2021-08-041-5/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We try our best to pass on the file location of a failure, including for fatal errors, but the reporting or logging machinery should not assume there is one. By passing on nullptr for the file location we allow the logging backends to decide how to handle the situation, e.g. by not emitting extra fields for failure location. This effectively reverts c25687fa0b6e4be043e1f8c92c093d8b06ca06c4, in favor of relying on the backends to cope with null filename, which they already did. As qFatal uses QMessageLogger, which by default disables file/line information in release builds, we need to explicitly enable this in our self-tests, to get uniform test results. Similarly, we disable file/line info from testlib itself, as reporting Qt internal file and line information for user diagnostics is less useful. The odd one out there is qtestdata.cpp, which still ends up in test output due to using QTEST_ASSERT instead of qFatal for its diagnostics. Cleaning up that, and unifying how we report testlib issues to the user, is left for another day. Pick-to: 6.2 Change-Id: Ib9451b8eed86fe3ade4a4dcaf0037e1a3450321c Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* Fix QFAIL() to interract correctly with QEXPECT_FAIL()Edward Welbourne2021-06-151-6/+11
| | | | | | | | | | | | | | | | | | | Previously, it went direct to QTestResults::addFailure() without going via the checking for expected failure. Add QTestResults::fail() to take care of this checking, as for verify() and compare(). Tidied up the code implementing expected failure and QFAIL(), while I was about it. Adjusted an existing test to verify that expecting a QFAIL() works, by using QFAIL() instead of QVERIFY(false). Remove the QVERIFY(false) whose comment brought this to my attention. [ChangeLog][QtTestLib][QFAIL] QEXPECT_FAIL() now correctly anticipates a subsequent QFAIL(). Previously QFAIL() counted as a fail regardless. Change-Id: Icc28cf70e5ff3006363791ea03aa01f2f591eb71 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* QTestlib: Speed up comparison of qsizetypeFriedemann Kleint2021-01-221-0/+10
| | | | | | | | | | | | | Add a qCompare() overload for qsizetype, preventing it falling back to the slow code path pre-formatting expected/actual. toString() should use the correct format from the already present int-types (see macro TO_STRING_IMPL). Complements 94aa350621e8a5c4ad3b438c10fc1c0a9ed3bc8a. Task-number: QTBUG-38890 Change-Id: Ieb8cea7de086141a2c80f93b4c1be01572be96df Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* QTestLib: add a abort-on-fail environment variableGiuseppe D'Angelo2020-12-041-11/+41
| | | | | | | | | | | | | | | | | | | | | When debugging a spurious failure it's extremely useful to run the test repeadtly into a debugger until a failure appears. When that happens, one wants to immediately start debugging. In so far, this has only been possible by placing breakpoints inside Qt itself (when a failure is logged). Add another way: an env variable, similar to QT_FATAL_WARNINGS, that makes any failure fatal (terminate() gets called. Bonus: you can control the termination handler!) [ChangeLog][QtTestLib][QtTest] When the QTEST_FATAL_FAIL environment variable is set to a non-zero value, a test immediately aborts its execution. This is useful to debug intermittent failures. Change-Id: If2395f964ea482c30b8c8feab98db7fdee701cd3 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io> Reviewed-by: Jason McDonald <macadder1@gmail.com>
* QTestResult: Don't pass a nullptr file to addFailure()Ulf Hermann2020-09-301-3/+5
| | | | | | | | addBFail() asserts on the file being non-null. The convention seems to be "Unknown File" for cases where we cannot determine the file. Change-Id: I3a4d0130352d77d75f264fad6f3bd47c6700ef4c Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
* QTestLib: Speed up QCOMPARE for QStringFriedemann Kleint2019-08-011-0/+26
| | | | | | | | | | | | Add overloads for qCompare() for QStringView making use of the fast formatting helper introduced by 94aa350621e8a5c4ad3b438c10fc1c0a9ed3bc8a for int. Speeds up the bug report example by a factor of 3..4. Task-number: QTBUG-38890 Change-Id: Icc706618b2f1d23b37d354a04d4e1d1cc4b5aee3 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* QTestLib: Introduce nullptrFriedemann Kleint2019-07-111-19/+17
| | | | | | | | Apply Fixits by Qt Creator with some amendments. Task-number: QTBUG-69413 Change-Id: I620e40a277dc2b20b0ec26fc32577e66b0456bb3 Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
* QTestLib: Speed up QCOMPARE for float, double, int, unsignedFriedemann Kleint2019-07-091-15/+101
| | | | | | | | | | | | Factor out a helper template formatting the QCOMPARE failure message delaying the formatting of the parameters with toString() and use that for float, double, int and unsigned. This removes the need to always format and allocate strings for the operands even in the success case, speeding up the QColor test from 3.3s to 700ms (Windows/release). Task-number: QTBUG-38890 Change-Id: I999484765bdaed921d3fc35f35a9fbbcd82a9704 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Add BXPASS and BXFAILOliver Wolff2019-01-291-3/+10
| | | | | | | | | | | | | | | Prioritize blacklisting over QEXPECT_FAIL so that a test that is blacklisted no longer fails if QEXPECT_FAIL returns true unexpectedly. To reflect this state properly, the two values of BXPASS and BXFAIL were added to testlib's output. [ChangeLog][Important Behavior Changes][QtTestLib] Blacklisting of tests will be taken into account for XPASS and XFAIL. A blacklisted test that causes an XPASS will no longer be a fail. Task-number: QTBUG-72928 Change-Id: Ia2232fdc714d405fa3fd9aea6c89eb2836bc5950 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Add Test Anything Protocol (TAP) reporterTor Arne Vestbø2018-03-141-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The Test Anything Protocol (TAP), was originally Perl's simple text-based interface between testing modules and test harnesses, but has since been adopted by a large number of producers and consumers in many different languages, which allows colorizing and summarizing test results. The format is very simple: TAP version 13 ok 1 - test description not ok 2 - test description --- message: 'Failure message' severity: fail expected: 123 actual: 456 ... ok 3 - test description # SKIP 1..3 The specification [1] is very brief, so the implementation has been based on how typical consumers behave, especially when it comes to the undefined diagnostics block. [1] http://testanything.org/tap-version-13-specification.html Change-Id: I616e802ea380165c678510e940ddc6607d39c92d Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
* Updated license headersJani Heikkinen2016-01-151-14/+20
| | | | | | | | | | | From Qt 5.7 -> LGPL v2.1 isn't an option anymore, see http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/ Updated license headers to use new LGPL header instead of LGPL21 one (in those files which will be under LGPL v3) Change-Id: I046ec3e47b1876cd7b4b0353a576b352e3a946d9 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
* Testlib: Avoid formatting unneeded messages.Friedemann Kleint2015-10-301-9/+11
| | | | | | | | | Brings down gui/painting/qcolor from 7s to 3s, reducing the calls to printf() helpers from 16E6 to 10E6. Task-number: QTBUG-38890 Change-Id: I34065e6f09fc9a14920b06aa6936908229f3f9c4 Reviewed-by: Frederik Gladhorn <frederik.gladhorn@theqtcompany.com>
* Update copyright headersJani Heikkinen2015-02-111-7/+7
| | | | | | | | | | | | | | | | | | Qt copyrights are now in The Qt Company, so we could update the source code headers accordingly. In the same go we should also fix the links to point to qt.io. Outdated header.LGPL removed (use header.LGPL21 instead) Old header.LGPL3 renamed to header.LGPL3-COMM to match actual licensing combination. New header.LGPL-COMM taken in the use file which were using old header.LGPL3 (src/plugins/platforms/android/extract.cpp) Added new header.LGPL3 containing Commercial + LGPLv3 + GPLv2 license combination Change-Id: I6f49b819a8a20cc4f88b794a8f6726d975e8ffbe Reviewed-by: Matti Paaso <matti.paaso@theqtcompany.com>
* Update license headers and add new license filesMatti Paaso2014-09-241-19/+11
| | | | | | | | | - Renamed LICENSE.LGPL to LICENSE.LGPLv21 - Added LICENSE.LGPLv3 - Removed LICENSE.GPL Change-Id: Iec3406e3eb3f133be549092015cefe33d259a3f2 Reviewed-by: Iikka Eklund <iikka.eklund@digia.com>
* Add support for blacklisting test functionsLars Knoll2014-09-211-2/+15
| | | | | | | | | | | | | | | | | We need to have a finer grained control over the tests we skip in our CI system. This adds a blacklisting mechanism that allows blacklisting individual test functions (or even test data) using a set of predefined matching keys for the operating system and some other relevant variables. QTestlib will search for a file called BLACKLIST in the test directory and parse it if found. The file contains a simple ini style list of functions to blacklist. For details see qtestblacklist.cpp. Change-Id: Id3fae4b264ca99970cbf9f45bfb85fa75c1fd823 Reviewed-by: Lars Knoll <lars.knoll@digia.com>
* Fix auto test failure output on QNXBernd Weimer2014-05-191-8/+9
| | | | | | | | | On QNX mbstowcs does not ignore last parameter (max) when first (dest) is NULL. Set it to sufficiently large value to yield proper results on QNX. Other platforms (standard libraries) will ignore this value anyway. Change-Id: Ie4695254d45082e151a052bf16de684af3b1ba1e Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* Remove duplicate symbol name: QTest::currentAppNameThiago Macieira2014-04-301-4/+4
| | | | | | | | | | It's declared in qtestcase.h as a function, so let's not declare the one in qtestresult.cpp as a static variable. None of the variables in qtestresult.cpp need to be in the QTest namespace, but we don't need to change them now. Change-Id: If6cc34642fdfe3ccda3b8cea7d053ead0db9ccbd Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
* Fix the alignment for non-ASCII stringsThiago Macieira2014-01-221-2/+4
| | | | | | | | | | | | | | | | | | d040681b6f3d03b349e9b9487fe89a611d03ee3c added support for aligning the test results for easier side-by-side comparison of the actual and expected values. However, it didn't take into account multibyte strings. That is, we would see: FAIL! tst_testcase::testcase: Compared values are not the same Actual (QString("é")): F0O Expected (expected) : FOO We use mbstowcs (multibyte string to wide char string) that calculates the length in wide chars of the output string. That's roughly equivalent to QString::fromLocal8Bit(string).toUcs4().size(). Change-Id: Ic2649951c50e05143da32a7fbef00a01e385c542 Reviewed-by: Jason McDonald <macadder1@gmail.com>
* Expose QTest::currentAppName() and remove hard-coded argv[0] in testsTor Arne Vestbø2013-09-261-5/+5
| | | | | | | | Except where we're actually testing QCoreApplication::applicationName() and friends. Change-Id: I25514884c11f43a4f82b1f818f822dc3d79f69a3 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@digia.com>
* TestLib: align the colons so the "actual" and "expected" align tooThiago Macieira2013-07-211-3/+5
| | | | | | | | | | | | | | | Before: FAIL! tst_testcase::testcase: Compared values are not the same Actual (actual): F0O Expected (expected): FOO Now: FAIL! tst_testcase::testcase: Compared values are not the same Actual (actual) : F0O Expected (expected): FOO Change-Id: I6f0768e4ef53e065b85a56879cecbad06fa34aef Reviewed-by: Jason McDonald <macadder1@gmail.com>
* Update copyright year in Digia's license headersSergio Ahumada2013-01-181-1/+1
| | | | | Change-Id: Ic804938fc352291d011800d21e549c10acac66fb Reviewed-by: Lars Knoll <lars.knoll@digia.com>
* Change copyrights from Nokia to DigiaIikka Eklund2012-09-221-24/+24
| | | | | | | | Change copyrights and license headers from Nokia to Digia Change-Id: If1cc974286d29fd01ec6c19dd4719a67f4c3f00e Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Sergio Ahumada <sergio.ahumada@digia.com>
* testlib: Remove obsolete internal compare_helper overload.Jason McDonald2012-03-211-9/+0
| | | | | Change-Id: Ic98faf360a713ac698f9bf1ff8aaad5a4c5c176b Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* testlib: Improve formatting of QCOMPARE failure messages.Jason McDonald2012-03-121-1/+1
| | | | | | | | | | | Make the various versions of the failure message align consistently so that it's a little easier to compare the actual and expected values. Of course, the value won't align nicely unless the "actual" and "expected" strings are the same length, but at least this commit makes that consistent across all versions of the message. Change-Id: If9ce231df3b5d279a06f6458fdb5da0aa4586068 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* testlib: Make QTestResult::verify() more robust.Jason McDonald2012-03-081-1/+3
| | | | | | | | | | The statementStr parameter should always be non-null, so assert if it is null. The description parameter can be null in some cases (particularly when the verify is successful, and thus no error description is going to be displayed), so tolerate this. Change-Id: I87b416d5f3b793bc608cd4aca14a4f7fe7527488 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* testlib: Improve verbose and XPASS outputJason McDonald2012-03-081-9/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, verbose (-v2) and XPASS test output showed all QCOMPAREs as "COMPARE()", making it impossible to see what was compared and difficult to match the output to the source of a test containing many calls to QCOMPARE. This commit changes testlib's internal compare_helper API so that string representations of the compared expressions are always passed to QTestResult::compare() when available, and can thus be shown in the verbose and XPASS output. The XPASS output has also been changed to state explicitly that the comparison succeeded unexpectedly, bringing it in line with the XPASS output resulting from a call to QVERIFY. This commit also changes all calls to compare_helper() to call the eight-argument version of the function, which simplifies much of the calling code. The now obsolete four-argument version of compare_helper() has been changed to output a warning that it is obsolete. It will be removed once other modules have had some time to catch up. The improved XPASS and verbose output is demonstrated by the expectfail and verbose2 selftests. Change-Id: I8baa46d5dd30e6c43b26f366c34dc5b64aab5f7c Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* testlib: make XPASS output slightly less confusing.Jason McDonald2012-02-201-1/+4
| | | | | | | | | | | | | | Prior to this commit, when an unexpected pass occurred the test output showed the message "XPASS : tst_foo::function() 'expr' returned FALSE", where the problem was actually that the expression evaluated to true when it was expected to be false. This commit changes the output to make it clear that the expression evaluated to true unexpectedly. Task-number: QTBUG-22118 Change-Id: Id22c178073d3b75789675ca37a8ef019029b1f91 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Remove code related to test location.Jason McDonald2012-02-161-15/+0
| | | | | | | | Testlib no longer does anything with the test location, and neither do any of Qt's tests, so the code is no longer needed. Change-Id: I1a464116179f3e26efadad6901f00f3c33e0ce84 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* testlib: Count passes, fails and skips consistently.Jason McDonald2012-02-131-13/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For data-driven tests, testlib previously counted one fail or skip for each data row that failed or skipped, while it counted only one pass for a test function where all rows passed and counted no passes for a test function where some rows passed and some rows failed. A similar problem also existed for benchmark tests, which could run multiple iterations of the same test, with each fail and skip being counted but only a single pass being counted for the entire series of iterations. This commit makes testlib count one pass, fail or skip for each data row. Test functions that are not data-driven count one result for the test function, as before. Benchmark tests count one pass, fail or skip per iteration. A side-effect of this change is that the test output in plain text, xml and light xml formats now shows a result for every data row and benchmark iteration executed, allowing post-processors to correctly calculate the total number of tests executed. Previously, individual rows were not shown in the test output if they passed, making such calculations impossible. The only change to the xunitxml output format is to correct a bug where no test result was recorded for a test function if the last data row was skipped and all other rows passed -- in which case the overall result should be a pass. Note that there is also a pre-existing bug in the xunit logger, where no result is reported if all rows are skipped; that bug is unaffected by this commit. Task-number: QTBUG-21848 Task-number: QTBUG-22124 Change-Id: I7e17177e10d6e89e55b9684c159bd506f21d002b Reviewed-by: Ed Baak <ed.baak@nokia.com>
* testlib: Clear ignored messages after every data rowJason McDonald2012-02-081-4/+5
| | | | | | | | | | | | | | | | | | | | Previously, ignored messages were only cleared at the end of each test function, i.e. after all data rows were finished. This meant that if a data row in a data-driven test function didn't cause all of the expected messages to be generated, the remaining messages would be carried over to the next data row. This would result in errors about missing messages being associated with the last data row rather than with the correct data row. This commit makes testlib check for missing ignored messages after running each data row rather than only doing so after the last data row. This commit also adds a regression test to demonstrate that ignored messages can no longer be carried over from one data row to another. Change-Id: Ibee51aa6e96866fbcbcb4acee1a8340a86a6a4ba Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* testlib: Clear expected failures after every data row.Jason McDonald2012-02-081-2/+7
| | | | | | | | | | | | | | | | | | | | | | Previously, expected failures were only cleared at the end of each test function, i.e. after all data rows were finished. This meant that if a data-driven test function called QEXPECT_FAIL and then didn't perform any further verification steps to trigger the expected failure, the expected failure would be carried over to the next data row, probably causing the first verification step in the test function to XPASS (with a seemingly irrelevant error message) for the next data row. This commit adds the new function QTestResult::finishedCurrentTestData() to cleanup after each data row is executed. This function treats calls to QEXPECT_FAIL without subsequent verification steps as a test failure. This commit also adds a regression test to demonstrate that expected failures can no longer be carried over from one data row to another. If run against the previous version of testlib, the new test would report a pass instead of an error. Change-Id: Ida5c7f080815b0dca9531131fed582b0918334cb Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* CodeCoverage: Save coverage data in QTestLog::stopLogging()Caroline Chao2012-02-071-0/+12
| | | | | | | | | | | | | | | | | Currently when tests are crashing, aborting or hanging, their status are reported with a status "Unknown" because SaveCoverageTool is never called for them. The status of the test given by the coverage tool should be the same as the one reported in the log output when the test is run. This change will allow more accuracy in the status reported. Add QTestResult::setCurrentAppname and QTestResult::currentAppname. To retrieve the name of the current application running. Task-number: QTQAINFRA-460 Change-Id: Icc476dc2d6cb28185e5447f1e79da6a8a31cad54 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com> Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-07 12:44:45 +0000