summaryrefslogtreecommitdiffstats
path: root/src/testlib/qtestcase.h
Commit message (Collapse)AuthorAgeFilesLines
* Extend QTest::failOnWarning() to a no-parameter fail-on-any-warningEdward Welbourne34 min.1-0/+1
| | | | | | | | | | | | | | | | | | | | Many users (albeit mostly in qtdeclarative) of failOnWarning() are passing a catch-all regexp, which is only supported when regular expression support is enabled. Make their lives easier and those checks independent of the feature by adding a third overload, taking no parameter, that fails on any (unanticipated) warning. Implementation is trivial - just put a null QVariant in failOnWarningList; it won't match either check on the entry that might exempt a test from failing on a warning, so any warning at all will trigger failure. [ChangeLog][QtTest] QTest::failOnWarning() now has a no-parameter overload to support the common case of fail-on-any-warning, without needing to construct a match-everything regular expression. Change-Id: Ic693f1c8619fd6e495543b85737d566134cf9d20 Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
* QTest: replace the majority of the ugly QCOMPARE_xx macroThiago Macieira10 days1-19/+43
| | | | | | | | | | | | | | | | | | Using templates to make the code look nicer and be more debuggable. Previously, if you tried to step into the QCOMPARE_xx, you'd have the creations of qxp::function_ref from the lambdas before you got to the actual comparison. And all of this was in-place from macro expansion, so you couldn't tell what was happening. Now, if you step into QCOMPARE_xx, you go to qCompareOp. There are 5 frames between that point and the actual comparison (four std::forward() and the Compare::compare()) and debuggers could step over std::forward. Even if they can't, both libstdc++ and libc++ implementations are simple casts, so there's nothing more to step into. Change-Id: I12a088d1ae424825abd3fffd171dbbf7adb7e7e2 Reviewed-by: Ivan Solovev <ivan.solovev@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* QTest: rip out qxp::function_ref from compare_helper()Thiago Macieira10 days1-14/+45
| | | | | | | | | | | | | | | | [ChangeLog][QtTest] The QCOMPARE_xx macros can now only find QTest::toString() expansions that are either found via Argument Dependent Lookup on the type in question or are an instatiation of the QTest::toString<T>() template. This matches the behavior of the QCOMPARE() macro. This changes the way how the toString() overloads are selected, so we need to explicilty constraint the main QTest::toString() template in order to pick the free functions when they exist. Change-Id: Ie28eadac333c4bcd8c08fffd17c54e768c5cffd0 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
* QTest: rip out qxp::function_ref from reportResult()Thiago Macieira10 days1-2/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is causing huge code bloat because everything is a local lambda. Instead, pass direct type-erased function and data pointers to the replacement function. Testing with tst_qcborvalue, this reduces the compilation time and the output binary size significantly: Before After Compiler Time Size Time Size GCC 13.2 136.99 s 202.3 MB 13.88 s 14.3 MB GCC 14.0 131.49 s 202.7 MB 14.69 s 14.4 MB Clang 17 77.2 s 146.7 MB 13.62 s 12.2 MB Clang 18 141.9 s 187.1 MB 13.62 s 12.4 MB This causes a difference in how toString() overloads are found. Previously it would match far more overloads because the toString() calls were expanded by the macro. Now, we depend on Argument-Dependent Lookup and associated namespaces, so toString() overloads should not be in the QTest namespace any more. With this patch applied, the testlib testcase of tst_selftest started failing, because nullptr is now handled differently. However, I consider it as a bugfix, because previously it was falling back to a default implementation, and now it is using the QTest::toString(std::nullptr_t) overload, which is a desired behavior. Update the reference files for tst_selftest with the new expected output. Task-number: QTBUG-124272 Change-Id: Ie28eadac333c4bcd8c08fffd17c5484186accdf6 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
* QTest: group all the toString() functionality in a single headerThiago Macieira10 days1-95/+1
| | | | | | | | | It was split between qtest.h and qtestcase.h, so the QTest::toString() specializations were not available in qtestcase.h. And it was confusing. Change-Id: Ie28eadac333c4bcd8c08fffd17c54fafc8014cc7 Reviewed-by: Ivan Solovev <ivan.solovev@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* QTest: add removed_api.cpp and move deprecated functions to itThiago Macieira10 days1-3/+0
| | | | | | | | | | | | | QtTest is not under the same Binary Compatibility guarantees as the other modules, but let's try and do it where it's easy. The toString() method can simply be removed from the header because it's not a member function. Its deprecation wasn't marked with a version number, so it starts now with 6.8. Change-Id: Ie28eadac333c4bcd8c08fffd17c54faca7057b9c Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
* QTest: expose API needed for Qt Quick Test to print crash backtracesMitch Curtis2024-04-041-0/+3
| | | | | | | | | | | | | | | | | | | | | | | | | As discussed in QTQAINFRA-6146, there were two potential approaches to enabling backtraces in Qt Quick tests: - Either via a thorough refactoring of quicktest.cpp so that each testcase is a Slot in a programmatically-created QObject-derived class, which will be executed via QTest::qExec() similar to the documented QTest way. This way the pre-existing code in QTest::qRun() will setup the fatal signal handler. - Or as a quick fix, modify quick_test_main_with_setup() to setup a FatalSignalHandler class and invoke prepareStackTrace() like it's already done in QTest::qRun(). This would require exporting these symbols in the private header. This patch enables the implementation of the latter, as it has a fairly light footprint, is easily revertable (should we need to), and allows us to immediately gain insight into crashes in CI. Task-number: QTQAINFRA-6146 Change-Id: Iedffc968acb3e570214df34884ab0afcb6b30850 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
* QTest: rename local `timeout` variable in QTRY_IMPL macroVolker Hilsheimer2024-02-091-3/+3
| | | | | | | | | | | | | `timeout` is a likely name for a variable in the scope in which a QTRY_ macro is used, so don't use that name in the macro itself. The other variables are all prefixed with `qt_test_`, so do that here as well, and be explicit about what the variable stores. Amends d4bb448cddce63e0c6a84a86020fa59dd32b2293. Task-number: QTBUG-121746 Change-Id: If05dccdc24a66e95a08156b820d185f184783ad6 Reviewed-by: Marc Mutz <marc.mutz@qt.io>
* QTest: allow passing chrono literal as QTRY_ timeoutMarc Mutz2024-02-071-12/+17
| | | | | | | | | | | | | | | | | | | By wrapping the use of the timeout value in QTRY_IMPL in a lambda that feeds the user input through the std::chrono::milliseconds constructor with std::chrono_literals in-scope, the macros continue to work with raw integral values as well as chrono literals not finer than millisecond granularity. Port all higher-level macros to pass a chrono literal and port some uses in tst_selftests. [ChangeLog][QtTest] The QTRY_*_WITH_TIMEOUT macros now also accept chrono literals (was: int milliseconds). Fixes: QTBUG-121746 Change-Id: Ib38406fc005a0a2c4ae3fd009760f73ad8bed355 Reviewed-by: Ivan Solovev <ivan.solovev@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* QTest: add opt-in changing QCOMPARE etc to exit with throw, not returnMarc Mutz2024-01-271-0/+55
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add exception classes and use them to change the control flow for QTEST_{FAIL,SKIP}_ACTION from return'ing from just the immediate function to the full way to the QTestLib infrastructure, here we filter them out. There are three modes: - If QT_NO_EXCEPTION, then we return - If QTEST_THROW_ON_... is also defined, #error out - Otherwise, if QTEST_THROW_ON_... is defined, always throw - Otherwise, the decision is made at runtime (with defaults read from QTEST_THROW_ON_... environment variables). Three selftests depend on the old behavior, as they explicitly check that multiple FAIL SKIP etc emerge, which the new framework, of course, prevents. Locally disable throwing at the test function level. Add initial docs and enable exceptions in all of the selftest subprograms to facilitate switching between the two runtime-selectable modes. [ChangeLog][QtTest] Added QTEST_THROW_ON_FAIL and QTEST_THROW_ON_SKIP C++ macros and environment variables that, when defined, change how QCOMPARE/QVERIFY/QSKIP etc exit the test function on failure. Instead of a return, exiting only the immediately-surrounding function, they throw a special exception instead, thereby exiting from subfunctions of the test function, all the way to QtTestLib. Fixes: QTBUG-66320 Change-Id: I96c38d2a1dcdd9de84942cf448a8bbf3ab6d3679 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
* QTestLib: prepare QVERIFY_THROWS_EXCEPTION for FAILED_ACTION = throwMarc Mutz2024-01-251-1/+4
| | | | | | | | | | | | | | | | | | | We can't have a QTEST_FAILED_ACTION in a try-block if we intend to allow QTEST_FAILED_ACTION to be a throw statement. We could, of course, add a whitelisting catch-block for the eventual TestFailedException, but as of now, the intent it to give users full control over the definition of QTEST_FAILED_ACTION, and they may have their own idea of what to throw. So add a bool variable to record whether no exception was thrown and use it to drag QTEST_FAILED_ACTION out of the try block. It's ok to be in catch blocks, no action needed there. Change-Id: I0b004e43b1db82cd8b5b12f900ed985e58a56807 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
* QTest: replace naked returns with QTEST_{FAIL,SKIP}_ACTION macrosMarc Mutz2024-01-251-12/+20
| | | | | | | | | | | | | ... defaulting to "return". This allows customizing these actions, incl. to eventually make them throwing exceptions instead (but that won't work for QVERIFY_THROWS_EXCEPTION, yet). Change-Id: I078a4ce48135bda2cf98fce78318a12d757d7aa5 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* QTest: Extract Method qCaught() from THROWS_(NO_)EXCEPTIONMarc Mutz2024-01-231-18/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | We've already done a lot in 59600a514ba99ed62b46237d8f160dea84474190, but we can do more: Thanks to std::exception_ptr, we can drag the handling of unexpected exceptions completely to out-of-line code, leaving only the catch(...) as inline. As a nice side-effect, we no longer need the nested try blocks in QVERIFY_THROWS_EXCEPTION to work around GCC -Wexceptions or MSVC C2312 complaining about two handlers for the same exception type (which can happen when exceptiontype is std::exception itself). This may not handle __cxxabi::__forced_unwind correctly, but it doesn't need to: Tests should not need to call THROWS_EXECPTION on code that could emit a pthread cancellation; tests are controlled environments. Keep the old qCaught() function for BC. We don't promise BC in QtTestLib, but it doesn't cost much. Pick-to: 6.7 Change-Id: I1e1627c6341b09197a8a79669fde061c47e6ba47 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Remove unneeded toString(QPair) declarationMarc Mutz2023-12-061-3/+0
| | | | | | | | | | | | It's literally the same as the std::pair one that immediately follows it, because QPair these days is just a template alias for std::pair. Wasn't flagged as an error because it's just a declaration, not a definition. Pick-to: 6.6 6.5 Change-Id: I5b47572f6e2a71edb47cbe224801a719ff1e060d Reviewed-by: Jason McDonald <macadder1@gmail.com>
* QCOMPARE: restore compatibility with braced-init-listsGiuseppe D'Angelo2023-07-071-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | a611a9f537f1187825339c2a2214c8ec4a23680f (in Qt 5) added support for mixed-type comparisons through QCOMPARE. That commit added a new overload for qCompare taking two types, T1 and T2; but it also left the same-type qCompare(T, T) overload around, guarded by a Qt 6 version check. The mixed-type version is however not a generalization of the same-type one, because it won't work if one of the arguments doesn't participate in FTAD. Case in point: braced-init-lists. In Qt 5 this worked: QCOMPARE(some_container, {42}); but in Qt 6 it does not work any more. The mixed-type overload fails deduction (can't deduce T2); in Qt 5 the same-type overload deduced T=SomeContainer, and {42} was used to select a constructor for SomeContainer. -- There's a partial, straightforward workaround for this: default T2 to T1 in the mized-type overload. In that case T2 has a "fallback" if it cannot be deduced. This is partial because of course doesn't address the case in which T1 cannot be deduced, but I don't think that is common at all. QList is special here, because it has qCompare overloads that makes it comparable with arrays, initializer lists and more. I don't like that very much -- we should probably have a qCompare(input_range, input_range) overload, but that's an exercise for C++20. Change-Id: I344ba33167829984978cd8d649a1904349a9edab Pick-to: 6.5 6.6 Reviewed-by: Marc Mutz <marc.mutz@qt.io> Reviewed-by: Milian Wolff <milian.wolff@kdab.com> Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
* Silence compiler warning about unreachable codeVolker Hilsheimer2023-06-301-2/+3
| | | | | | | | | | | | | | | | | | The warning has been reported by users, but never observed in CI, even though the only path that can lead to the last return statement becoming unreachable is very common in Qt: if a QDebug stream operator exists, then we always return early. Nevertheless, rewrite the code to have a single return statement. Task-number: QTBUG-112371 Fixes: QTBUG-114944 Pick-to: 6.6 6.5 Change-Id: Iaf9ec683ceceedb00771fb0743a09dcc8f50ba3f Reviewed-by: Axel Spoerl <axel.spoerl@qt.io> Reviewed-by: Pavel Dubsky <pavel.dubsky@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Jøger Hansegård <joger.hansegard@qt.io>
* Move the formatting of <chrono> durations to QDebug & QtTestThiago Macieira2023-05-061-0/+3
| | | | | | | | | | | | [ChangeLog][QtCore][QDebug] Added pretty formatting of C++ <chrono> durations. [ChangeLog][QtTest] Added pretty formatting of C++ <chrono> durations for QCOMPARE expressions. Change-Id: I3b169860d8bd41e9be6bfffd1757cc087ba957fa Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
* QTest: Rename QCOMPARE_XX parameters to add meaningMårten Nordheim2023-03-081-27/+27
| | | | | | | | | | | | | | | The parameter names were previously "lhs" and "rhs", which is incredibly abstract for a testing framework. One of the parameters will tend to be a baseline value to compare against while the other is the result of some action we want to test. Thus I suggest they be renamed "computed" and "baseline". This way we can, hopefully, retain the semantic that the 'left'/first argument is the computed ('actual' in QCOMPARE) value while the 'right'/second argument is the baseline ('expected' in QCOMPARE.) Change-Id: I3e0fdce2a3f1faca06fdf7184ef6e0eb9724d990 Reviewed-by: Ivan Solovev <ivan.solovev@qt.io>
* Add QTest::currentTestResolved() and use in testlibEdward Welbourne2022-09-221-3/+4
| | | | | | | | | | | | | | | | | | The QTRY_* macros and QTestEventLoop have (since 6.3) been exiting their loops early if the test has failed. Where that was appropriate, they should also have been exiting early on skip. [ChangeLog][QtTest] Added QTest::currentTestResolved(), which is true if the test has failed or skipped. The QTRY_*() macros and QTestEventLoop now use this, rather than QTest::currentTestFailed(), to test whether they should stop looping, so that they also do so on a skip. Task-number: QTBUG-104441 Change-Id: Ibf3d5a095b35e6670bc3daf756f05b66f7f3ef9b Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Mitch Curtis <mitch.curtis@qt.io> Reviewed-by: Jason McDonald <macadder1@gmail.com>
* Skip early return from test loops during cleanup()Edward Welbourne2022-09-221-5/+7
| | | | | | | | | | | | | | | | The QTRY_* macros and QTestEventLoop exit early if the test has resolved; however, in the cleanup phase of a test, even if the test has failed, these loops should continue as normal. [ChangeLog][QtTest] During the cleanup() phase of a test, the QTRY_* macros and QTestEventLoop now ignore the test resolution, in contrast to when they are used from the test itself, which (since 6.3.0) exits the loops early if the test has failed. Pick-to: 6.4 6.3 Fixes: QTBUG-104441 Change-Id: I2673161967cbbc57815155af698a9338ab98a686 Reviewed-by: Jason McDonald <macadder1@gmail.com>
* Fix warning from testing assignmentVolker Hilsheimer2022-09-031-1/+1
| | | | | | | | | | | | | | | | | | Tests that do QTRY_VERIFY(a = b); generate a compiler warning: using the result of an assignment as a condition without parentheses [-Wparentheses]: place parentheses around the assignment to silence this warning Do that centrally in the QTRY_TIMEOUT_DEBUG_IMPL macro definition. Pick-to: 6.4 Change-Id: I4a4b0161c4e16c0e1e27a68f33a41efdaa2c962c Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* Implement the batch_tests featureMikolaj Boc2022-08-241-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | An approach of test batching (joining multiple tests into a single binary) has been taken, due to long linking times/binary size on certain platforms, including WASM. This change adds a new feature 'batch_test_support' in Qt testlib. Based on the value of the feature, test batching may become enabled with the -batch-tests switch. Batching works for every target added via qt_internal_add_test. When first such target is being processed, a new combined target for all of the future test sources is created under the name of 'test_batch'. CMake attempts to merge the parameters of each of the tests, and some basic checks are run for parameter differences that are impossible to reconcile. On the C++ level, convenience macros instantiating the tests are redefined when batch_tests is on. The new, changed behavior triggered by the changes in the macros registers the tests in a central test registry, where they are available for execution based solely on their test name. The test name is interoperable with the names CMake is aware of, so CTest is able to run the tests one by one in the combined binary. Task-number: QTBUG-105273 Change-Id: I2b6071d58be16979bd967eab2d405249f5a4e658 Reviewed-by: Topi Reiniö <topi.reinio@qt.io>
* Skip some spurious parentheses on macro argument usesEdward Welbourne2022-07-251-10/+10
| | | | | | | | | | | | | | When an argument to one macro appears as a single argument to another macro, the usual "put it in parentheses" rule doesn't apply since it's necessarily something the right shape to be a macro parameter. In some cases, furthermore, the added parentheses could show up in the output produced by a test when reporting an expression that was compared or verified. Also removed the parentheses around one comparison. Pick-to: 6.4 Change-Id: I226f4356fb057351187dac2134cc12c06026c40c Reviewed-by: Ivan Solovev <ivan.solovev@qt.io> Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
* Skip semicolon after uses of QTRY_IMPL() and QTRY_LOOP_IMPL()Edward Welbourne2022-07-251-6/+8
| | | | | | | | | | | | The macros' expansions end with an if-block and a for-block, respectively, so the following semicolon is superfluous. Since we control these macros' use (they're internals), just skip the semicolons, rather than wrapping their bodies in do-while(0). Pick-to: 6.4 Change-Id: I53f7786a66a0dc7709ac9f96d49985edafceec39 Reviewed-by: Ivan Solovev <ivan.solovev@qt.io> Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
* Move the early-failure checks in QTRY_IMPL()Edward Welbourne2022-07-251-6/+8
| | | | | | | | | | | These checks should not be included in the #expr reported when the test fails. They're an internal implementation detail of the loop. Split a long line while I was about it. Pick-to: 6.4 Change-Id: Iaea0478967d01cd72ef5a5e9a6501d4be2324b18 Reviewed-by: Ivan Solovev <ivan.solovev@qt.io> Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
* Use debug stream in QTest::toString's default fallback if possibleVolker Hilsheimer2022-07-131-2/+11
| | | | | | | | | | | | | | | | | | | | For built-in types, this is a compile-time assert - we should not have any types in Qt for which we have neither debug streaming nor a QTest::toString specialization implemented. A build of most of Qt submodules passes with this change, after minor modifications to some tests. We cannot declare QSizeHint::Policy as a metatype after the QMetaType has already been instantiated for it, and the QDebug stream operator for QElaspedTimer needs to be correctly declared within the namespace. Add a self-test function for a custom type, and update reference files of the self-test. Task-number: QTBUG-104867 Pick-to: 6.4 Change-Id: I2936db5933f4589fce45f47cf2f3224ed614d8c9 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
* testlib: Don't print QCOMPARE values if they lack string representationVolker Hilsheimer2022-07-101-11/+0
| | | | | | | | | | | | | | | | | | | | | | | Before 0681a2dd5a8095baddb5905fb21a58ce19b958c5, QCOMPARE'ing types for which no QTest::toString specialization exists did not output Actual and Expected lines on failure, as that would only print <null> for both values (which then look like the same value, confusingly). Commit 0681a2dd5a8095baddb5905fb21a58ce19b958c5 changed that behavior, and started printing the confusing <null> values. Take care of the logic in the formatFailMessage function: if both values are nullptr, then print only the variable names, but not the confusing <null> text representation of the values. Remove dead and duplicated code related to the formatting logic, add a self-test function, and update the expected_cmptest files. Fixes: QTBUG-104867 Pick-to: 6.4 Change-Id: I4be98e79f91196b14690a2cc0a68ffd50b431a45 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* QTest: switch some text-conversion functions to use qsizetypeMårten Nordheim2022-06-281-2/+2
| | | | | | | | | To avoid potential narrowing. Pick-to: 6.4 Task-number: QTBUG-104125 Change-Id: I37bfc5c49e7c919f5204a76a905758a92527d864 Reviewed-by: Marc Mutz <marc.mutz@qt.io>
* QTestLib: rework QTest::compare_helper()Ivan Solovev2022-06-031-8/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | [ChangeLog][QTestLib] QCOMPARE now evaluates toString() on its arguments lazily, speeding up the general case where the comparison doesn't fail. This is true for the QCOMPARE functionality provided by Qt. If you specialized qCompare() for your own types, then you need to change its implementation in line with Qt's own qCompare() specializations in order to enable this feature. [ChangeLog][QTestLib] QCOMPARE calls with nullptr argument(s) will now print the actual and expected values upon failure. Previously it was not like that because of the compareHelper() overload in qtestresult.cpp that treated the presence of nullptr-arguments as a reason to ignore formatFailMessage() call. New implementation does not have this check, and correctly executes formatFailMessage() for all arguments. Note that the qCompare() overloads that call QTestResult::compare() internally were not affected by this patch, because they already defer toString() invocation until the comparison fails. Some numbers, collected against shared release developer build. I checked how this change affects the test execution. The idea was to pick some tests for types that do not have a specific QTestResult::compare overload, so I picked a couple of QByteArray tests. The comparison is done by running a test 10 times and taking the average execution duration, as reported in the log. tst_qbytearrayapisymmetry: Before: 15.6 ms After: 14.2 ms tst_qbytearray: Before: 41 ms After: 36 ms The benefit is around 9% and 12% respectively. Fixes: QTBUG-98874 Change-Id: I7d59ddc760168b15974e7720930f629fb34efa13 Reviewed-by: Marc Mutz <marc.mutz@qt.io>
* Add QTRY_COMPARE_{EQ,NE,LT,LE,GT,GE}_WITH_TIMEOUT()Ivan Solovev2022-06-031-0/+36
| | | | | | | | | | | | [ChangeLog][QTestLib] Add QTRY_COMPARE_{EQ,NE,LT,LE,GT,GE}_WITH_TIMEOUT macros that repeatedly execute QCOMPARE_{EQ,NE,LT,LE,GT,GE} until either the comparison returns true or the timeout expires. Also add QTRY_COMPARE_{EQ,NE,LT,LE,GT,GE} macros that simply invoke the *_WITH_TIMEOUT versions with the usual timeout of five seconds. Task-number: QTBUG-98873 Change-Id: Ib0d7d1c8c997f442b46acd85da738a8f512cc875 Reviewed-by: Marc Mutz <marc.mutz@qt.io>
* Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}()Ivan Solovev2022-06-031-0/+32
| | | | | | | | | | | | | | | | | | | | | [ChangeLog][QTestLib] Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}() macros. These new macros behave similarly to QVERIFY(a op b), where 'op' is ==, !=, <, <=, >, >= respectively, but print a formatted error message with argument values in case of failure. The formatting is done lazily, which means that the strings will be generated only when the comparison fails. Also add a new test for tst_selftest and generate expected output for it. Fixes: QTBUG-98873 Task-number: QTBUG-98874 Change-Id: Ic8074798901d7a469b1f58d5cd28bbf49a3da1db Reviewed-by: Marc Mutz <marc.mutz@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Use SPDX license identifiersLucie Gérard2022-05-161-38/+2
| | | | | | | | | | | | | Replace the current license disclaimer in files by a SPDX-License-Identifier. Files that have to be modified by hand are modified. License files are organized under LICENSES directory. Task-number: QTBUG-67283 Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
* QtTestLib: replace remaining uses of QL1String with QL1StringViewSona Kurazyan2022-05-021-4/+4
| | | | | | | | | Remove unneeded \fn qdoc lines as a drive-by. Task-number: QTBUG-98434 Change-Id: Id93ddbb38b97a8f5a6734bfbc82686ccb3a87aa6 Reviewed-by: Marc Mutz <marc.mutz@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Fix QTest::toString() over QT_TYPESAFE_FLAGS QFlagsMarc Mutz2022-03-041-2/+2
| | | | | | | | | | | | | | QT_TYPESAFE_FLAGS allows explicit casts only to QFlags<T>::Int, which is either int or unsigned int. The cast to the resp. other type fails. To fix, first convert to QFlags<T>::Int with toInt(), and only then cast to int or unsigned int. Fixes: QTBUG-101399 Pick-to: 6.3 Change-Id: Ie74d53adc601cdf19708265b040092780676058f Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Giuseppe D'Angelo <giuseppe.dangelo@kdab.com>
* Replace out-of-date comment on QTRY_IMPL(), about qWaitFor()Edward Welbourne2021-12-091-3/+9
| | | | | | | | | | | | | | | | | | A comment on QTRY_IMPL() helpfully mentioned the bug number of the issue that prevented us from using qWaitFor; and MSVC < 2017 are no longer supported so it looked possible to make the change suggested. However, the implementation has, in the mean time, grown some features not (currently) provided by qWaitFor(), so update the comment. In the process, fix one minor error in QTRY_TIMEOUT_DEBUG_IMPL() where parentheses enclosed the wrong text; (2 * timeoutValue) would not produce the intended result if timeoutValue were 1 + 500, for example. The macro parameter should be enclosed, not the expression using it as parameter to a further macro. Task-number: QTBUG-59096 Change-Id: Ie16e297def9d1f983cd5964bc628b2c3c0aaecda Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* QTest: fix grammar in static_assert()Marc Mutz2021-11-271-2/+2
| | | | | | Change-Id: I6fd408702bb43e37434b382cb6cb9057ca192064 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* Long live QVERIFY_THROWS_NO_EXCEPTION!Marc Mutz2021-11-261-2/+19
| | | | | | | | | Counter-part to QVERIFY_THROWS_EXCEPTION. [ChangeLog][QTest] Added QVERIFY_THROWS_NO_EXCEPTION macro. Change-Id: Ib6a80c8e810d5e2298ff00d608dae04e7a0c3e8f Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* QTest: de-inline QVERIFY_THROWS_EXCEPTION message formattingMarc Mutz2021-11-261-5/+4
| | | | | | | | | | | | Extract Method QTest::qCaught() to take the string handling out of the header. This should help a bit in speeding up compilation of large unit test files (provided they use QVERIFY_THROWS_EXCEPTION), although I have no data to support that. Since we changed the error message, update the selftest accordingly. Change-Id: Id4a3c8c34d5df8d0c7a861106d269097f4a6de5c Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Add QTest::failOnWarningMitch Curtis2021-11-261-0/+4
| | | | | | | | | | | | | | This solves the long-standing problem of not being able to easily fail a test when a certain warning is output. [ChangeLog][QtTest] Added QTest::failOnWarning. When called in a test function, any warning that matches the given pattern will cause a test failure. The test will continue execution when a failure is added. All patterns are cleared at the end of each test function. Fixes: QTBUG-70029 Change-Id: I5763f8d4acf1cee8178be43a503619fbfb0f4f36 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Long live QVERIFY_THROWS_EXCEPTION!Marc Mutz2021-11-261-2/+12
| | | | | | | | | | | | | Use variable args macros to swallow any extra commas in the expression. To use this, the type of the exception has to be first. Use Eddy's suggestion for a new name to avoid breaking the old macro. [ChangeLog][QtTest] Added QVERIFY_THROWS_EXCEPTION, replacing QVERIFY_EXCEPTION_THROWN, which has therefore been deprecated. Change-Id: I16825c35bae0631c5fad5a9a3ace4d6edc067f83 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* QVERIFY_EXCEPTION_THROWN: re-throw unknown exceptionsMarc Mutz2021-11-251-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | Swallowing unknown exceptions is dangerous business, as the exception might be a pthread cancellation token, the swallowing of which would terminate the program. Instead of returning from the catch-all-clause, therefore, re-throw the unknown exception. Fix tst_verifyexceptionthrown failure cases that use non-std::exception-derived true negative exceptions to not let the exception escape from the test function. As a drive-by, pretty up the macro's docs. [ChangeLog][QtTest][QVERIFY_EXCEPTION_THROWN] Now re-throws unknown exceptions (= not derived from std::exception) (was: swallowed them and returned from the test function), in order to play nice with pthread cancellation. Pick-to: 6.2 5.15 Change-Id: Ic036d4a9ed4b7683fa67e27af8bcbae0eefdd0da Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* QTest: mark qFail() as coldMarc Mutz2021-11-231-0/+1
| | | | | | | | | | Unit tests should not fail, so a call to qFail() is by definition exceptional. Therefore, mark the function as cold. It probably doesn't matter in the grand scheme of things, but it also doesn't cost much. Change-Id: I0cafcfa65ff285812155d0687deded8d26cf4efd Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Deprecate QTest::qWarn() and replace its last useEdward Welbourne2021-09-231-12/+4
| | | | | | | | The one place that actually called it with file and line number was internal to QtTest, so could call QtTestLog::warn(). Change-Id: I8be10fc81996cc3de4d2eea3ecedc59671ffe759 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* testlib: Deprecate QWARN() in favor of qWarning()Tor Arne Vestbø2021-08-041-3/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | The QtTest best practices documentations recommends using output mechanisms such as qDebug() and qWarning() for diagnostic messages, and this is also what most of our own tests do. The QWARN() macro and corresponding internal QTest::qWarn() function was added when QtTest was first implemented, but was likely meant as an internal implementation detail, like its cousin QTestLog::info(), which does not have any corresponding macro. This theory is backed by our own QtTest self-test (tst_silent) describing the output from QWARN() as "an internal testlib warning". The only difference between QWARN() and qWarning(), besides the much richer feature set of the latter, is that qWarning() will not pass on file and line number information in release mode, but QWARN() will. This is an acceptable loss of functionality, considering that the user can override this behavior by defining QT_MESSAGELOGCONTEXT. [ChangeLog][QtTest] QWARN() has been deprecated in favor of qWarning() Pick-to: 6.2 Change-Id: I5a2431ce48c47392244560dd520953b9fc735c85 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* Fix QFAIL() to interract correctly with QEXPECT_FAIL()Edward Welbourne2021-06-151-2/+2
| | | | | | | | | | | | | | | | | | | Previously, it went direct to QTestResults::addFailure() without going via the checking for expected failure. Add QTestResults::fail() to take care of this checking, as for verify() and compare(). Tidied up the code implementing expected failure and QFAIL(), while I was about it. Adjusted an existing test to verify that expecting a QFAIL() works, by using QFAIL() instead of QVERIFY(false). Remove the QVERIFY(false) whose comment brought this to my attention. [ChangeLog][QtTestLib][QFAIL] QEXPECT_FAIL() now correctly anticipates a subsequent QFAIL(). Previously QFAIL() counted as a fail regardless. Change-Id: Icc28cf70e5ff3006363791ea03aa01f2f591eb71 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* QtTest: move QTRY_X error message formatting out-of-lineMarc Mutz2021-06-151-3/+3
| | | | | | | | | | | Clazy warns about the fromUtf8() call with a constant argument, for every use of the macro, so hide the stuff behind a compiler firewall. Also fix the format injection error by using QLatin1String::arg() instead of QString::arg() chaining. Change-Id: I4bb4d4af56443540efc0c38c75819aa152a441fc Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* QTestlib: Add formatting for QObject * in QCOMPAREFriedemann Kleint2021-05-261-0/+22
| | | | | | | | | | | Output object name and class in QCOMPARE(). This should help to debug flaky QWidget tests that for example check on focusWidget(). [ChangeLog][QtTestLib] QCOMPARE() now reports QObject * values by class and objectName(). Task-number: QTBUG-64446 Change-Id: Ife04e89bba04fc78d077c8f0f07af17a17c9cf8c Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* QTestlib: Fix formatting of pointers in QCOMPAREFriedemann Kleint2021-05-231-1/+2
| | | | | | | | | | After 1ed8a7bff503aacf55f7f880ddaad461ea15e5e1, volatile needs to be specified for toString(), else it is not used by compare_ptr_helper(). Add an overload. Pick-to: 6.1 5.15 Change-Id: I3c335f324df346233623272d1014c8360ca33160 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* QTestlib: Speed up comparison of qsizetypeFriedemann Kleint2021-01-221-0/+5
| | | | | | | | | | | | | Add a qCompare() overload for qsizetype, preventing it falling back to the slow code path pre-formatting expected/actual. toString() should use the correct format from the already present int-types (see macro TO_STRING_IMPL). Complements 94aa350621e8a5c4ad3b438c10fc1c0a9ed3bc8a. Task-number: QTBUG-38890 Change-Id: Ieb8cea7de086141a2c80f93b4c1be01572be96df Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* QTestLib: Show the values of unregistered enumsChristian Kandeler2021-01-131-1/+7
| | | | | | | ... on a failed QCOMPARE. Change-Id: I653894927e49fad92c21409d03ed70880ca510f6 Reviewed-by: Jason McDonald <macadder1@gmail.com>
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-06 19:01:37 +0000