summaryrefslogtreecommitdiffstats
path: root/tests/auto/testlib
Commit message (Collapse)AuthorAgeFilesLines
...
* testlib: make it possible to test double-clicks with discrete eventsShawn Rutledge2022-10-167-6/+68
| | | | | | | | | | | | | | | | | | | | | | The timestamp will no longer be incremented by 500ms after a mouse release if the delay has been explicitly specified. The default delay is 1 ms since f5010c49a37729375e37e6fe8cea60dd4b880d83 but the running timestamp was unconditionally post-incremented by 500ms after every mouse release, to prevent double-clicks, which were always deemed as unintended (because we have a mouseDClick function for that). Now, we do that 500ms increment only if the user has not provided a delay value in the function argument at all. We have often found it useful in our own tests to generate double-clicks "the hard way", by sending indivdual events, so as to be able to check state in some target object at each step, as shown in the new snippet. [ChangeLog][QtTest] QTest::mouseRelease() and mouseClick() can now be used to test double-clicks, by specifying a realistic timestamp delay. Fixes: QTBUG-102441 Change-Id: I8e8d242061f79efb4c6e02638645e03661a9cd92 Reviewed-by: Richard Moe Gustavsen <richard.gustavsen@qt.io>
* Autotest/Unix: request zero-sized core dumps for crashing codeThiago Macieira2022-10-141-0/+8
| | | | | | | | | | | | | Unix systems have got crash loggers in the past 15-20 years, notably macOS and Linux (abrtd, systemd-coredumpd, etc.). By setting the core dump limit to zero, those tools should be mostly inhibited from running and thus not interfere with the parent process' timeouts. Even for systems without core dump loggers, disabling the writing of a core dump to the filesystem should also help. Pick-to: 6.4 Change-Id: I12a088d1ae424825abd3fffd171d112d0671effe Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* tests: skip tst_selftests on Wayland and XWaylandLiang Qi2022-10-141-0/+5
| | | | | | | | | | | | | QWindow::requestActivate() is not supported. We have one tst_selftests binary, and will test it with both xcb and wayland qpa plugin. A runtime check and skip will have different restult files, which is not implemented in testlib yet. Task-number: QTBUG-107578 Pick-to: 6.4 6.2 Change-Id: Idc8cb24c6f42a9f0f4dc9493e3fd1a5803ba7ce0 Reviewed-by: Liang Qi <liang.qi@qt.io>
* Port from container.count()/length() to size()Marc Mutz2022-10-042-18/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is semantic patch using ClangTidyTransformator: auto QtContainerClass = expr(hasType(namedDecl(hasAnyName(<classes>)))).bind(o) makeRule(cxxMemberCallExpr(on(QtContainerClass), callee(cxxMethodDecl(hasAnyName({"count", "length"), parameterCountIs(0))))), changeTo(cat(access(o, cat("size"), "()"))), cat("use 'size()' instead of 'count()/length()'")) a.k.a qt-port-to-std-compatible-api with config Scope: 'Container'. <classes> are: // sequential: "QByteArray", "QList", "QQueue", "QStack", "QString", "QVarLengthArray", "QVector", // associative: "QHash", "QMultiHash", "QMap", "QMultiMap", "QSet", // Qt has no QMultiSet Change-Id: Ibe8837be96e8d30d1846881ecd65180c1bc459af Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
* Skip early return from test loops during cleanup()Edward Welbourne2022-09-226-113/+17
| | | | | | | | | | | | | | | | The QTRY_* macros and QTestEventLoop exit early if the test has resolved; however, in the cleanup phase of a test, even if the test has failed, these loops should continue as normal. [ChangeLog][QtTest] During the cleanup() phase of a test, the QTRY_* macros and QTestEventLoop now ignore the test resolution, in contrast to when they are used from the test itself, which (since 6.3.0) exits the loops early if the test has failed. Pick-to: 6.4 6.3 Fixes: QTBUG-104441 Change-Id: I2673161967cbbc57815155af698a9338ab98a686 Reviewed-by: Jason McDonald <macadder1@gmail.com>
* Fix error in TAP test logger's regex to recognize QVERIFY() outputEdward Welbourne2022-09-222-4/+4
| | | | | | | | | | | The regex used a greedy .+ to match an actual dot and a space, preceding an open parenthesis, with the result that if the message contained any parentheses that .+ swallowed everything up to the last of them. Correct the regex so that recently-added tests' error messages show up correctly. Change-Id: I6e52c9b2a6e7959335fcddbb4266f65b589eba68 Reviewed-by: Jason McDonald <macadder1@gmail.com>
* Add test-case for TAP logging glitchEdward Welbourne2022-09-227-7/+11
| | | | | | | | | | | | The recently added test for QTestEventLoop produced a message which the TAP test-logger mis-parses. Since that message shall soon go away (when I fix the bug the new test exists to verify), modify one of cmptest's QVERIFY2() messages to trip up the same bug, so we can spot any regressions on this in future (after the imminent fix to this TAP issue). Change-Id: Ibbe9931c01c75df529c9571e2bbdbd34010dd9ec Reviewed-by: Jason McDonald <macadder1@gmail.com>
* Include global data tags in teamcity's test function namesEdward Welbourne2022-09-132-425/+435
| | | | | | | | | | | | Use the standard mechanism instead of a hand-rolled way of identifying function/data-tag combinations. Adds missing data relevant to identifying test cases, some of which previously had apparently the same name, with no hint at why the test-case was seemingly run repeatedly. Change-Id: I6225c6d1990069c94a1f1c8dbb179993b96076e7 Reviewed-by: Jason McDonald <macadder1@gmail.com> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Fix cmake warnings from using deprecated PUBLIC_LIBRARIESVolker Hilsheimer2022-09-051-1/+1
| | | | | | Pick-to: 6.4 Change-Id: I8d4f0d9e98edc4e06dda9f58e62197572b4cbdbc Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
* Change the license of all CMakeLists.txt and *.cmake files to BSDLucie Gérard2022-08-2362-62/+62
| | | | | | | Task-number: QTBUG-105718 Change-Id: I5d3ef70a31235868b9be6cb479b7621bf2a8ba39 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
* Fix DeferredFlag implementation for QTestEventLoopEdward Welbourne2022-08-121-4/+12
| | | | | | | | | | | As discovered in the expanded testing of QTRY_COMPARE() using the same class, the timer needs a context object and a slot to call. This amends commit 35ad157d88c7bfcb9b90b01111b0f43dd2e012d9 Pick-to: 6.4 6.3 Task-number: QTBUG-104441 Change-Id: I41fc23de84ce8c7d6608db0005276a2071974494 Reviewed-by: Jason McDonald <macadder1@gmail.com>
* Test QTRY_COMPARE() and expand testing of QTRY_VERIFY*()Edward Welbourne2022-08-127-36/+180
| | | | | | | | | | | | | In the process, simplify the latter while adding some actual time-variation for the QTRY_* loop to navigate round - based on the extendedcompare test's ClassWithDeferredSetter. Testing remains primitive, but is at least a bit more thorough. Pick-to: 6.4 Change-Id: I40be8fb485f3f18f0a4f4bc62ad36cccac691979 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Mitch Curtis <mitch.curtis@qt.io> Reviewed-by: Jason McDonald <macadder1@gmail.com>
* Remove the stale expected_crashes_[345].txtEdward Welbourne2022-08-123-16/+0
| | | | | | | | | | | These contain output only ever sent to stderr, which tst_selftests no longer looks at; it merely verifies stderr is empty for all tests except those expected to produce stderr output, and ignores the stderr output for these last. Change-Id: I50fee445d84c2d125e6db5303fc389e87746455e Reviewed-by: Jason McDonald <macadder1@gmail.com> Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
* Add some testing of QTestEventLoopEdward Welbourne2022-08-0810-2/+333
| | | | | | | | | | | | Fairly minimal for now, just enough to verify a bug and serve as the sign of success when it's fixed. Tests fail in ways they shouldn't, for now; see expected_eventloop.* for details, notably "Earlier test failed to clean up" messages. Pick-to: 6.4 6.3 Task-number: QTBUG-104441 Change-Id: I59be4aa5f21fed23b19a0593a8c2f6c9956507df Reviewed-by: Jason McDonald <macadder1@gmail.com>
* Add license headers to cmake filesLucie Gérard2022-08-0362-0/+190
| | | | | | | | | | | | CMakeLists.txt and .cmake files of significant size (more than 2 lines according to our check in tst_license.pl) now have the copyright and license header. Existing copyright statements remain intact Task-number: QTBUG-88621 Change-Id: I3b98cdc55ead806ec81ce09af9271f9b95af97fa Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
* CMake: Don't use PUBLIC_LIBRARIES for tests and test helpersAlexandru Croitor2022-07-2861-62/+62
| | | | | Change-Id: I9b7404e1d3a78fe0726ec0f5ce1461f6c209e90d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
* Test skip and fail in cleanup() as well as in cleanupTestCase()Edward Welbourne2022-07-2534-58/+312
| | | | | | | | | | | | | | | | The skipcleanup and failcleanup tests were actually testing skip and fail in cleanupTestCase(), not in cleanup(). Add almost-duplicate tests and clean up so that we now have {fail,skip}cleanup(,testcase} tests to cover all four cases. Generated expected output. The new tests (with old names) get their fail or skip - during cleanup() - reported against the test instead of the cleanupTestCase function. (Results for {init,cleanup}TestCase() are always reported, even when these slots are not defined, as no-op passes.) Pick-to: 6.4 Change-Id: I0988d1696b50c0e2f30c45ddc25e1bd0bfd2151a Reviewed-by: Ivan Solovev <ivan.solovev@qt.io> Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
* Use debug stream in QTest::toString's default fallback if possibleVolker Hilsheimer2022-07-1313-63/+131
| | | | | | | | | | | | | | | | | | | | For built-in types, this is a compile-time assert - we should not have any types in Qt for which we have neither debug streaming nor a QTest::toString specialization implemented. A build of most of Qt submodules passes with this change, after minor modifications to some tests. We cannot declare QSizeHint::Policy as a metatype after the QMetaType has already been instantiated for it, and the QDebug stream operator for QElaspedTimer needs to be correctly declared within the namespace. Add a self-test function for a custom type, and update reference files of the self-test. Task-number: QTBUG-104867 Pick-to: 6.4 Change-Id: I2936db5933f4589fce45f47cf2f3224ed614d8c9 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
* testlib: Don't print QCOMPARE values if they lack string representationVolker Hilsheimer2022-07-107-54/+101
| | | | | | | | | | | | | | | | | | | | | | | Before 0681a2dd5a8095baddb5905fb21a58ce19b958c5, QCOMPARE'ing types for which no QTest::toString specialization exists did not output Actual and Expected lines on failure, as that would only print <null> for both values (which then look like the same value, confusingly). Commit 0681a2dd5a8095baddb5905fb21a58ce19b958c5 changed that behavior, and started printing the confusing <null> values. Take care of the logic in the formatFailMessage function: if both values are nullptr, then print only the variable names, but not the confusing <null> text representation of the values. Remove dead and duplicated code related to the formatting logic, add a self-test function, and update the expected_cmptest files. Fixes: QTBUG-104867 Pick-to: 6.4 Change-Id: I4be98e79f91196b14690a2cc0a68ffd50b431a45 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* qtestlib: exit with error if a test is invoked with unknown data tagDimitrios Apostolou2022-06-281-0/+52
| | | | | | | | | | | | | | Previously trying to execute a test function with an unknown data tag would print an error message but exit with 0. This patch stores a test failure, and continues trying to execute the rest of the command line arguments, if any. In the end the process exits with the usual exit code (number of failed tests) which is now !=0. Pick-to: 6.4 6.3 6.2 Fixes: QTBUG-24240 Change-Id: Id4d422035f173e01e77ca88028dfd94dc0f9085c Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* FatalSignalHandler: print some more information from siginfo_tThiago Macieira2022-06-281-5/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | The siginfo_t parameter allows us to show what process sent a signal or the crashing address. Additionally, it allows us to determine if the crashing signal was indeed sent due to a crash. The selftest tst_crashes produces now: $ QTEST_DISABLE_STACK_DUMP=1 ./crashes ********* Start testing of tst_Crashes ********* Config: Using QtTest library 6.4.0, Qt 6.4.0 (x86_64-little_endian-lp64 shared (dynamic) debug build; by GCC 11.2.1 20220420 [revision 691af15031e00227ba6d5935c1d737026cda4129]), opensuse-tumbleweed 20220428 PASS : tst_Crashes::initTestCase() Received signal 11 (SIGSEGV), code 1, for address 0x0000000000000004 Function time: 0ms, total time: 0ms [1] 201995 segmentation fault (core dumped) QTEST_DISABLE_STACK_DUMP=1 ./crashes The last line comes from the shell. The code isn't decoded, but on Linux it's a SEGV_MAPERR. macOS prints exactly the same thing. I've updated one of the expected_crashes_*.txt output that doesn't seem possible (the "Received a fatal error" message does not appear in Qt anywhere). Pick-to: 6.4 Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16ebc8391234f0e2 Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
* FatalSignalHandler: print the signal name on crashThiago Macieira2022-06-082-2/+2
| | | | | | | | | | | | | It's easier to remember what "SIGSEGV" means instead of "11". GNU libc has offered sigabbrev_np() (non-portable) since 2.32; for older libcs, we'll be happy with a hardcoded list. Selftest updated to match... though it didn't seem to be necessary. Pick-to: 6.4 Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16ebc66ecf6e9465 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Update Catch2 to v2.13.9Marc Mutz2022-06-081-5/+9
| | | | | | | | | Two patches applied to upstream release. Pick-to: 6.4 6.3 6.2 Fixes: QTBUG-103732 Change-Id: Id64b65c4567433806047a2a34fa85ab5f260e6cc Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* QTestLib: rework QTest::compare_helper()Ivan Solovev2022-06-037-6/+32
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | [ChangeLog][QTestLib] QCOMPARE now evaluates toString() on its arguments lazily, speeding up the general case where the comparison doesn't fail. This is true for the QCOMPARE functionality provided by Qt. If you specialized qCompare() for your own types, then you need to change its implementation in line with Qt's own qCompare() specializations in order to enable this feature. [ChangeLog][QTestLib] QCOMPARE calls with nullptr argument(s) will now print the actual and expected values upon failure. Previously it was not like that because of the compareHelper() overload in qtestresult.cpp that treated the presence of nullptr-arguments as a reason to ignore formatFailMessage() call. New implementation does not have this check, and correctly executes formatFailMessage() for all arguments. Note that the qCompare() overloads that call QTestResult::compare() internally were not affected by this patch, because they already defer toString() invocation until the comparison fails. Some numbers, collected against shared release developer build. I checked how this change affects the test execution. The idea was to pick some tests for types that do not have a specific QTestResult::compare overload, so I picked a couple of QByteArray tests. The comparison is done by running a test 10 times and taking the average execution duration, as reported in the log. tst_qbytearrayapisymmetry: Before: 15.6 ms After: 14.2 ms tst_qbytearray: Before: 41 ms After: 36 ms The benefit is around 9% and 12% respectively. Fixes: QTBUG-98874 Change-Id: I7d59ddc760168b15974e7720930f629fb34efa13 Reviewed-by: Marc Mutz <marc.mutz@qt.io>
* Add QTRY_COMPARE_{EQ,NE,LT,LE,GT,GE}_WITH_TIMEOUT()Ivan Solovev2022-06-037-7/+191
| | | | | | | | | | | | [ChangeLog][QTestLib] Add QTRY_COMPARE_{EQ,NE,LT,LE,GT,GE}_WITH_TIMEOUT macros that repeatedly execute QCOMPARE_{EQ,NE,LT,LE,GT,GE} until either the comparison returns true or the timeout expires. Also add QTRY_COMPARE_{EQ,NE,LT,LE,GT,GE} macros that simply invoke the *_WITH_TIMEOUT versions with the usual timeout of five seconds. Task-number: QTBUG-98873 Change-Id: Ib0d7d1c8c997f442b46acd85da738a8f512cc875 Reviewed-by: Marc Mutz <marc.mutz@qt.io>
* Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}()Ivan Solovev2022-06-0310-1/+3985
| | | | | | | | | | | | | | | | | | | | | [ChangeLog][QTestLib] Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}() macros. These new macros behave similarly to QVERIFY(a op b), where 'op' is ==, !=, <, <=, >, >= respectively, but print a formatted error message with argument values in case of failure. The formatting is done lazily, which means that the strings will be generated only when the comparison fails. Also add a new test for tst_selftest and generate expected output for it. Fixes: QTBUG-98873 Task-number: QTBUG-98874 Change-Id: Ic8074798901d7a469b1f58d5cd28bbf49a3da1db Reviewed-by: Marc Mutz <marc.mutz@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Guard the use of QWinEventNotifier in testsAlexey Edelev2022-05-172-0/+4
| | | | | | | | | Add the Q_OS_WIN32 guard for the include of QWinEventNotifier. Change-Id: I7824b2ee236a370c83fd85a2f594a39cf36b36e6 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io> Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
* Use SPDX license identifiersLucie Gérard2022-05-1663-1724/+127
| | | | | | | | | | | | | Replace the current license disclaimer in files by a SPDX-License-Identifier. Files that have to be modified by hand are modified. License files are organized under LICENSES directory. Task-number: QTBUG-67283 Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
* Remove remnants of the old Intel C++ compilerThiago Macieira2022-05-111-1/+1
| | | | | | | | | | | | | We don't support it any more. I don't think it has ever properly compiled Qt 6 (and it's no longer working for me against GCC 12's libstdc++ headers). If you report a bug against it, Intel support's first question is if you can try instead the new Clang/LLVM-based oneAPI C++ compiler. So we support only that one, which identifies itself as Q_CC_CLANG. Change-Id: I5ff8e16fcdcb4ffd9ab6fffd16eb57a092c8439e Reviewed-by: Marc Mutz <marc.mutz@qt.io>
* Replace uses of _qs with _s in testsSona Kurazyan2022-04-071-1/+3
| | | | | | Task-number: QTBUG-101408 Change-Id: If092a68828a1e8056259cf90d035d9a87989244b Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* TAP test logger: report B?XFAIL (mostly) as a messageEdward Welbourne2022-03-293-61/+141
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, only the first B?XFAIL would be reported, all others would be discarded. Furthermore, if a B?XFAIL had happened, B?PASS was also not reported, since the B?XFAIL served as test line. However, if the B?XFAIL was followed by a SKIP, B?XPASS or B?FAIL, these were reported as normal, producing exactly the kind of duplicated test line that the skipping of B?PASS was meant to supply. So change B?XFAIL to be reported among the messages, but retain the TODO annotation of the first on the test line of a subsequent B?PASS, if nothing more drastic happens in the mean time. So now more than one B?XFAIL can be reported, the test is still marked as a TODO and we don't get duplicate test lines for a subsequent non-passing result. This replaces the bool m_wasExpectedFail member with a QTestCharBuffer m_firstExpectedFail that records the first XFAIL's TODO line (so its isEmpty() fully replaces m_wasExpectedFail). Previously, the at/file/line information for a B?XFail would be supplied as top-level keys in the YAML block for a "Pass" reported as not ok due to the XFail, as this location information is now part of the B?XFail's message in the extensions/messages block. Duplicating the first B?XFail's location at top level would add complexity and is arguably misleading, as the test result is really a pass (after ignoring known issues), and the location of the pass is indeterminate (nominally the end of the test function, but actually also after the cleanup() call for this test, when relevant), which is why a Pass has no location information. Task-number: QTBUG-96844 Change-Id: Ib3f24f56266ff49bf3bc9759ac8263ac69a62130 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Move TAP's messages block to after the primary report detailsEdward Welbourne2022-03-2916-165/+101
| | | | | | | | | In the process, split it from its comments block and don't bother with a YAML block if it would only have contained comments. Task-number: QTBUG-96844 Change-Id: I08c20f796252bb270ba9caa4c055cdcc0843a88b Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Include file and line information in TAP diagnostics blocksEdward Welbourne2022-03-295-0/+30
| | | | | | | | | | | When producing a diagnostics block, include the file and line information, if we have it, to describe it. This presently only adds this information for skip, but could in principle do the same for a B?XPass. Task-number: QTBUG-96844 Change-Id: I6cc375d98e2369eba262010f9c2dfbcba931a6f1 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* TAP test logger: move messages into the diagnostics blockEdward Welbourne2022-03-2918-2281/+4636
| | | | | | | | | | | | | | | | | | | | | | | | | | | Our TAP output was delivering messages as comments before the test line, where TAP clearly expects the details of a test to follow its test line. Version 13 provides a YAML block to deliver diagnostics and encourages use of this, so accumulate our messages in a QTestCharBuffer instead of emitting them one by one. However, messages produced after a test has produced its test line belong to that test, but are too late to be included in its diagnostics block, so should be emitted immediately as before, albeit now with a type prefix. This at least separates such messages, from the end of one test, from messages produced early in the next. In the process, add a type-prefix to each, to make clear what type of message it was. Since the Yamlish supported by TAP consumers doesn't support a way to have many messages, use the extensions: top-level hash tag with a messages: sub-tag to gather our messages as a list. (This expands at least one expected output file significantly and substantially rewrites some others.) Add methods to QTestCharBuffer, and a helper function, to support this. Task-number: QTBUG-96844 Change-Id: If44a33da5879ed1670ef0980042599afd516f9d2 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Fix parsing of .cmake.conf in generate_expected_output.pyEdward Welbourne2022-03-251-3/+7
| | | | | | | | | | | It relied on the set(QT_REPO_MODULE_VERSION ...) line being the first thing in the file. It no longer is, and the variable is referenced in an earlier condition. Change-Id: I2bd2abd2a5680782ddd61f4bf4a4a3265b444d52 Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io> Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io> Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
* Avoid duplicate descriptions in TAP's incident YAMLEdward Welbourne2022-03-221-13/+0
| | | | | | | | | | | | | | When a B?Fail's description doesn't match the QVERIFY/QCOMPARE regexes, it got output as a comment in the YAML block after the test line had already reported it as a TODO comment. An empty description would also have lead to an empty comment in the YAML block. Condition this fallback output case on there being a description that hasn't yet been reported in the test line. Task-number: QTBUG-96844 Change-Id: Id7fe81d26ddb01da3d8003ada8fa590a5e1a166f Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* TAP test logger: treat blacklisted XFail the same as XFailEdward Welbourne2022-03-221-3/+0
| | | | | | | | | | | In all places but one they were treated the same; fix that last one to match the rest. This removes one line from the YAML block for each blacklisted XFail test; in each case, this message duplicates the one on its "not ok ... # TODO..." test line. Change-Id: Iff2028afccd979db7f2c84e85d1b78541900008e Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* CMake: Add BUILTIN_TESTDATA option to qt_internal_add_testAlexey Edelev2022-02-111-19/+9
| | | | | | | | | | | Multiple tests use qt_internal_add_resource that copies the functionality that is already implemented inside the qt_internal_add_test function. Simplify these test by replacing the qt_internal_add_resource call with the new BUILTIN_TESTDATA option. Change-Id: I18475b817d6f87264f0de53817d6c26c5ccab4e2 Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
* QPlainTestLogger: do not print "failure location" if there's no failureGiuseppe D'Angelo2022-02-021-0/+2
| | | | | | | | | | | | | | | When running a test under increased verbosity levels, QTestLib would print "failure location" for every QTest function (QVERIFY, QCOMPARE, ...), even if there's no failure at all. Keep the code centralized, but split the formatting of failures and non-failures (other messages). Task-number: QTBUG-96844 Change-Id: I3c508653176b68579dc0eb0cffcc153a52da2e2c Pick-to: 5.15 6.2 6.3 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
* Catch2: upgrade to 2.13.8Samuli Piippo2022-01-313-382/+753
| | | | | | | | | | | | | | | Update Catch2 header to latest released version. Needed to build correctly with glibc 2.34. Two patches applied on top of the upstream release. Previously used Qt specific change extracted to a separate patch file. Fixes: QTBUG-95842 Pick-to: 6.3 6.2 Change-Id: I67e442bbe756ff00c96a45eabf593f7aecddd628 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Exclude tst_selftests when feature process is disabledTatiana Borisova2022-01-272-8/+2
| | | | | | | | | | - It should not be built for systems without process feature. Task-number: QTBUG-99123 Pick-to: 6.2 6.3 Change-Id: I71caa59c2168435894c7d1afcc8226e44178439f Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Tidy up tst_Warnings: comments and #if-eryEdward Welbourne2022-01-211-3/+9
| | | | | | | | | It was missing #if-ery on feature regularexpression for one test that depends on it. One of its comments had a long line. Added some annotations to make clear what's going on in messier tests. Change-Id: I06d8748a134591f93b36029713e52ffd826a24dc Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Remove unused .qrc filesJoerg Bornemann2022-01-171-5/+0
| | | | | | | | Task-number: QTBUG-94446 Change-Id: I136d8b4ab070a832866aa50b5701fc6bd863df8a Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
* Make counts of various types of test result add up correctlyEdward Welbourne2022-01-1328-82/+593
| | | | | | | | | | | | | | | | | | | | | | | | | Added tests for repeated skips and failures (from within void lambdas, to simulate skips and failures from within event handlers). These exhibit yet more ways to count more than one outcome for a test. The new QTest::failOnWarning() can also provoke more than one failure from a single test, and several existing selftests exhibited various ways for the Totals line's counts to add up to more than the number of actual tests run. Fixed counting so that only the first decisive incident is counted. Tests can still report later failure or skipping, but only the first is counted. Added a currentTestState in qtestlog.cpp, by which it keeps track of whether the test has resolved to a result, and clearCurrentTestState() by which other code can reset that at the end of each test. This brought to light various places where test-end clean-up was not being handled - due to failure or skipping in a *_data() method or init, or a skip in cleanup. Fixes: QTBUG-95661 Change-Id: I5d24a37a53d3db225fa602649d8aad8f5ed6c1ad Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Properly indent qtestlib's XML outputDimitrios Apostolou2021-12-1791-13536/+13535
| | | | | | | | | Also regenerate the expected output for tst_selftests, to match new output. Changed one line source code in tst_seftests for the same purpose. Change-Id: I930ba4bb290568d6f67a8910a781725f01f08bf1 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Fix helper script to run successfully on X11Dimitrios Apostolou2021-12-171-1/+1
| | | | | | | | | Without the fix, some tests from tst_selftests were failing with: Invalid MIT-MAGIC-COOKIE-1 key Change-Id: I2d07f28d9b2c69fe8575fdf10b362d962f456970 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* TAP test logger: avoid dangling space on TODO and SKIP linesEdward Welbourne2021-12-091-1/+1
| | | | | | | | | If an incident has an empty description we produced output with a dangling space. Avoid triggering the inanity 'bot with those. Reorganise the code to save some duplication. Change-Id: I1dc29fa8ad4449a4584f11dddcf002e405cd9238 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* TAP test logger: skip XFail results as well as pass after an XFailEdward Welbourne2021-12-092-24/+0
| | | | | | | | | | The rationale for skipping pass after XFail is "to emit a single test point for" the test; emitting several XFails violates that aim. Task-number: QTBUG-96844 Change-Id: Ia8626dfc2dded234b3aa530fc2dc2324f1e28400 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* Make Skip an incident in test loggingEdward Welbourne2021-12-0943-212/+315
| | | | | | | | | | | | | | | Skip ends the test (albeit inconclusively). Rearrange the enums in the abstract logger, move code to handle skip between relevant function and tidy up various things that became simpler as a result. Also reorder the message enum, and its switches, to separate testlib's internals from the usual Qt messages, and put each group in ascending order of severity. Task-number: QTBUG-96844 Change-Id: I2c7a634b9f849830d64eafa750155e66e244b729 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* Match up testlib selftest's lists of generated filesEdward Welbourne2021-12-0952-1229/+421
| | | | | | | | | | | | | | | | | | | | | | | | The actual files present for expected data, the test function that selected which tests to skip and the python function to select which to not generate data for weren't in sync with one another. The test-code's reason for omitting three tests was that we lacked data files for them. So generate those and skip that exception. The generator script's code to decide which to generate didn't exclude anything like as many, so update it to match the test-code. In the process, save repeating a startswith test that was used both positively and negatively, unifying two conditions. Extend the generator script's handling of its --skip-callgrind option by auto-setting that option if valgrind isn't available, to match the driver program's similar skipping. The generated data included many files for tests we skip and, as mentioned already, lacked files for some tests we only skipped because we lacked them. Remove the unused files, add the lacking ones. Change-Id: If91696cdd95b7b0d5f3d686bff839f1bf15e121b Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
generated by cgit v1.2.3 (git 2.25.1) at 2024-04-26 21:28:46 +0000