summaryrefslogtreecommitdiffstats
path: root/tests/auto/testlib
Commit message (Collapse)AuthorAgeFilesLines
* Tidy up tst_Warnings: comments and #if-eryEdward Welbourne2022-01-211-3/+9
| | | | | | | | | It was missing #if-ery on feature regularexpression for one test that depends on it. One of its comments had a long line. Added some annotations to make clear what's going on in messier tests. Change-Id: I06d8748a134591f93b36029713e52ffd826a24dc Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Remove unused .qrc filesJoerg Bornemann2022-01-171-5/+0
| | | | | | | | Task-number: QTBUG-94446 Change-Id: I136d8b4ab070a832866aa50b5701fc6bd863df8a Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
* Make counts of various types of test result add up correctlyEdward Welbourne2022-01-1328-82/+593
| | | | | | | | | | | | | | | | | | | | | | | | | Added tests for repeated skips and failures (from within void lambdas, to simulate skips and failures from within event handlers). These exhibit yet more ways to count more than one outcome for a test. The new QTest::failOnWarning() can also provoke more than one failure from a single test, and several existing selftests exhibited various ways for the Totals line's counts to add up to more than the number of actual tests run. Fixed counting so that only the first decisive incident is counted. Tests can still report later failure or skipping, but only the first is counted. Added a currentTestState in qtestlog.cpp, by which it keeps track of whether the test has resolved to a result, and clearCurrentTestState() by which other code can reset that at the end of each test. This brought to light various places where test-end clean-up was not being handled - due to failure or skipping in a *_data() method or init, or a skip in cleanup. Fixes: QTBUG-95661 Change-Id: I5d24a37a53d3db225fa602649d8aad8f5ed6c1ad Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Properly indent qtestlib's XML outputDimitrios Apostolou2021-12-1791-13536/+13535
| | | | | | | | | Also regenerate the expected output for tst_selftests, to match new output. Changed one line source code in tst_seftests for the same purpose. Change-Id: I930ba4bb290568d6f67a8910a781725f01f08bf1 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Fix helper script to run successfully on X11Dimitrios Apostolou2021-12-171-1/+1
| | | | | | | | | Without the fix, some tests from tst_selftests were failing with: Invalid MIT-MAGIC-COOKIE-1 key Change-Id: I2d07f28d9b2c69fe8575fdf10b362d962f456970 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* TAP test logger: avoid dangling space on TODO and SKIP linesEdward Welbourne2021-12-091-1/+1
| | | | | | | | | If an incident has an empty description we produced output with a dangling space. Avoid triggering the inanity 'bot with those. Reorganise the code to save some duplication. Change-Id: I1dc29fa8ad4449a4584f11dddcf002e405cd9238 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* TAP test logger: skip XFail results as well as pass after an XFailEdward Welbourne2021-12-092-24/+0
| | | | | | | | | | The rationale for skipping pass after XFail is "to emit a single test point for" the test; emitting several XFails violates that aim. Task-number: QTBUG-96844 Change-Id: Ia8626dfc2dded234b3aa530fc2dc2324f1e28400 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* Make Skip an incident in test loggingEdward Welbourne2021-12-0943-212/+315
| | | | | | | | | | | | | | | Skip ends the test (albeit inconclusively). Rearrange the enums in the abstract logger, move code to handle skip between relevant function and tidy up various things that became simpler as a result. Also reorder the message enum, and its switches, to separate testlib's internals from the usual Qt messages, and put each group in ascending order of severity. Task-number: QTBUG-96844 Change-Id: I2c7a634b9f849830d64eafa750155e66e244b729 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* Match up testlib selftest's lists of generated filesEdward Welbourne2021-12-0952-1229/+421
| | | | | | | | | | | | | | | | | | | | | | | | The actual files present for expected data, the test function that selected which tests to skip and the python function to select which to not generate data for weren't in sync with one another. The test-code's reason for omitting three tests was that we lacked data files for them. So generate those and skip that exception. The generator script's code to decide which to generate didn't exclude anything like as many, so update it to match the test-code. In the process, save repeating a startswith test that was used both positively and negatively, unifying two conditions. Extend the generator script's handling of its --skip-callgrind option by auto-setting that option if valgrind isn't available, to match the driver program's similar skipping. The generated data included many files for tests we skip and, as mentioned already, lacked files for some tests we only skipped because we lacked them. Remove the unused files, add the lacking ones. Change-Id: If91696cdd95b7b0d5f3d686bff839f1bf15e121b Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Refactor QTEST*_MAIN() implementationsEdward Welbourne2021-12-099-54/+33
| | | | | | | | | | | | | | | | | | The various variants duplicated some rather complex code around varying setup in the middle. Rework in terms of a macro that defines main() and takes the setup code as a parameter. That setup code also had some common structure, so package that in a setup macro that takes the class to be used. Reworked various testlib selftests that were using QTEST_MAIN_IMPL(); change to use the new QTEST_MAIN_WRAPPER() and TEST_MAIN_SETUP(). These might be better dealt with by supporting a second form of the initMain() test-setup function in the test classes, that takes references for argc and argv, to let a test massage its command-line options. Change-Id: I7fb16b38d51c80ba2f5c9c82f3b7a37ffc636795 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Fix x86 preprocessor check in testlib selftestsEdward Welbourne2021-12-094-10/+20
| | | | | | | | | | | | | | | | | When deciding whether to run benchlibcallbrind, the code only tested _i386; it now tests __x86_64 as well, to match a recent change in the test itself. As there, reverse the test to reduce negations, flipping the stanzas it selects between. Also tidy up the code that tests for valgrind being present - and actually return true, to skip the test, when it claims to be skipping the test. Updated test results, now that the test can actually be run and produce sensible output. Added an _2.txt that matches the results presently seen in Coin on RHEL 8.4 (despite the fact that a local build on such a VM produces output matching the _1.txt results). Change-Id: Ibce09dca06a1eeb73e90fb1345834998683df9d8 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* Fix argv hacking in tst_benchlibcallgrind.cppEdward Welbourne2021-12-099-85/+23
| | | | | | | | | | | | | | | | | | It unconditionally added -callgrind to its own command-line options, but the way testlib handles this argument is, in QTest::qRun(), to re-run the program under the control of valgrind --tool=callgrind, removing the -callgrind command-line option from the test and adding -callgrindchild to its command-line options. So we shouldn't re-add the -callgrind option in the resulting recursive call. The test now runs quickly, producing sensible output, where previously it took a very long time. Revised the drivers to reflect this speed-up, but continue skipping the non-.txt formats to save the need for variant-output files for many formats. To match that, removed the unused non-.txt results files. Change-Id: Iaa99c1b5964d50bccfc6076a21896791b6bbf289 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* Replace a random define with a __has_include() checkEdward Welbourne2021-12-091-7/+8
| | | | | | | | | | | | | The given reason for needing to define it was that there was no way to test whether <valgrind/valgrind.h> is available to #include; but we now require C++17 hence __has_include(). However, moc doesn't seem to grok that, so move the test's #if-ery inside the test, since otherwise it gets omitted because the test's metatype doesn't know it's there. Change-Id: I75a100787b98a52fad4cfb0b047318a115c998e2 Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
* tst_BenchlibCallgrind: fix x86 preprocessor checkEdward Welbourne2021-12-061-4/+4
| | | | | | | | | | | Its twoHundredMillionInstructions() test has #if-ery to limit it to gcc and x86; however, it was testing only __i386 for the x86 part, where gcc defines __x86_64 instead on modern 64-bit systems. In the process, invert the condition and the branches it controls - positive tests are easier to comprehend. Change-Id: I8e906c606c48aa5034e02e3ed5d042fbb1f2ecbc Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* Long live QVERIFY_THROWS_NO_EXCEPTION!Marc Mutz2021-11-267-13/+68
| | | | | | | | | Counter-part to QVERIFY_THROWS_EXCEPTION. [ChangeLog][QTest] Added QVERIFY_THROWS_NO_EXCEPTION macro. Change-Id: Ib6a80c8e810d5e2298ff00d608dae04e7a0c3e8f Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* QTest: de-inline QVERIFY_THROWS_EXCEPTION message formattingMarc Mutz2021-11-266-30/+30
| | | | | | | | | | | | Extract Method QTest::qCaught() to take the string handling out of the header. This should help a bit in speeding up compilation of large unit test files (provided they use QVERIFY_THROWS_EXCEPTION), although I have no data to support that. Since we changed the error message, update the selftest accordingly. Change-Id: Id4a3c8c34d5df8d0c7a861106d269097f4a6de5c Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Add QTest::failOnWarningMitch Curtis2021-11-267-7/+569
| | | | | | | | | | | | | | This solves the long-standing problem of not being able to easily fail a test when a certain warning is output. [ChangeLog][QtTest] Added QTest::failOnWarning. When called in a test function, any warning that matches the given pattern will cause a test failure. The test will continue execution when a failure is added. All patterns are cleared at the end of each test function. Fixes: QTBUG-70029 Change-Id: I5763f8d4acf1cee8178be43a503619fbfb0f4f36 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Long live QVERIFY_THROWS_EXCEPTION!Marc Mutz2021-11-261-6/+44
| | | | | | | | | | | | | Use variable args macros to swallow any extra commas in the expression. To use this, the type of the exception has to be first. Use Eddy's suggestion for a new name to avoid breaking the old macro. [ChangeLog][QtTest] Added QVERIFY_THROWS_EXCEPTION, replacing QVERIFY_EXCEPTION_THROWN, which has therefore been deprecated. Change-Id: I16825c35bae0631c5fad5a9a3ace4d6edc067f83 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* QVERIFY_EXCEPTION_THROWN: re-throw unknown exceptionsMarc Mutz2021-11-251-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | Swallowing unknown exceptions is dangerous business, as the exception might be a pthread cancellation token, the swallowing of which would terminate the program. Instead of returning from the catch-all-clause, therefore, re-throw the unknown exception. Fix tst_verifyexceptionthrown failure cases that use non-std::exception-derived true negative exceptions to not let the exception escape from the test function. As a drive-by, pretty up the macro's docs. [ChangeLog][QtTest][QVERIFY_EXCEPTION_THROWN] Now re-throws unknown exceptions (= not derived from std::exception) (was: swallowed them and returned from the test function), in order to play nice with pthread cancellation. Pick-to: 6.2 5.15 Change-Id: Ic036d4a9ed4b7683fa67e27af8bcbae0eefdd0da Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* QtBase: replace windows.h with qt_windows.hYuhang Zhao2021-11-231-1/+1
| | | | | | | | | | We have some special handling in qt_windows.h, use it instead of the original windows.h Change-Id: I12fa45b09d3f2aad355573dce45861d7d28e1d77 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Rename command-option to generate_expected_output.pyEdward Welbourne2021-10-011-4/+4
| | | | | | | | | | | The --skip-benchlib option was misnamed: there are several benchlib tests, of which it only skipped the callgrind one. As there is no other test involving callgrind, rename to --skip-callgrind. Change-Id: I0179fd35dd79c525f79e4a28a626e964323409bf Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Add missing dependencies of tests on their helper programsEdward Welbourne2021-09-171-28/+4
| | | | | | | | | | | | | | | | | This ensures that a command such as $ ninja tst_qlocale && ninja tst_qlocale_check will automagically build the syslocaleapp program that the test runs from a subtest. Similar for testlib's selftests and tst_QProcess. As a drive-by, pruned some legacy comments from when CMakeLists.txt files were generated from .pro files. Change-Id: I67691a8175aaef124d4104cf1898193993408bdf Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Andrei Golubev <andrei.golubev@qt.io>
* Fix Catch2 compiling for INTEGRITYTatiana Borisova2021-09-142-1/+32
| | | | | | | | | | - Exclude Green Hills compiler from Catch2's POSIX signal handling. GHS, at least on INTEGRITY, doesn't support full POSIX signals. Task-number: QTBUG-96176 Pick-to: 6.2 6.2.0 Change-Id: Ifec06dca43ed766cb7335e40fc357d0d7bc463a6 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Tidy up in counting, skip and skipinit selftestsEdward Welbourne2021-09-093-8/+11
| | | | | | | | | | | | Minor clean-up. Split two long lines and added braces, called for by Qt coding style. Turned some QDebug("this line should never be reached...") into QFAIL(); that should make the effor a bit more prominent if it ever happens. Likewise for a qDebug() << "ERROR: ..." for code not meant to run. Change-Id: I1d34fe202f269eb3ad8f58feaabed809e4fec99c Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io>
* Avoid senseless duplication of code in a testlib selftestEdward Welbourne2021-09-011-36/+24
| | | | | | | | Delegate repeated tasks to functions - DRY ! Change-Id: I8973d2b9cd1f7d0fe94a9ae4aebcff434821e210 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* Add a repeated-failure selftest for testlibEdward Welbourne2021-09-017-33/+263
| | | | | | | | | | | This mimics the effects of test-code delegating checks to other functions, which can lead to repeated failures. The failing test should only contribute one to the total of failed tsts. Drive-by: correct an existing test's reporting of its own name. Task-number: QTBUG-95661 Change-Id: I370fd2aee378d9fdd35826f0d11677483c60423d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* Add more tests of blacklisting, combined with XPASS and XFAILEdward Welbourne2021-08-278-9/+250
| | | | | | | | | | Include counting of test types, to catch two more cases where totals don't add up. Task-number: QTBUG-95661 Change-Id: I9fe5424bc6652c61a065bf2889333e2ed9437c81 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Include check for the Totals line in testlib's expectedfail selftestEdward Welbourne2021-08-277-1/+71
| | | | | | | | | | | Count how many we expect to pass, fail and be skipped, so we can report that for comparison with the actual totals line: this reveals some double-counting, marked with a FIXME. Task-number: QTBUG-95661 Change-Id: I8b0f13ded5202ed476d8abfee70ed60f9e639bf9 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Make clear why QTestLog::addB?XFail() don't add to countersEdward Welbourne2021-08-272-6/+6
| | | | | | | | | | | | | This reverts commit 904617dfb83f39a6a379635b64fea6fcd00f241a and makes clear to future readers why that wouldn't be a sensible change. Update the test's data, eliminating a case of duplicate counting that was caused by the reverted commit. Task-number: QTBUG-95661 Pick-to: 5.12 5.15 6.1 6.2 Change-Id: Ice6d3ab06ca171e0d6eb0fac757f1ab774e229f0 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Expand testing of QEXPECT_FAIL()Edward Welbourne2021-08-177-255/+981
| | | | | | | | | Test both Abort and Continue cases. Test more with successive marked as expected failure. Test cases with a QSKIP after the marked check. Unify data functions where practical. Change-Id: I2eade5e4dd0907d23e37137ce3d93cd5ca79f802 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Don't write redundant log message information in JUnit reporterTor Arne Vestbø2021-08-1016-2276/+0
| | | | | | | | | | | | | The separation between <system-out> and <system-err> is sufficient, and we can't expect consumers to interpret our custom comment format. The type of the text node has been renamed to more accurately identify its purpose. Pick-to: 6.2 Change-Id: I63c8ff17529fc087e1b695698350a6711eb5e68d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Rename QAbstractTestLogger::QSystem to QCriticalTor Arne Vestbø2021-08-105-8/+6
| | | | | | | | | | | | | It's used to handle qCritical() messages, and corresponds to QtMsgType's QtCriticalMsg, so change its name to reflect what it really is. In the process, reclassify the -maxwarnings overflow message as a warning, rather than a critical message. Pick-to: 6.2 Change-Id: I87626117a547ae4498d5dc352b93bd6db8bfb332 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Move stdout/stderr reporting into individual <testcase> elementsTor Arne Vestbø2021-08-1048-2403/+2484
| | | | | | | | | | | | | | | The original Ant JUnit reporter only writes <system-err> and <system-out> to the <testsuite>, but more modern reporters such as Maven Surefire scopes output to each individual <testcase>. This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.: https://github.com/jenkinsci/junit-plugin/commit/145eb5c98 Pick-to: 6.2 Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Only generate expecations for silent test with plain loggerTor Arne Vestbø2021-08-106-239/+10
| | | | | | | | | | | The -silent option to tests is only supported with the plain text logger, so we don't need to maintain expectation files for the others. Pick-to: 6.2 Change-Id: I0f42bfe90d82b7ce04f550c747d4a80e99621e74 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Produce <error> elements on fatal errors in JUnit reporterTor Arne Vestbø2021-08-106-34/+17
| | | | | | | | | Test errors represents unanticipated problems, e.g., an unhandled exception, or a problem with the implementation of the test. Pick-to: 6.2 Change-Id: I87219e7ffdea56862278f005de44526ad97545f0 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Don't report blacklisted tests as failures in JUnit reporterTor Arne Vestbø2021-08-061-1/+1
| | | | | | | | | | | | | We don't produce <failure> elements for them, so we shouldn't include them in the total count of failures. In the future we might produce <skipped> elements instead, but for now remove the incorrect increment. Pick-to: 6.2 Change-Id: Ia89a5cdaf79f6aa2f92be53180112c40ad6da2c6 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Write failure details as <failure> content in JUnit reporterTor Arne Vestbø2021-08-068-168/+224
| | | | | | | Pick-to: 6.2 Change-Id: Ica48769e7dfcabdc4bc8f0ed058bc22e29a0b632 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Indent CDATA element in JUnit reporterTor Arne Vestbø2021-08-0621-2289/+2289
| | | | | | | | | | The data itself is not indented, as consumers may read it as verbatim data. Pick-to: 6.2 Change-Id: Ia934616cea273feadc3a45d7c74726d4f804f0dc Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Clean up tst_blacklistTor Arne Vestbø2021-08-058-141/+16
| | | | | | | | | | | | | | | | | | | | | | | 33d7f76f0e847d7e0fb00dd6056e7bba45b8b1e7 attempted to exercise more of the blacklisting code, but in doing so didn't account for the fact that the features it added didn't actually work. Adding a global blacklisting of * will result in blacklisting all tests, regardless of any inverse conditions "!*" later on, such as for the 'pass' test. This would have been evident by looking at the expectation files, which still showed the test as a BPASS. The broken feature should be fixed, but in the meantime let's clean up the test so it doesn't misrepresent reality. At the same time we also remove the messages() tests, which was copied straight out of tst_silent, but doesn't serve any purpose for the blacklist selftest. Pick-to: 6.2 Change-Id: I20e4693300aad5e9ff5f17ad22e3dfcb5c49fd0e Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Don't report additional 'pass' test point for blacklisted XFailTor Arne Vestbø2021-08-051-2/+1
| | | | | | | | | | Regression after 9906cc57ed3eed64d534f43c677bb16e08561bb6. Pick-to: 6.2 Change-Id: I5566f70c66d248426c7a41b6de1cfb92f104cc64 Reviewed-by: Oliver Wolff <oliver.wolff@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Report skipped tests in JUnit reporter as <skipped> elementsTor Arne Vestbø2021-08-0549-153/+104
| | | | | | | | | | | | | | | | | The Apache Ant and Surefire Maven specs document a <skipped> element that can be used to signify skipped test, with a corresponding total skipped test attribute on the <testsuite>. The element includes an optional message attribute, documented in the Surefire spec, and in the Ant source code, but not yet documented in the reverse-engineered Ant spec: https://github.com/windyroad/JUnit-Schema/pull/11 Pick-to: 6.2 Task-number: QTBUG-95424 Change-Id: Ib6417a41b9c328836f4017e6ebf7f7e9cd91288d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Don't report JUnit errors attribute without any <error> elementsTor Arne Vestbø2021-08-0528-28/+28
| | | | | | | | | | | The errors attribute on the <testsuite> element represents the number of <error> elements, but we do not produce any at the moment. Pick-to: 6.2 Task-number: QTBUG-95424 Change-Id: I7196d622a9a6bbb7e79ed2c2886984d539abb1da Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Improve JUnit XML conformanceTor Arne Vestbø2021-08-0551-800/+1102
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Simplify JUnit test loggerTor Arne Vestbø2021-08-055-18/+9
| | | | | | | | | | | | | - Use the right name for the attribute (AI_Message), rather than fixing it up in QTestJUnitStreamer. - Don't pretend that we're adding line and file information, only to discard it in QTestJUnitStreamer. - Don't pretend to add benchmark information, only to discard it in QTestJUnitStreamer. Pick-to: 6.2 Change-Id: Ib6eadc12300157216fe9c6e8bcfebd7eb8a3ea68 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Pass on file location on failure, but don't assume we have oneTor Arne Vestbø2021-08-0447-88/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We try our best to pass on the file location of a failure, including for fatal errors, but the reporting or logging machinery should not assume there is one. By passing on nullptr for the file location we allow the logging backends to decide how to handle the situation, e.g. by not emitting extra fields for failure location. This effectively reverts c25687fa0b6e4be043e1f8c92c093d8b06ca06c4, in favor of relying on the backends to cope with null filename, which they already did. As qFatal uses QMessageLogger, which by default disables file/line information in release builds, we need to explicitly enable this in our self-tests, to get uniform test results. Similarly, we disable file/line info from testlib itself, as reporting Qt internal file and line information for user diagnostics is less useful. The odd one out there is qtestdata.cpp, which still ends up in test output due to using QTEST_ASSERT instead of qFatal for its diagnostics. Cleaning up that, and unifying how we report testlib issues to the user, is left for another day. Pick-to: 6.2 Change-Id: Ib9451b8eed86fe3ade4a4dcaf0037e1a3450321c Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* testlib: Deprecate QWARN() in favor of qWarning()Tor Arne Vestbø2021-08-044-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | The QtTest best practices documentations recommends using output mechanisms such as qDebug() and qWarning() for diagnostic messages, and this is also what most of our own tests do. The QWARN() macro and corresponding internal QTest::qWarn() function was added when QtTest was first implemented, but was likely meant as an internal implementation detail, like its cousin QTestLog::info(), which does not have any corresponding macro. This theory is backed by our own QtTest self-test (tst_silent) describing the output from QWARN() as "an internal testlib warning". The only difference between QWARN() and qWarning(), besides the much richer feature set of the latter, is that qWarning() will not pass on file and line number information in release mode, but QWARN() will. This is an acceptable loss of functionality, considering that the user can override this behavior by defining QT_MESSAGELOGCONTEXT. [ChangeLog][QtTest] QWARN() has been deprecated in favor of qWarning() Pick-to: 6.2 Change-Id: I5a2431ce48c47392244560dd520953b9fc735c85 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* testlib: Regenerate test expectationsTor Arne Vestbø2021-07-298-63/+64
| | | | | | | | | | After c25687fa0b6e4be043e1f8c92c093d8b06ca06c4 and 539553a57216c4ece292ff54fb623807c334cdb6. Pick-to: 6.2 Change-Id: I34b436f20b9c480f6948aa97430228936955808e Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
* testlib: Run test result expectation tests with stdout as outputTor Arne Vestbø2021-07-291-16/+20
| | | | | | | | | | | | | For historical reasons a few of the subtests are skipped when running with anything but the plain text logger to stdout. To ensure we have as broad test coverage as possible for the expected output of the various loggers we run these tests in stdout-mode. Pick-to: 6.2 Change-Id: I856905d1543afe89710533657a55bd599c0305fd Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Accurately name JUnit test, and only run for JUnitXML reporterTor Arne Vestbø2021-07-2912-264/+32
| | | | | | | | | | | | | The JUnit reporter was initially named xunit, but the naming was inaccurate and the reporter was renamed in 27db9e458cef512fca3a6b5c9ebbcda7a8172428. The corresponding test has now been renamed as well, and as an added bonus we only run it for that reporter. Pick-to: 6.2 Change-Id: I59cb7d949514cdf46a0199a53a7a3e39f833207c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
* Update float16 part of TestLib selftestEdward Welbourne2021-07-288-3168/+15
| | | | | | | | | | | | | | | | In commit 9bad096c091b2c2106c2f325dfca74be111eee58 I gave qFuzzyIsNull(qfloat16) a more forgiving threshold, consistent with qFloatCompare(qfloat16, qfloat16)'s fractional threshold. The selftest failed to catch two of the tests failing, so fix one of them to use different values, and remove one. Updated test expection for txt and deleted for other formats, as they're skipped (in tst_selftests.cpp) for this test. Refined the generator script to know about this test only being tested for txt. Pick-to: 6.2 Change-Id: I109547cf92178bb9f5ff0b06e0b3bb40c881b41b Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-03 09:17:23 +0000