summaryrefslogtreecommitdiffstats
path: root/tests/auto/testlib/selftests/generate_expected_output.py
Commit message (Collapse)AuthorAgeFilesLines
* Correct license for tools filesLucie Gérard2024-03-051-1/+1
| | | | | | | | | | | | According to QUIP-18 [1], all tools file should be LicenseRef-Qt-Commercial OR GPL-3.0-only WITH Qt-GPL-exception-1.0 [1]: https://contribute.qt-project.org/quips/18 Pick-to: 6.7 Task-number: QTBUG-121787 Change-Id: Icd5d5be2e04819617e68ff142924de1773bebbad Reviewed-by: Kai Köhne <kai.koehne@qt.io>
* Change license for tests filesLucie Gérard2024-02-041-1/+1
| | | | | | | | | | | | According to QUIP-18 [1], all tests file should be LicenseRef-Qt-Commercial OR GPL-3.0-only [1]: https://contribute.qt-project.org/quips/18 Pick-to: 6.7 Task-number: QTBUG-121787 Change-Id: I9657df5d660820e56c96d511ea49d321c54682e8 Reviewed-by: Christian Ehrlicher <ch.ehrlicher@gmx.de>
* QTest: add opt-in changing QCOMPARE etc to exit with throw, not returnMarc Mutz2024-01-271-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add exception classes and use them to change the control flow for QTEST_{FAIL,SKIP}_ACTION from return'ing from just the immediate function to the full way to the QTestLib infrastructure, here we filter them out. There are three modes: - If QT_NO_EXCEPTION, then we return - If QTEST_THROW_ON_... is also defined, #error out - Otherwise, if QTEST_THROW_ON_... is defined, always throw - Otherwise, the decision is made at runtime (with defaults read from QTEST_THROW_ON_... environment variables). Three selftests depend on the old behavior, as they explicitly check that multiple FAIL SKIP etc emerge, which the new framework, of course, prevents. Locally disable throwing at the test function level. Add initial docs and enable exceptions in all of the selftest subprograms to facilitate switching between the two runtime-selectable modes. [ChangeLog][QtTest] Added QTEST_THROW_ON_FAIL and QTEST_THROW_ON_SKIP C++ macros and environment variables that, when defined, change how QCOMPARE/QVERIFY/QSKIP etc exit the test function on failure. Instead of a return, exiting only the immediately-surrounding function, they throw a special exception instead, thereby exiting from subfunctions of the test function, all the way to QtTestLib. Fixes: QTBUG-66320 Change-Id: I96c38d2a1dcdd9de84942cf448a8bbf3ab6d3679 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
* qtestlib: Add option to skip blacklisted testsKalle Viironen2023-11-221-4/+5
| | | | | | | | | | | | | | | Add a command-line option "-skipblacklisted" to testlib to skip blacklisted test cases. Currently, blacklisted test cases are run, but results are ignored. For test code coverage measurements, it's important to see the code coverage of the actually tested code in comparison to the code that was run but not actually tested. The default approach remains unchanged, meaning that blacklisted tests are run with the results ignored. Fixes: QTBUG-112793 Change-Id: I6fe0a6353cb1c021e0232c79bb4f404632fb0bce Reviewed-by: Jason McDonald <macadder1@gmail.com>
* selftests: respect ASAN_OPTIONSMitch Curtis2023-10-161-2/+2
| | | | | | | | | | | | | Ensure that this environment variable is respected, as ASAN errors (e.g. memory leaks) can cause the selftests to fail since they expect no errors in their output. Fix an incorrect function name in a comment, while we're at it. Fixes: QTBUG-118041 Pick-to: 6.5 6.6 Change-Id: I07502101aabd8743df898ae8fe4a693c4733c4af Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* tst_selftests: split the qFatal() test away from tst_SilentThiago Macieira2023-09-061-2/+3
| | | | | | | | | | This allows us to control whether to run this particular test in ASan mode or with specific loggers or not. Adding the expected log output for tst_silent for other loggers is left as an exercise to the reader. Change-Id: Ifa1111900d6945ea8e05fffd177f1548c8c8e714 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Correct some testlib selftest data: blacklisted does not qFatal()Edward Welbourne2023-07-311-1/+1
| | | | | | | | | The blacklisted test does not crash, as its expected output indeed shows. Pick-to: 6.5 6.6 Change-Id: I07522a7d065b5f39620975a3546bcd156024c41d Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
* Add some testing of QTestEventLoopEdward Welbourne2022-08-081-2/+2
| | | | | | | | | | | | Fairly minimal for now, just enough to verify a bug and serve as the sign of success when it's fixed. Tests fail in ways they shouldn't, for now; see expected_eventloop.* for details, notably "Earlier test failed to clean up" messages. Pick-to: 6.4 6.3 Task-number: QTBUG-104441 Change-Id: I59be4aa5f21fed23b19a0593a8c2f6c9956507df Reviewed-by: Jason McDonald <macadder1@gmail.com>
* Test skip and fail in cleanup() as well as in cleanupTestCase()Edward Welbourne2022-07-251-6/+7
| | | | | | | | | | | | | | | | The skipcleanup and failcleanup tests were actually testing skip and fail in cleanupTestCase(), not in cleanup(). Add almost-duplicate tests and clean up so that we now have {fail,skip}cleanup(,testcase} tests to cover all four cases. Generated expected output. The new tests (with old names) get their fail or skip - during cleanup() - reported against the test instead of the cleanupTestCase function. (Results for {init,cleanup}TestCase() are always reported, even when these slots are not defined, as no-op passes.) Pick-to: 6.4 Change-Id: I0988d1696b50c0e2f30c45ddc25e1bd0bfd2151a Reviewed-by: Ivan Solovev <ivan.solovev@qt.io> Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>
* Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}()Ivan Solovev2022-06-031-1/+1
| | | | | | | | | | | | | | | | | | | | | [ChangeLog][QTestLib] Add QCOMPARE_{EQ,NE,LT,LE,GT,GE}() macros. These new macros behave similarly to QVERIFY(a op b), where 'op' is ==, !=, <, <=, >, >= respectively, but print a formatted error message with argument values in case of failure. The formatting is done lazily, which means that the strings will be generated only when the comparison fails. Also add a new test for tst_selftest and generate expected output for it. Fixes: QTBUG-98873 Task-number: QTBUG-98874 Change-Id: Ic8074798901d7a469b1f58d5cd28bbf49a3da1db Reviewed-by: Marc Mutz <marc.mutz@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Use SPDX license identifiersLucie Gérard2022-05-161-27/+2
| | | | | | | | | | | | | Replace the current license disclaimer in files by a SPDX-License-Identifier. Files that have to be modified by hand are modified. License files are organized under LICENSES directory. Task-number: QTBUG-67283 Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
* Fix parsing of .cmake.conf in generate_expected_output.pyEdward Welbourne2022-03-251-3/+7
| | | | | | | | | | | It relied on the set(QT_REPO_MODULE_VERSION ...) line being the first thing in the file. It no longer is, and the variable is referenced in an earlier condition. Change-Id: I2bd2abd2a5680782ddd61f4bf4a4a3265b444d52 Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io> Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io> Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
* Fix helper script to run successfully on X11Dimitrios Apostolou2021-12-171-1/+1
| | | | | | | | | Without the fix, some tests from tst_selftests were failing with: Invalid MIT-MAGIC-COOKIE-1 key Change-Id: I2d07f28d9b2c69fe8575fdf10b362d962f456970 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Match up testlib selftest's lists of generated filesEdward Welbourne2021-12-091-5/+43
| | | | | | | | | | | | | | | | | | | | | | | | The actual files present for expected data, the test function that selected which tests to skip and the python function to select which to not generate data for weren't in sync with one another. The test-code's reason for omitting three tests was that we lacked data files for them. So generate those and skip that exception. The generator script's code to decide which to generate didn't exclude anything like as many, so update it to match the test-code. In the process, save repeating a startswith test that was used both positively and negatively, unifying two conditions. Extend the generator script's handling of its --skip-callgrind option by auto-setting that option if valgrind isn't available, to match the driver program's similar skipping. The generated data included many files for tests we skip and, as mentioned already, lacked files for some tests we only skipped because we lacked them. Remove the unused files, add the lacking ones. Change-Id: If91696cdd95b7b0d5f3d686bff839f1bf15e121b Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Fix argv hacking in tst_benchlibcallgrind.cppEdward Welbourne2021-12-091-1/+1
| | | | | | | | | | | | | | | | | | It unconditionally added -callgrind to its own command-line options, but the way testlib handles this argument is, in QTest::qRun(), to re-run the program under the control of valgrind --tool=callgrind, removing the -callgrind command-line option from the test and adding -callgrindchild to its command-line options. So we shouldn't re-add the -callgrind option in the resulting recursive call. The test now runs quickly, producing sensible output, where previously it took a very long time. Revised the drivers to reflect this speed-up, but continue skipping the non-.txt formats to save the need for variant-output files for many formats. To match that, removed the unused non-.txt results files. Change-Id: Iaa99c1b5964d50bccfc6076a21896791b6bbf289 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* Rename command-option to generate_expected_output.pyEdward Welbourne2021-10-011-4/+4
| | | | | | | | | | | The --skip-benchlib option was misnamed: there are several benchlib tests, of which it only skipped the callgrind one. As there is no other test involving callgrind, rename to --skip-callgrind. Change-Id: I0179fd35dd79c525f79e4a28a626e964323409bf Reviewed-by: Ievgenii Meshcheriakov <ievgenii.meshcheriakov@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* testlib: Only generate expecations for silent test with plain loggerTor Arne Vestbø2021-08-101-8/+10
| | | | | | | | | | | The -silent option to tests is only supported with the plain text logger, so we don't need to maintain expectation files for the others. Pick-to: 6.2 Change-Id: I0f42bfe90d82b7ce04f550c747d4a80e99621e74 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Improve JUnit XML conformanceTor Arne Vestbø2021-08-051-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Accurately name JUnit test, and only run for JUnitXML reporterTor Arne Vestbø2021-07-291-1/+3
| | | | | | | | | | | | | The JUnit reporter was initially named xunit, but the naming was inaccurate and the reporter was renamed in 27db9e458cef512fca3a6b5c9ebbcda7a8172428. The corresponding test has now been renamed as well, and as an added bonus we only run it for that reporter. Pick-to: 6.2 Change-Id: I59cb7d949514cdf46a0199a53a7a3e39f833207c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
* Update float16 part of TestLib selftestEdward Welbourne2021-07-281-0/+7
| | | | | | | | | | | | | | | | In commit 9bad096c091b2c2106c2f325dfca74be111eee58 I gave qFuzzyIsNull(qfloat16) a more forgiving threshold, consistent with qFloatCompare(qfloat16, qfloat16)'s fractional threshold. The selftest failed to catch two of the tests failing, so fix one of them to use different values, and remove one. Updated test expection for txt and deleted for other formats, as they're skipped (in tst_selftests.cpp) for this test. Refined the generator script to know about this test only being tested for txt. Pick-to: 6.2 Change-Id: I109547cf92178bb9f5ff0b06e0b3bb40c881b41b Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Brush up QTestlib/generate_expected_output.pyFriedemann Kleint2021-01-151-72/+78
| | | | | | | | | | | | | | | - Generate the expected files in the source tree, removing the need to copy them over - Add proper option parsing, add options for formats and to skip the callgrind test, which locks up - Determine the script location by __file__ - Determine the Qt version by reading the .cmake.conf file - Introduce f-strings - Print the usage when invoked in the wrong directory Task-number: QTBUG-86540 Change-Id: Idabb50a14db60127374b7a1271951dbbbc85d131 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Add start time and test duration to JUnit XMLTor Arne Vestbø2020-07-241-0/+2
| | | | | | | As defined by https://llg.cubic.org/docs/junit/ Change-Id: Ic7683f3d49c529674f8467d591528d4a65d3add8 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
* testlib: Output JUnitXML attributes in right orderTor Arne Vestbø2020-07-241-1/+1
| | | | | | | | | | | | | | | | | | The attributes are, like the elements, maintained in reverse order in the underlying QTestCoreList, so we need to iterate them backwards when printing out the resulting XML to reflect the order they were added. This results in e.g.: <testcase name="passingBenchmark" result="pass"> Instead of: <testcase result="pass" name="passingBenchmark"> Change-Id: Ic2eeab8de05ffedd0c41977358d5b40ff77878b1 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
* testlib selftest: include 'keyboard' test in generate_expected_output.pyTor Arne Vestbø2020-07-221-1/+1
| | | | | Change-Id: Ia1d759454b2931ca838942cf9b08f8b62dfd61ce Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
* testlib selftest: unify xml quoted asserts that include line numbersTor Arne Vestbø2020-07-221-1/+1
| | | | | Change-Id: If66303bab10c23dd212a6abcdff3f145630e2401 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
* testlib selftest: rebaseline tap results to not include line numbersTor Arne Vestbø2020-07-221-0/+2
| | | | | Change-Id: I6271ac1984a3829ba1c8dcc9d6419940cb89c62b Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
* Fix testlib/selftest generate_expected_output.pyTor Arne Vestbø2020-07-211-1/+1
| | | | | | | Amends 40bc8d51298adf9d342de8195d6ead81a9cc6bcb. Change-Id: Iacaabec0c620db84278656501e64fc2b765e052f Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
* testlib/selftests: Fix generate_expected_output.pyFriedemann Kleint2020-07-171-34/+22
| | | | | | | | | Hardcode the tests which were previously scanned from the source. See qtbase/24e83de8d1924b8003c84f1df05b7befea2c5120. Change-Id: I8fb05568977f86726b20948a9c2d1cfce5cba161 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* QCoreApplication: force the process locale codec to UTF-8Thiago Macieira2020-05-051-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | As discussed in the mailing list and in the Qt Contributor Summit 2019. Tested on Linux, macOS and FreeBSD, showing these fallbacks: OS environment setlocale() call FreeBSD empty "C.UTF-8" FreeBSD LC_ALL=C "C.UTF-8" Linux empty "C.UTF-8" Linux LC_ALL=C "C.UTF-8" Linux LANG=en_US "en_US.UTF-8" Linux LANG=de_DE@euro "de_DE.UTF-8" Linux LANG=en_GB.iso885915 "en_GB.UTF-8" Linux LANG=hy_AM.armscii8 "hy_AM.UTF-8" Linux LANG=ja_JP.sjis "ja_JP.UTF-8" Linux LANG=ru_RU.koi8r "ru_RU.UTF-8" macOS empty "UTF-8" macOS LC_ALL=C "UTF-8" Versions tested: FreeBSD 12.1, Linux w/ glibc 2.30, macOS 10.14.2. See * https://wiki.qt.io/Qt_Contributor_Summit_2019_-_QtCore * https://lists.qt-project.org/pipermail/development/2019-October/037791.html Change-Id: Ia2aa807ffa8a4c798425fffd15d97ddb4f35b0ae Reviewed-by: Thiago Macieira <thiago.macieira@intel.com> Reviewed-by: Lars Knoll <lars.knoll@qt.io>
* testlib: Move subtest argument handling to individual testsTor Arne Vestbø2020-02-061-20/+1
| | | | | | | | The test orchestrator shouldn't have to deal with the individual options needed for each test. Change-Id: I78bbf4850cc649e625bd08a7aedf02267ba1314d Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Clarify that our XUnit reporter is actually a JUnit reporterTor Arne Vestbø2020-02-031-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The reporter was probably named 'xunit' based on the historical use of xUnit to refer to testing frameworks derived from Smalltalk's SUnit. These frameworks typically added their own prefix, e.g. JUnit for Java, RUnit for R, etc. The most popular of these was the JUnit framework, and the corresponding XML output produced by the Ant built tool became somewhat of a de facto standard, which is probably why we chose to model our reporter after it. Nowadays however, naming it 'xunit' is problematic as there is actually a testing famework named xUnit.net, typically shortened to, you guessed it: xunit. Test report consumers will typically have a junit mode, and an xunit mode, and the latter could easily be mistaken for what testlib outputs, unless we clarify this. The clarification also allows us to safely extend our support for the JUnit XML format to incorporate some elements that are nowadays common, but where we are lagging behind the standard. [ChangeLog][QTestLib] The formerly named 'xunitxml' test reporter has been renamed to what it actually is: a JUnit test reporter, and is now triggered by passing -o junitxml to the test binary. Change-Id: Ieb20d3d2b5905c74e55b98174948cc70870c0ef9 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io> Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
* Test QTest's signal dumperMårten Nordheim2019-03-271-0/+12
| | | | | | | | | It has been completely untested for a while and saw some breakage. So let's add a selftest for it. ('-vs' when running tests) Change-Id: Ibfb5ac0a2d741de7c3f519d91202d4977996045e Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Merge remote-tracking branch 'origin/5.12' into 5.13Liang Qi2019-02-081-1/+1
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Conflicts: src/android/templates/AndroidManifest.xml src/network/ssl/qsslsocket_mac.cpp src/widgets/styles/qstylesheetstyle.cpp tests/auto/corelib/kernel/qtimer/BLACKLIST tests/auto/testlib/selftests/blacklisted/tst_blacklisted.cpp tests/auto/testlib/selftests/expected_blacklisted.lightxml tests/auto/testlib/selftests/expected_blacklisted.tap tests/auto/testlib/selftests/expected_blacklisted.teamcity tests/auto/testlib/selftests/expected_blacklisted.txt tests/auto/testlib/selftests/expected_blacklisted.xml tests/auto/testlib/selftests/expected_blacklisted.xunitxml tests/auto/testlib/selftests/expected_float.tap tests/auto/testlib/selftests/expected_float.teamcity tests/auto/testlib/selftests/expected_float.txt tests/auto/testlib/selftests/expected_float.xunitxml Done-With: Christian Ehrlicher <ch.ehrlicher@gmx.de> Done-With: Edward Welbourne <edward.welbourne@qt.io> Done-With: Timur Pocheptsov <timur.pocheptsov@qt.io> Change-Id: If93cc432a56ae3ac1b6533d0028e4dc497415a52
| * generate_expected_output.py: placate FutureWarning from pythonEdward Welbourne2019-01-311-1/+1
| | | | | | | | | | | | | | | | | | | | Regexes have long specified that a [ as the first character inside a [...] is just a literal [, but apparently we need to escape it now, to avoid a "nested set" FutureWarning. Change-Id: I76a48c9aafb0684a1d6b0d5284fe9852c9ea0e43 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io> Reviewed-by: Samuel Gaist <samuel.gaist@idiap.ch>
* | Add a testlib selftest for the watchdogEdward Welbourne2018-12-061-3/+6
| | | | | | | | | | | | | | | | Verify that it does cut in after the specified time has elapsed. Task-number: QTPM-1385 Change-Id: Ib18e8d6af28339f79cca4d62b869287ce07b8cc1 Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
* | generate_expected_output.py: refine environment used for subtestsEdward Welbourne2018-12-061-26/+64
| | | | | | | | | | | | | | | | | | Match the environment tst_selftests.cpp uses for subtests more faithfully. Extends b22e50acda. In the process, tweak how crashers are handling, in preparation for the watchdog test. Change-Id: I09a046460f6f3bff0b12069fad6c1437d89572ce Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
* | Test that mismatches of datatype in test data columns are rejectedEdward Welbourne2018-12-061-2/+2
| | | | | | | | | | | | | | | | | | | | One test for bad data for the column, another for a bad QFETCH. Incidentally extend blacklist testing by blacklisting them. Reorganise a QEMU condition that needed extended as part of this. Task-number: QTPM-1385 Change-Id: Iac72ada19760321c5c9264ddfff7740d1fdd0700 Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
* | generate_expected_output.py: match tst_selftest's test environmentsEdward Welbourne2018-11-011-2/+21
|/ | | | | | | | | The testlib selftest sets various things in the environment for crashing tests; the generator for its expected output should set the same things, as they affect what output is produced. Change-Id: Iec2ed59982ea1043582573530c33619d8e8ed08e Reviewed-by: Jędrzej Nowacki <jedrzej.nowacki@qt.io>
* Remove mention of selftest.qrc from generator script's docEdward Welbourne2018-05-081-3/+1
| | | | | | | | This supplements b1945604a78626ed25ad8afe70a64c2eac76a2b4, which removed the qrc file in favor of test/test.pro coding for it. Change-Id: I15507c89ca14fa6e6b8223de671ffff7092272d0 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Make generated selftest output match for in-source buildsEdward Welbourne2018-04-171-5/+17
| | | | | | | | | | | When generate_expected_output.py is run for an in-source build, the raw output contains no paths to the sources for the script to whittle down, as it does for shadow builds, to just the path from qtbase down. So kludge together some extra regexes that can fix that up and tweak some relevant code to provide them with the data they need. Change-Id: I656d7126087bd9ad20b2af6835fba314d90a171d Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
* Merge remote-tracking branch 'origin/5.11' into devQt Forward Merge Bot2018-03-151-1/+1
|\ | | | | | | Change-Id: I8b5a10d897a926078895ae41f48cdbd2474902b8
| * testlib: Silence all Qt logging in selftestsTor Arne Vestbø2018-03-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | Otherwise random debug messages from Qt might mess up the expected vs actual results. The setting of QT_LOGGING_RULES in initTestcase has been removed, as the selftest overrides that for each invocation of a subtests, via the processEnvironment() function. Task-number: QTQAINFRA-1631 Change-Id: I855d31274f8261f8b125df23409353f7101be0e4 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
* | testlib: Add Test Anything Protocol (TAP) reporterTor Arne Vestbø2018-03-141-1/+1
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The Test Anything Protocol (TAP), was originally Perl's simple text-based interface between testing modules and test harnesses, but has since been adopted by a large number of producers and consumers in many different languages, which allows colorizing and summarizing test results. The format is very simple: TAP version 13 ok 1 - test description not ok 2 - test description --- message: 'Failure message' severity: fail expected: 123 actual: 456 ... ok 3 - test description # SKIP 1..3 The specification [1] is very brief, so the implementation has been based on how typical consumers behave, especially when it comes to the undefined diagnostics block. [1] http://testanything.org/tap-version-13-specification.html Change-Id: I616e802ea380165c678510e940ddc6607d39c92d Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
* Force QT_LOGGING_RULES in testlib's selftestEdward Welbourne2017-10-111-1/+7
| | | | | | | | | | | | | | | | | | On some systems (e.g. Debian/buster) a standard Qt package may install a qtlogging.ini file, e.g. in /etc/xdg/QtProject/, which suppresses QDEBUG output. A ~/.config/QtProject/qtlogging.ini could interfere similarly. This can break the selftest, when run with expected_* files generated without such interference. Likewise, if those expected_* files are generated on a system such similar, they won't work on CI. Given that this caused confusion and delay with integrating the latest set of updates to the expected_* files, it seemed best to save others from similar bafflement. So set a standard value for QT_LOGGING_RULES in both the selftest and the generator script, so we get consistency. Change-Id: I649e2f1f6ead21edf8af051aaee286e369fed064 Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
* generate_expected_output.py: more refinements to filtering of noiseEdward Welbourne2017-06-131-5/+6
| | | | | | | | Two more ways line numbers were making it through. Corrected a doc-string to tell nearer to the truth. Change-Id: I946aaeb936d47fffe50d7ec15e2524992cc9e428 Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
* generate_expected_output.py: refer to tst_selftests.cpp for test listEdward Welbourne2017-06-131-11/+76
| | | | | | | | Document that the saved output is used by tst_selftests.cpp and use a crude parse of it to get the list of subdirs that it actually tests. Change-Id: I73023228c9e547f965b7749dd66de7ef09c3815e Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
* Merge remote-tracking branch 'origin/5.6' into 5.7Liang Qi2016-09-051-82/+177
|\ | | | | | | | | | | | | Conflicts: tests/auto/testlib/selftests/generate_expected_output.py Change-Id: If856162abf9a24ae2c9946d336a7d1da03520fa7
| * Major re-write of generate_expected_output.pyEdward Welbourne2016-08-291-82/+178
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Restructured, separated the canonicalising of output lines out (into an object that prepares the necessary regexes and replacements) suppress more, changed the path-stripping to strip qtbase's parent rather than os.getcwd() and took account of shadow builds (so both source tree and build tree provide prefixes we want to strip from paths). Also cope with $PWD potentially having symlinks in it, where os.getcwd() is canonical. It's possible some output might name files elsewhere in the source tree; these won't be filtered by the prior cwd prefix removal; and, in any case, the problem with cwd is only that the ancestry of qtbase is apt to vary; paths relative to there should be consistent between test runs. This change shall lead to a one-off rewrite of all expected_* files; but it should now catch all paths. By stripping both build root and source root (when different) it also avoids differences for those doing out-of-source ("shadow") builds. In our XML formats, any hyphens in root paths (e.g. I had Qt-5.6 in my build root's path) got represented by a character entity, confounding the replacement; so also do replacement that catches this. We may discover other character entity subsitutions needed along with this. Now filtering line numbers and timing information, including benchmark results; these numbers all get replaced with 0 to avoid noisy diffs. Also purging dangling hspace, to placate sanity-bot. The module can now be imported - the code it runs is packaged as a main() function that a __name__ == '__main__' stanza runs - and all data is localised to where it's needed, rather than held in globals. Tidied up and organized the existing regexes. There are doc-strings; there is a short usage comment. Data is localised rather than global and modern pythonic idioms get used where apt. Regexes are compiled once instead of repeatedly. An object looks after the list of patterns to apply and its construction handles all anticipated problems. Failures are mediated by an exception. The output file now gets written once, instead of twice (once before editing, then over-write to edit), and Popen uses text mode, so that write can do the same. Its command is delivered as an array, avoiding the need to invoke a shell. Instead of relying on qmake being in our path (which might give us a bogus QT_VERSION if the one in path doesn't match our build tree), use the relative path to qmake - we rely on being run in a specific directory in the build tree, after all. Escape dots in the version properly, so that 51730 doesn't get mistaken for 5.7.0 (for example), and moved this check later in the sequence (matching a smaller target makes it more likely to falsely match). Overtly check we are in the right directory and tell the user what we actually need, if run from the wrong place. Simplify handling of the unsupported use-case for MS-Windows (but note what would be needed for it). Change-Id: Ibdff8f8cae173f6c31492648148cc345ae29022b Reviewed-by: Mitch Curtis <mitch.curtis@qt.io> Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
* | Merge remote-tracking branch 'origin/5.6' into 5.7Edward Welbourne2016-07-151-0/+1
|\| | | | | | | | | | | | | | | | | | | | | Conflicts: qmake/library/qmakeevaluator.cpp One side changed the iterator to use ranged-for, the other changed its body; they only conflicted because the latter had to add braces around the body, intruding on the for-line. Trivial resolution. Change-Id: Ib487bc3bd6e3c5225db15f94b9a8f6caaa33456b
| * Testlib: generate_expected_output.py: Run in C localeFriedemann Kleint2016-07-051-0/+1
| | | | | | | | | | | | | | | | Otherwise, float numbers are formatted using a decimal comma in German, causing huge diffs and failures. Change-Id: Icd85a293d0564cac6be244eb0793611920d0c89c Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-13 07:20:07 +0000