summaryrefslogtreecommitdiffstats
path: root/util/testrunner
Commit message (Collapse)AuthorAgeFilesLines
* qt-testrunner: Commit forgotten README fileDimitrios Apostolou2022-06-161-0/+24
| | | | | | | Change-Id: Ie307b95947a6910bd65bb22ecf938fae9de22f9d Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> (cherry picked from commit 1ab354b2453bce60cd5cd800d87a83fc9e30a0c6) Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>
* Add an exception in testrunner for QDocLuca Di Sera2022-05-191-1/+3
| | | | | | | | | | | | | | | QDoc recently added some tests that use Catch2 instead of testlib. Similarly to selftests, the normal behavior of `qt-testrunner.py` will fail, trying to pass an argument that is not supported by Catch2 when running the tests in CI. To avoid the issue, an exception for the test executables that QDoc uses was added to `qt-testrunner.py`, using the already available mechanism that is used for selftests. Change-Id: I5eb4b75f239eda0c493fb06ece164545d159372b Reviewed-by: Topi Reiniö <topi.reinio@qt.io> Reviewed-by: Dimitrios Apostolou <jimis@qt.io>
* Use SPDX license identifiersLucie Gérard2022-05-163-87/+6
| | | | | | | | | | | | | Replace the current license disclaimer in files by a SPDX-License-Identifier. Files that have to be modified by hand are modified. License files are organized under LICENSES directory. Task-number: QTBUG-67283 Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
* Generalize exclusion listDimitrios Apostolou2022-04-221-4/+8
| | | | | | | | | | It seems there are some tests in the various Qt modules that skip writing the XML testlog file. If there are good reasons for that (for example test is a custom shellscript) then it should be added to the DUMB_TESTS list. Change-Id: Iffb00e1fda42bb8e7338a94c9320e45900db1bc1 Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
* qt-testrunner: improve error loggingDimitrios Apostolou2022-04-221-9/+12
| | | | | | | | Do not print full stack traces when a controlled exception occurs, instead print just the exception name and the message. Change-Id: I5669862d2e98c550d88d1d11fb2bf49f197cfc17 Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
* qt-testrunner: be more strict if bad XML log files are writtenDimitrios Apostolou2022-04-052-11/+34
| | | | | | | | | | | | | | | | | | | | | | | If a test returns 0 but writes an XML logfile that contains FAIL or a corrupted XML file, then qt-testrunner considers it a CRASH and exits with 3. Previously any test execution returning 0 (success) was considered a PASS. Changing this behavior with this patch finds a lot of test crashes on Android (QTBUG-100470), because the tests are run indirectly on the emulator and the test wrapper process could not detect the crash, thus returning 0 to qt-testrunner. But the corrupt XML file is caught now. Likewise, if a test returns != 0 but the XML logfile contains only PASS, qt-testrunner considers it a FAIL. This used to be the case but now tests are added. Finally changed logging for such cases from INFO to WARNING. Task-number: QTBUG-100470 Change-Id: I404c9d2211c7de027bf776d1914519d37f513ca1 Reviewed-by: Assam Boudjelthia <assam.boudjelthia@qt.io>
* Clean up QNX platform configs and improve testrunner scriptsDimitrios Apostolou2022-02-141-0/+7
| | | | | | | | | | | | | | | | | | | | QNX requires its own TESTRUNNER script in order to SSH to the QEMU guest and run the tests. Simplify it a lot and fix the way it passes arguments so that they retain spaces and quotation. Also wrap it with the generic TESTRUNNER script for CI, which is qt-testrunner.py, so that tests can re-run in case of flakiness. Delete prefix.sh as the environment variables are now properly set in the Coin platform config files. Avoid executing extra chmod commands to make scripts executable, but use the right Coin instruction for that. For reference, we use 493 for file mode which equals to 755 in octal, as Coin expects the file mode in decimal. Change-Id: Ife4d1caef606f48b92ba1da1cfb14ec0dea11ef2 Reviewed-by: Pasi Petäjäjärvi <pasi.petajajarvi@qt.io>
* Write XML test log files even for the flaky test re-runsDimitrios Apostolou2022-01-301-6/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously the qt-testrunner script was avoiding to write XML log files for the flaky test re-runs, since parsing them always provides limited information about what went wrong (for example unparseable files) and might miss issues completely (for example a crashed test might not even write a log file), or even mislead if multiple different data points about a single testcase end up into the database.. But currently Coin is the system that uploads test information to the testresults database and expects to find such log files in an attempt to send as much information as possible regarding failures. Re-enabling these log files will allow Coin to deduce as much information as possible about the test runs. This is a temporary step at least, until the test-uploading logic is separated from Coin. Then it will be easier to find the full information about what happened with flaky or crashed tests by special logs from qt-testrunner.py, which is the test re-running orchestrator and therefore has more knowledge about what went wrong. Fixes: QTQAINFRA-4754 Change-Id: I69e4469cbe6c0a6ca5b2473eaf8d034632564342 Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io> Reviewed-by: Toni Saario <toni.saario@qt.io> Reviewed-by: Simo Fält <simo.falt@qt.io>
* qt-testrunner: do not try to re-run initTestCase and othersDimitrios Apostolou2022-01-304-23/+53
| | | | | | | | | | | | | Some function names are special for qtestlib, in the sense that they can not be specified as a command line argument to run individually. In such cases qt-testrunner treats the failure specially and tries once to re-run the full test executable. Fixes: QTBUG-89011 Change-Id: I0cc25f91c57374e5ac65ade10e2e223fe969f211 Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io> Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
* qt-testrunner: be more clear that it was the test that crashedDimitrios Apostolou2022-01-181-4/+9
| | | | | | | Task-number: QTBUG-99970 Change-Id: Id2d01b6ab7d428356d9dfc953107014791393d35 Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io> Reviewed-by: Jani Heikkinen <jani.heikkinen@qt.io>
* qt-testrunner: disable JUnit XML test logsDimitrios Apostolou2022-01-121-3/+1
| | | | | | | | It causes crashes because of interleaved execution of Quick tests. Task-number: QTBUG-98350 Change-Id: I61e26f5da5617e06810db8eb7991f17eecdd3acb Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Enable verbose logging in the last re-run of a failing testDimitrios Apostolou2022-01-033-8/+29
| | | | | | Fixes: COIN-728 Change-Id: I08802e377e26e5dd7f7d1c44b5efe4280b09f957 Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
* Introduce qt-testrunner.pyDimitrios Apostolou2022-01-034-0/+878
Script that wraps Qt test execution in order to iron-out flakiness. Task-number: QTBUG-96353 Change-Id: Ie8ee780e3f4f8d086c080a7784f9f68fd1867be2 Reviewed-by: Daniel Smith <Daniel.Smith@qt.io>
generated by cgit v1.2.3 (git 2.25.1) at 2024-06-20 07:01:13 +0000