summaryrefslogtreecommitdiffstats
path: root/tests/auto/testlib/selftests
Commit message (Collapse)AuthorAgeFilesLines
* Added -datatags option to QTestLibJo Asplin2011-11-049-1/+249
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (Note: This feature is ported from Qt 4.8. See the following commits: 01575deafb7d26ca2431374e92c6d71de96547c7 4866d1ba8afbab61e102942d1ea93b81fea053d6 ) Passing the -datatags option to a QTestLib program prints the available data tags to standard output. For completeness, the test case name is also printed at the start of each output line. (Although the file name is supposed to match the lower-case version of the test case name, this is currently not true in all cases (particularly not under tests/benchmarks/). Even if there was a script to enforce this convention, the -datatags option provides this information in a reliable way.) Data tags for each test function (f() in this case) are printed in four different ways depending on the presence of local and global data tags: Case 1: No tags: tst_MyTestCasetst_MyTestCase f Case 2: Local tags only: tst_MyTestCase f local tag 1 tst_MyTestCase f local tag 2 ... Case 3: Global tags only: tst_MyTestCase f __global__ global tag 1 tst_MyTestCase f __global__ global tag 2 ... Case 4: Local and global tags: tst_MyTestCase f local tag 1 __global__ global tag 1 tst_MyTestCase f local tag 2 __global__ global tag 1 ... tst_MyTestCase f local tag 1 __global__ global tag 2 tst_MyTestCase f local tag 2 __global__ global tag 2 ... ... Note that the string __global__ is assumed to be highly unlikely to occur in a data tag (if it does, an ambiguity results). Change-Id: Ib51aa0c3c32ad52e52ce519729292cf8f0ec5d50 Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com> Reviewed-by: Sergio Ahumada <sergio.ahumada@nokia.com>
* Allow extra selftests without modifying source.Jason McDonald2011-11-031-7/+10
| | | | | Change-Id: I21708b725611eb64bdb0414cee08ca00dd62b213 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* all remaining tests: eliminated usage of qttest_p4.prfRohan McGovern2011-10-2535-203/+34
| | | | | | | | | qttest_p4.prf was added as a convenience for Qt's own autotests in Qt4. It enables various crufty undocumented magic, of dubious value. Stop using it, and explicitly enable the things from it which we want. Change-Id: I02fe27b2c1800f929250fa8694ca2976c9661a12 Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
* Remove QTest::SkipMode from qtestlib API.Jason McDonald2011-10-2118-22/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | When the SkipAll mode is used, tests only report a SKIP for the first line of test data and subsequent lines are not reported at all. This behaviour makes it impossible for anything post-processing test results to accurately report test pass- and run- rates because they cannot see how many lines of test data were skipped. This commit removes SkipMode. QSKIPs in regular test functions and data functions are treated the same as SkipSingle, so that every skipped line of local or global test data is reported in the test log. QSKIPs elsewhere are treated the same as SkipAll -- skipping in init() causes the next test function to be skipped entirely, and skipping in initTestCase() or initTestCase_data() causes all test functions to be skipped. This commit only changes qtestlib and the selftests. A further commit will change the autotests to remove the SkipMode parameter from QSKIP calls. Note that the change in expected output for the globaldata selftest is deliberate, as the QSKIP in the skipLocal test function has effectively changed from SkipAll to SkipSingle. Task-number: QTBUG-21851, QTBUG-21652 Change-Id: I7b1c53fe7ca9dde032810b789d967e2a402bbe5d Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com> Reviewed-by: Alex <alex.blasche@nokia.com>
* Sanitize paths in selftest data.Jason McDonald2011-10-1950-174/+174
| | | | | | | | Use a standard path for filenames that appear in the selftest's expected test data. This will make future patches smaller. Change-Id: I04b2e739d261f80d20b834e5b33c5b6e88d26379 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Remove skipglobal testlib selftest.Jason McDonald2011-10-199-172/+1
| | | | | | | | This test duplicates the skipinitdata selftest and has slightly less informative output. Change-Id: Ifd40e3ef8030059ec8fa0089ce5b2a994624abeb Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Reduce the runtime of tst_selftestsRohan McGovern2011-10-181-0/+6
| | | | | | | | | | | The addition of testing for multiple loggers greatly increased the total runtime of this test, in fact making it the slowest testcase in Qt. Fortunately this is only due to a couple of slow subtests whose behavior is unlikely to be affected by the loggers. Change it to run these slow subtests just for a couple of loggers, instead of all the combinations. Change-Id: Ie90f6c0ca29470ed6a7c4e2e185f852602a2d162 Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
* Make QTest::qWarn() use __FILE__ and __LINE__.Jason McDonald2011-09-281-6/+7
| | | | | | | | | | This change makes QTest::qWarn() consistent with similar testlib functions, which take __FILE__ and __LINE__. Change-Id: I12977cb30672899ab38877b4a656f0cc0c7ea02c Reviewed-on: http://codereview.qt-project.org/3559 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Re-enabled tests that are now passing on LinuxJo Asplin2011-09-261-2/+0
| | | | | | | | | Task-number: QTBUG-21424 Change-Id: I093d90a2713a7e7ddace31a27a08d47e3c2da588 Reviewed-on: http://codereview.qt-project.org/5546 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Matthew Cattell <matthew.cattell@nokia.com>
* Tidy and sort the list of selftestsJason McDonald2011-09-261-44/+38
| | | | | | | Change-Id: I123a775e2d56fc0c64d43e98a1d2e3ab68394076 Reviewed-on: http://codereview.qt-project.org/5492 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Use placeholder for Qt version in expected outputJason McDonald2011-09-2639-53/+51
| | | | | | | | | | | | | The selftests already used the @INSERT_QT_VERSION_HERE@ placeholder in place of the real Qt and QTestlib versions in the expected test output of the subtests. For unknown reasons, the same was not true for the expected plain text output. In the past, this has caused Release Managers to waste time incrementing the version numbers in these files. Change-Id: I52f7870486fce128c04d53ff06978afa947474fd Reviewed-on: http://codereview.qt-project.org/5375 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Remove obsolete exclusion from selftest checkingJason McDonald2011-09-261-3/+0
| | | | | | | | | There is no test with a datatag "float", so this code is never executed. Change-Id: I9dd234e6575eb1c33075705edf77eb1d977061c9 Reviewed-on: http://codereview.qt-project.org/5362 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Don't XFAIL on the contents of assert messages.Jason McDonald2011-09-261-28/+6
| | | | | | | | | | | | | | | Q_ASSERT uses __FILE__, which has undefined contents, and thus the selftest cannot always match Q_ASSERT messages in actual output with those in expected output. The test was calling QEXPECT_FAIL() in this case, but doing so implies that (a) it's a failure and (b) we intend to fix it one day. Neither of these is true, so instead we should simply tolerate this difference in actual and expected outputs, in the same way that we do in a couple of other similar cases. Change-Id: I8b8609a400e9b798936875f6eb1ca18a83660efc Reviewed-on: http://codereview.qt-project.org/5358 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Allow tests to log to multiple destinationsJason McDonald2011-09-231-20/+116
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Each destination and the format of output to write there is specified by adding "-o filename,format" to the command-line. The special filename "-" indicates that the log output is written to the standard output stream, though standard output can be used as a destination at most once. The old-style testlib output options are still supported, but can only be used to specify one logging destination, as before. If no logging options are given on the command-line, a plain text log will go to the console, as before. To log to the console in plain text and to the file "test_output" in xunit format, one would invoke a test in the following way: tst_foo -o test_output,xunitxml -o -,txt This commit also enhances the selftests to test with multiple loggers, but negative tests (e.g. bad combinations of command-line options) are left for future task QTBUG-21567. Task-number: QTBUG-20615 Change-Id: If91e752bc7001657e15e427aba9d25ab0a29a0b0 Reviewed-on: http://codereview.qt-project.org/4125 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Enable testlib self-testing with multiple loggersJason McDonald2011-09-231-129/+164
| | | | | | | | | | | | Modify the selftest to be able to run each subtest with a list of one or more test loggers. The addition of tests that use this capability will be part of a subsequent commit. Task-number: QTBUG-20615 Change-Id: Iac3efe8220e8245aa7e5589348d2c86b8034dd28 Reviewed-on: http://codereview.qt-project.org/5292 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Don't run multiexec test with file loggingJason McDonald2011-09-211-2/+7
| | | | | | | | | | | | | | The multiexec test runs the same test object five times. If the -o option is given, the output file is overwritten by each run of the test object, meaning that tst_selftest only sees 1/5 of the test output in a file compared with what it sees on the console. This makes it impossible to use the expected output file to verify the test for both console and file output. This issue is noted in QTBUG-21561. Change-Id: I00031a2ea43a7ef78e8317473a089306ec062d8e Reviewed-on: http://codereview.qt-project.org/5270 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Remove printf calls from warnings selftestJason McDonald2011-09-212-18/+1
| | | | | | | | | | | | | | | | | The warnings selftest verifies that qtestlib correctly handles output sent via qDebug() and friends. The test had a number of printf calls that were used to tell the user what output they should expect to see next, but by bypassing testlib's logging, these made the testlog differ depending on whether the test log was directed to the console or a file. The printf calls are no longer needed, as any regressions in qDebug will be detected by tst_selftests, which compares the output of the warnings test with a reference copy of the expected output. Change-Id: I8c3a3237463141fa494c50ca02062760cb583090 Reviewed-on: http://codereview.qt-project.org/5200 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Improve globaldata testlib selftest.Jason McDonald2011-09-205-84/+394
| | | | | | | | | | | | | Make the test use qDebug instead of printf, so that the reference test output is the same regardless of whether the log goes to the console or a file. Also rename the test class, as it was evidently cut-n-pasted from the subtest selftest without being renamed. Change-Id: I8df1f0989f2e8c33c1f829ba8110fd740ff70f63 Reviewed-on: http://codereview.qt-project.org/5184 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Split subtest selftestJason McDonald2011-09-2014-246/+655
| | | | | | | | | | | | This test was attempting to verify two completely unrelated things, so this commit splits it into two tests. Also, printf calls are replaced by qDebug so that the test does not bypass the testlib loggers. Change-Id: I1a202af38ce2c69690a32d93405ba604ec6cabee Reviewed-on: http://codereview.qt-project.org/5178 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Remove waitwithoutgui testlib selftestJason McDonald2011-09-209-86/+1
| | | | | | | | | | | | | This test is not useful for finding bugs in qtestlib's logging code, because it bypasses the qtestlib loggers and doesn't play nice with tst_selftest. Neither is this test very useful for finding bugs in QTest::qWait(), as the test only proves the qWait() terminates, not that it waits accurately, or even that it waits at all. Change-Id: Ia5dd7cbaf3a6fbb4e94e54ed155263580e495694 Reviewed-on: http://codereview.qt-project.org/5173 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Fix autotest gui/widgets dependenciesKent Hansen2011-09-192-1/+2
| | | | | | | | | | | | | | | | | This is in preparation of removing testlib's dependency on QtGui and QtWidgets. Autotests that need QtWidgets api must explicitly include it (since the types are no longer provided by the QtGui master header). Autotests that don't need QtGui or QtWidgets api shouldn't link against those libraries. Change-Id: I2808289068514fcac582808828ad4634e2631733 Reviewed-on: http://codereview.qt-project.org/5093 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Lars Knoll <lars.knoll@nokia.com>
* Simplify selftestJason McDonald2011-09-171-17/+11
| | | | | | | | | | Don't store separate strings for the logger name and the associated file suffix -- just use the same string everywhere. Change-Id: Ie7d1af6bf906b5ac09fbd5fcc5754b68036fb370 Reviewed-on: http://codereview.qt-project.org/5060 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Improve cleanup of selftestsJason McDonald2011-09-171-2/+3
| | | | | | | | | | | | The selftest was only attempting to delete the output file at the end of the entire test run. This file should actually be deleted after each subtest to ensure that it does not cause confusing errors if the next subtest fails to overwrite the output file. Change-Id: Ia3827926f71a697c6108de3f2b08c969f761d978 Reviewed-on: http://codereview.qt-project.org/5051 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Remove obsolete QEXPECT_FAIL callsJason McDonald2011-09-171-2/+0
| | | | | | | | | | The removed calls related to tests that were removed when redundant logging functionality was removed. Change-Id: Iaebbc109eaaddb440ae78e1f4a1b4e874a8b2960 Reviewed-on: http://codereview.qt-project.org/5050 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Improve compliance with Qt coding guidelinesJason McDonald2011-09-171-60/+48
| | | | | | | | | | | | * Use C++ comment delimiters consistently * Use a space after control flow keywords * Don't nest "else if" blocks unnecessarily * Follow indenting rules * Added comments describing some internal functions Change-Id: I72fd1eaf5c5f3130945fbd2b64fa59e19ad2913d Reviewed-on: http://codereview.qt-project.org/5049 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Fix XFAIL in testlib selftestJason McDonald2011-09-174-5/+5
| | | | | | | | | | | Commit 949be8cbdd2d346647accbbd9bdab879256470ec modified the assert test case but failed to update the line number of the assert in the expected test output files. Change-Id: I0d4f55c0b8bc1de8487dd8c6267aa459000cda26 Reviewed-on: http://codereview.qt-project.org/5048 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Disabled tests for refactor->master integrationJo Asplin2011-09-131-1/+1
| | | | | | | | | | | | | | | | | | | | | | | This change disables failing tests to increase the likelihood that 'make check' terminates with a zero exit status. This is done to prevent the CI system from blocking further commits once qtbase#refactor is merged into qtbase#master. Failing tests are marked as insignificant (by putting 'CONFIG += insignificant_test' in the .pro file). Note: This is a temporary measure that needs to be cleaned up once the refactor->master integration is complete. Ideally all disabled tests need to be enabled and passing. All changes will be marked by the string QTBUG-21402 in a comment for easy location. Task-number: QTBUG-21402 Change-Id: Ic4400671671f6d9b75b106a6aa4755b20e74378d Reviewed-on: http://codereview.qt-project.org/4678 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Gunnar Sletta <gunnar.sletta@nokia.com>
* Moved autotests in category 'testlib' into new directory structureJo Asplin2011-09-12211-0/+26140
Task-number: QTBUG-21353 Change-Id: I2e2ad4b7570f80b3e6500821f769e967f0a75027 Reviewed-on: http://codereview.qt-project.org/4438 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-01 00:08:27 +0000