summaryrefslogtreecommitdiffstats
path: root/src/testlib/qtestresult.cpp
Commit message (Collapse)AuthorAgeFilesLines
* Updated license headersJani Heikkinen2016-01-151-14/+20
| | | | | | | | | | | From Qt 5.7 -> LGPL v2.1 isn't an option anymore, see http://blog.qt.io/blog/2016/01/13/new-agreement-with-the-kde-free-qt-foundation/ Updated license headers to use new LGPL header instead of LGPL21 one (in those files which will be under LGPL v3) Change-Id: I046ec3e47b1876cd7b4b0353a576b352e3a946d9 Reviewed-by: Lars Knoll <lars.knoll@theqtcompany.com>
* Testlib: Avoid formatting unneeded messages.Friedemann Kleint2015-10-301-9/+11
| | | | | | | | | Brings down gui/painting/qcolor from 7s to 3s, reducing the calls to printf() helpers from 16E6 to 10E6. Task-number: QTBUG-38890 Change-Id: I34065e6f09fc9a14920b06aa6936908229f3f9c4 Reviewed-by: Frederik Gladhorn <frederik.gladhorn@theqtcompany.com>
* Update copyright headersJani Heikkinen2015-02-111-7/+7
| | | | | | | | | | | | | | | | | | Qt copyrights are now in The Qt Company, so we could update the source code headers accordingly. In the same go we should also fix the links to point to qt.io. Outdated header.LGPL removed (use header.LGPL21 instead) Old header.LGPL3 renamed to header.LGPL3-COMM to match actual licensing combination. New header.LGPL-COMM taken in the use file which were using old header.LGPL3 (src/plugins/platforms/android/extract.cpp) Added new header.LGPL3 containing Commercial + LGPLv3 + GPLv2 license combination Change-Id: I6f49b819a8a20cc4f88b794a8f6726d975e8ffbe Reviewed-by: Matti Paaso <matti.paaso@theqtcompany.com>
* Update license headers and add new license filesMatti Paaso2014-09-241-19/+11
| | | | | | | | | - Renamed LICENSE.LGPL to LICENSE.LGPLv21 - Added LICENSE.LGPLv3 - Removed LICENSE.GPL Change-Id: Iec3406e3eb3f133be549092015cefe33d259a3f2 Reviewed-by: Iikka Eklund <iikka.eklund@digia.com>
* Add support for blacklisting test functionsLars Knoll2014-09-211-2/+15
| | | | | | | | | | | | | | | | | We need to have a finer grained control over the tests we skip in our CI system. This adds a blacklisting mechanism that allows blacklisting individual test functions (or even test data) using a set of predefined matching keys for the operating system and some other relevant variables. QTestlib will search for a file called BLACKLIST in the test directory and parse it if found. The file contains a simple ini style list of functions to blacklist. For details see qtestblacklist.cpp. Change-Id: Id3fae4b264ca99970cbf9f45bfb85fa75c1fd823 Reviewed-by: Lars Knoll <lars.knoll@digia.com>
* Fix auto test failure output on QNXBernd Weimer2014-05-191-8/+9
| | | | | | | | | On QNX mbstowcs does not ignore last parameter (max) when first (dest) is NULL. Set it to sufficiently large value to yield proper results on QNX. Other platforms (standard libraries) will ignore this value anyway. Change-Id: Ie4695254d45082e151a052bf16de684af3b1ba1e Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* Remove duplicate symbol name: QTest::currentAppNameThiago Macieira2014-04-301-4/+4
| | | | | | | | | | It's declared in qtestcase.h as a function, so let's not declare the one in qtestresult.cpp as a static variable. None of the variables in qtestresult.cpp need to be in the QTest namespace, but we don't need to change them now. Change-Id: If6cc34642fdfe3ccda3b8cea7d053ead0db9ccbd Reviewed-by: Olivier Goffart <ogoffart@woboq.com>
* Fix the alignment for non-ASCII stringsThiago Macieira2014-01-221-2/+4
| | | | | | | | | | | | | | | | | | d040681b6f3d03b349e9b9487fe89a611d03ee3c added support for aligning the test results for easier side-by-side comparison of the actual and expected values. However, it didn't take into account multibyte strings. That is, we would see: FAIL! tst_testcase::testcase: Compared values are not the same Actual (QString("é")): F0O Expected (expected) : FOO We use mbstowcs (multibyte string to wide char string) that calculates the length in wide chars of the output string. That's roughly equivalent to QString::fromLocal8Bit(string).toUcs4().size(). Change-Id: Ic2649951c50e05143da32a7fbef00a01e385c542 Reviewed-by: Jason McDonald <macadder1@gmail.com>
* Expose QTest::currentAppName() and remove hard-coded argv[0] in testsTor Arne Vestbø2013-09-261-5/+5
| | | | | | | | Except where we're actually testing QCoreApplication::applicationName() and friends. Change-Id: I25514884c11f43a4f82b1f818f822dc3d79f69a3 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@digia.com>
* TestLib: align the colons so the "actual" and "expected" align tooThiago Macieira2013-07-211-3/+5
| | | | | | | | | | | | | | | Before: FAIL! tst_testcase::testcase: Compared values are not the same Actual (actual): F0O Expected (expected): FOO Now: FAIL! tst_testcase::testcase: Compared values are not the same Actual (actual) : F0O Expected (expected): FOO Change-Id: I6f0768e4ef53e065b85a56879cecbad06fa34aef Reviewed-by: Jason McDonald <macadder1@gmail.com>
* Update copyright year in Digia's license headersSergio Ahumada2013-01-181-1/+1
| | | | | Change-Id: Ic804938fc352291d011800d21e549c10acac66fb Reviewed-by: Lars Knoll <lars.knoll@digia.com>
* Change copyrights from Nokia to DigiaIikka Eklund2012-09-221-24/+24
| | | | | | | | Change copyrights and license headers from Nokia to Digia Change-Id: If1cc974286d29fd01ec6c19dd4719a67f4c3f00e Reviewed-by: Lars Knoll <lars.knoll@digia.com> Reviewed-by: Sergio Ahumada <sergio.ahumada@digia.com>
* testlib: Remove obsolete internal compare_helper overload.Jason McDonald2012-03-211-9/+0
| | | | | Change-Id: Ic98faf360a713ac698f9bf1ff8aaad5a4c5c176b Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* testlib: Improve formatting of QCOMPARE failure messages.Jason McDonald2012-03-121-1/+1
| | | | | | | | | | | Make the various versions of the failure message align consistently so that it's a little easier to compare the actual and expected values. Of course, the value won't align nicely unless the "actual" and "expected" strings are the same length, but at least this commit makes that consistent across all versions of the message. Change-Id: If9ce231df3b5d279a06f6458fdb5da0aa4586068 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* testlib: Make QTestResult::verify() more robust.Jason McDonald2012-03-081-1/+3
| | | | | | | | | | The statementStr parameter should always be non-null, so assert if it is null. The description parameter can be null in some cases (particularly when the verify is successful, and thus no error description is going to be displayed), so tolerate this. Change-Id: I87b416d5f3b793bc608cd4aca14a4f7fe7527488 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* testlib: Improve verbose and XPASS outputJason McDonald2012-03-081-9/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, verbose (-v2) and XPASS test output showed all QCOMPAREs as "COMPARE()", making it impossible to see what was compared and difficult to match the output to the source of a test containing many calls to QCOMPARE. This commit changes testlib's internal compare_helper API so that string representations of the compared expressions are always passed to QTestResult::compare() when available, and can thus be shown in the verbose and XPASS output. The XPASS output has also been changed to state explicitly that the comparison succeeded unexpectedly, bringing it in line with the XPASS output resulting from a call to QVERIFY. This commit also changes all calls to compare_helper() to call the eight-argument version of the function, which simplifies much of the calling code. The now obsolete four-argument version of compare_helper() has been changed to output a warning that it is obsolete. It will be removed once other modules have had some time to catch up. The improved XPASS and verbose output is demonstrated by the expectfail and verbose2 selftests. Change-Id: I8baa46d5dd30e6c43b26f366c34dc5b64aab5f7c Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* testlib: make XPASS output slightly less confusing.Jason McDonald2012-02-201-1/+4
| | | | | | | | | | | | | | Prior to this commit, when an unexpected pass occurred the test output showed the message "XPASS : tst_foo::function() 'expr' returned FALSE", where the problem was actually that the expression evaluated to true when it was expected to be false. This commit changes the output to make it clear that the expression evaluated to true unexpectedly. Task-number: QTBUG-22118 Change-Id: Id22c178073d3b75789675ca37a8ef019029b1f91 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Remove code related to test location.Jason McDonald2012-02-161-15/+0
| | | | | | | | Testlib no longer does anything with the test location, and neither do any of Qt's tests, so the code is no longer needed. Change-Id: I1a464116179f3e26efadad6901f00f3c33e0ce84 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* testlib: Count passes, fails and skips consistently.Jason McDonald2012-02-131-13/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For data-driven tests, testlib previously counted one fail or skip for each data row that failed or skipped, while it counted only one pass for a test function where all rows passed and counted no passes for a test function where some rows passed and some rows failed. A similar problem also existed for benchmark tests, which could run multiple iterations of the same test, with each fail and skip being counted but only a single pass being counted for the entire series of iterations. This commit makes testlib count one pass, fail or skip for each data row. Test functions that are not data-driven count one result for the test function, as before. Benchmark tests count one pass, fail or skip per iteration. A side-effect of this change is that the test output in plain text, xml and light xml formats now shows a result for every data row and benchmark iteration executed, allowing post-processors to correctly calculate the total number of tests executed. Previously, individual rows were not shown in the test output if they passed, making such calculations impossible. The only change to the xunitxml output format is to correct a bug where no test result was recorded for a test function if the last data row was skipped and all other rows passed -- in which case the overall result should be a pass. Note that there is also a pre-existing bug in the xunit logger, where no result is reported if all rows are skipped; that bug is unaffected by this commit. Task-number: QTBUG-21848 Task-number: QTBUG-22124 Change-Id: I7e17177e10d6e89e55b9684c159bd506f21d002b Reviewed-by: Ed Baak <ed.baak@nokia.com>
* testlib: Clear ignored messages after every data rowJason McDonald2012-02-081-4/+5
| | | | | | | | | | | | | | | | | | | | Previously, ignored messages were only cleared at the end of each test function, i.e. after all data rows were finished. This meant that if a data row in a data-driven test function didn't cause all of the expected messages to be generated, the remaining messages would be carried over to the next data row. This would result in errors about missing messages being associated with the last data row rather than with the correct data row. This commit makes testlib check for missing ignored messages after running each data row rather than only doing so after the last data row. This commit also adds a regression test to demonstrate that ignored messages can no longer be carried over from one data row to another. Change-Id: Ibee51aa6e96866fbcbcb4acee1a8340a86a6a4ba Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* testlib: Clear expected failures after every data row.Jason McDonald2012-02-081-2/+7
| | | | | | | | | | | | | | | | | | | | | | Previously, expected failures were only cleared at the end of each test function, i.e. after all data rows were finished. This meant that if a data-driven test function called QEXPECT_FAIL and then didn't perform any further verification steps to trigger the expected failure, the expected failure would be carried over to the next data row, probably causing the first verification step in the test function to XPASS (with a seemingly irrelevant error message) for the next data row. This commit adds the new function QTestResult::finishedCurrentTestData() to cleanup after each data row is executed. This function treats calls to QEXPECT_FAIL without subsequent verification steps as a test failure. This commit also adds a regression test to demonstrate that expected failures can no longer be carried over from one data row to another. If run against the previous version of testlib, the new test would report a pass instead of an error. Change-Id: Ida5c7f080815b0dca9531131fed582b0918334cb Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* CodeCoverage: Save coverage data in QTestLog::stopLogging()Caroline Chao2012-02-071-0/+12
| | | | | | | | | | | | | | | | | Currently when tests are crashing, aborting or hanging, their status are reported with a status "Unknown" because SaveCoverageTool is never called for them. The status of the test given by the coverage tool should be the same as the one reported in the log output when the test is run. This change will allow more accuracy in the status reported. Add QTestResult::setCurrentAppname and QTestResult::currentAppname. To retrieve the name of the current application running. Task-number: QTQAINFRA-460 Change-Id: Icc476dc2d6cb28185e5447f1e79da6a8a31cad54 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com> Reviewed-by: Jason McDonald <jason.mcdonald@nokia.com>
* Remove pass/fail/skip count getters from QTestResult.Jason McDonald2012-01-301-15/+0
| | | | | | | | These functions are no longer used, as the counters have been moved to QTestLog. Change-Id: I7cd13fc92a18920706fcc712804c304f9c333969 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Remove "All rights reserved" line from license headers.Jason McDonald2012-01-301-1/+1
| | | | | | | | | | As in the past, to avoid rewriting various autotests that contain line-number information, an extra blank line has been inserted at the end of the license text to ensure that this commit does not change the total number of lines in the license header. Change-Id: I311e001373776812699d6efc045b5f742890c689 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Eliminate QTestResult::ignoreMessage().Jason McDonald2012-01-301-5/+0
| | | | | | | | | | The removed function was just a wrapper around QTestLog::ignoreMessage() and was only called twice (once in qtestcase.cpp and once in qtdeclarative). Better to just call the desired function directly and avoid the indirection. Change-Id: Ib900f35f5d584e8f323cef9b0e7b4f8a507c199e Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Update contact information in license headers.Jason McDonald2012-01-231-1/+1
| | | | | | | Replace Nokia contact email address with Qt Project website. Change-Id: I431bbbf76d7c27d8b502f87947675c116994c415 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Move pass/fail/skip counters from QTestResult to QTestLog.Jason McDonald2012-01-171-15/+5
| | | | | | | | | | | | | | | This change will enable further simplification of QTestResult and QTestLog in the future, including removing a circular dependency between the two classes. The "getter" functions in QTestResult are retained for now, but will be removed in a future commit, once QtQuickTest has been changed to call the getters that this commit adds to QTestLog. This commit is adapted from an unfinished change by Henrik Hartz. Change-Id: Ife7f80ac6a4310449a4712e96e0bea6c02139a5a Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Remove unused QTestResult::allDataPassed().Jason McDonald2012-01-161-5/+0
| | | | | | | The removed function was private API and was not used anywhere in Qt. Change-Id: Ia262d119ebba30f6968cb7145043475bf88ffa39 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Rename QTestLog::addIgnoreMessage to QTestLog::ignoreMessage.Jason McDonald2012-01-161-1/+1
| | | | | Change-Id: I4cfcd00d444d0812c58afab6ffd43dee37db1340 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Update copyright year in license headers.Jason McDonald2012-01-051-1/+1
| | | | | Change-Id: I02f2c620296fcd91d4967d58767ea33fc4e1e7dc Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Stop using QTest::qt_snprintf() in testlib.Jason McDonald2011-12-121-5/+5
| | | | | | | | After the previous commit, QTest::qt_snprintf() is equivalent to qsnprintf(), so just use that instead. Change-Id: I89ad6e3749ba5efb1926b0b618a904e8baca9f52 Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Make testlib use #include consistently.Jason McDonald2011-10-211-4/+4
| | | | | Change-Id: I36b5da3f832d2588072405d93143173edc29f4af Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Remove mode parameter from QTestLog::addSkip().Jason McDonald2011-08-291-3/+2
| | | | | | | | | | The mode parameter is never used by the logging code, so there's little value in passing it there. Change-Id: Ibe2cbe5eaf457a7e3ffd3aea3a4be7c8278c91b6 Reviewed-on: http://codereview.qt.nokia.com/3547 Reviewed-by: Qt Sanity Bot <qt_sanity_bot@ovi.com> Reviewed-by: Rohan McGovern <rohan.mcgovern@nokia.com>
* Update licenseheader text in source files for qtbase Qt moduleJyri Tahtela2011-05-241-17/+17
| | | | | | | Updated version of LGPL and FDL licenseheaders. Apply release phase licenseheaders for all source files. Reviewed-by: Trust Me
* Initial import from the monolithic Qt.Qt by Nokia2011-04-271-0/+352
This is the beginning of revision history for this module. If you want to look at revision history older than this, please refer to the Qt Git wiki for how to use Git history grafting. At the time of writing, this wiki is located here: http://qt.gitorious.org/qt/pages/GitIntroductionWithQt If you have already performed the grafting and you don't see any history beyond this commit, try running "git log" with the "--follow" argument. Branched from the monolithic repo, Qt master branch, at commit 896db169ea224deb96c59ce8af800d019de63f12