summaryrefslogtreecommitdiffstats
path: root/src/testlib/qtaptestlogger.cpp
Commit message (Collapse)AuthorAgeFilesLines
* Fix error in TAP test logger's regex to recognize QVERIFY() outputEdward Welbourne2022-09-221-1/+1
| | | | | | | | | | | The regex used a greedy .+ to match an actual dot and a space, preceding an open parenthesis, with the result that if the message contained any parentheses that .+ swallowed everything up to the last of them. Correct the regex so that recently-added tests' error messages show up correctly. Change-Id: I6e52c9b2a6e7959335fcddbb4266f65b589eba68 Reviewed-by: Jason McDonald <macadder1@gmail.com>
* Testlib: teach TAP test logger to support new QCOMPARE_* operatorsIvan Solovev2022-06-031-10/+67
| | | | | | | | | | | | | | | | | | | | | | The TAP test logger will now correctly print comparison types such as QCOMPARE_NE or QCOMPARE_LT, and also provide a proper expected/wanted value (by adding proper arithmetical operators in front of the value). Sample output: type: QCOMPARE_GE message: Left value is expected to be greater than or equal to right value, but is not wanted: >= 1 (rhs) found: 0 (lhs) expected: >= 1 (rhs) actual: 0 (lhs) at: tst_ExtendedCompare::compareUnregistereEnum() (tst_extendedcompare.cpp:232) file: tst_extendedcompare.cpp line: 232 As a drive-by: make some variables const. Task-number: QTBUG-98873 Change-Id: Idb54eaabcb937b42d3fc844f30041aab82d73f69 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Use SPDX license identifiersLucie Gérard2022-05-161-38/+2
| | | | | | | | | | | | | Replace the current license disclaimer in files by a SPDX-License-Identifier. Files that have to be modified by hand are modified. License files are organized under LICENSES directory. Task-number: QTBUG-67283 Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1 Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org> Reviewed-by: Lars Knoll <lars.knoll@qt.io> Reviewed-by: Jörg Bornemann <joerg.bornemann@qt.io>
* QtTestLib: replace remaining uses of QL1String with QL1StringViewSona Kurazyan2022-05-021-1/+1
| | | | | | | | | Remove unneeded \fn qdoc lines as a drive-by. Task-number: QTBUG-98434 Change-Id: Id93ddbb38b97a8f5a6734bfbc82686ccb3a87aa6 Reviewed-by: Marc Mutz <marc.mutz@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* QtTestLib: replace QLatin1String uses with _L1/_s/QStringLiteralSona Kurazyan2022-05-021-19/+19
| | | | | | | Task-number: QTBUG-98434 Change-Id: Ie327fd4af1880002e5a1e09b43384f2b709625e7 Reviewed-by: Marc Mutz <marc.mutz@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* QtTestLib: stop using QLatin1Char constructor for creating char literalsSona Kurazyan2022-04-281-1/+1
| | | | | | | | Required for porting away from QLatin1Char/QLatin1String in scope of QTBUG-98434. Change-Id: I3debae5f481037958bfb65caabca97a0d4681eb9 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* Tidy up QTapTestLogger::addIncident()'s regex parsingEdward Welbourne2022-03-291-13/+14
| | | | | | | | | | It had some repetition that could be refactored out, some long lines and a bool it worked out the hard way, when it had previously determined the answer in passing without recording it. Change-Id: I9e53ed087dfbe8067686b27b6cf9ac32040fbf19 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
* TAP test logger: report B?XFAIL (mostly) as a messageEdward Welbourne2022-03-291-102/+120
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, only the first B?XFAIL would be reported, all others would be discarded. Furthermore, if a B?XFAIL had happened, B?PASS was also not reported, since the B?XFAIL served as test line. However, if the B?XFAIL was followed by a SKIP, B?XPASS or B?FAIL, these were reported as normal, producing exactly the kind of duplicated test line that the skipping of B?PASS was meant to supply. So change B?XFAIL to be reported among the messages, but retain the TODO annotation of the first on the test line of a subsequent B?PASS, if nothing more drastic happens in the mean time. So now more than one B?XFAIL can be reported, the test is still marked as a TODO and we don't get duplicate test lines for a subsequent non-passing result. This replaces the bool m_wasExpectedFail member with a QTestCharBuffer m_firstExpectedFail that records the first XFAIL's TODO line (so its isEmpty() fully replaces m_wasExpectedFail). Previously, the at/file/line information for a B?XFail would be supplied as top-level keys in the YAML block for a "Pass" reported as not ok due to the XFail, as this location information is now part of the B?XFail's message in the extensions/messages block. Duplicating the first B?XFail's location at top level would add complexity and is arguably misleading, as the test result is really a pass (after ignoring known issues), and the location of the pass is indeterminate (nominally the end of the test function, but actually also after the cleanup() call for this test, when relevant), which is why a Pass has no location information. Task-number: QTBUG-96844 Change-Id: Ib3f24f56266ff49bf3bc9759ac8263ac69a62130 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Move TAP's messages block to after the primary report detailsEdward Welbourne2022-03-291-14/+24
| | | | | | | | | In the process, split it from its comments block and don't bother with a YAML block if it would only have contained comments. Task-number: QTBUG-96844 Change-Id: I08c20f796252bb270ba9caa4c055cdcc0843a88b Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Include file and line information in TAP diagnostics blocksEdward Welbourne2022-03-291-1/+1
| | | | | | | | | | | When producing a diagnostics block, include the file and line information, if we have it, to describe it. This presently only adds this information for skip, but could in principle do the same for a B?XPass. Task-number: QTBUG-96844 Change-Id: I6cc375d98e2369eba262010f9c2dfbcba931a6f1 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* TAP test logger: move messages into the diagnostics blockEdward Welbourne2022-03-291-23/+97
| | | | | | | | | | | | | | | | | | | | | | | | | | | Our TAP output was delivering messages as comments before the test line, where TAP clearly expects the details of a test to follow its test line. Version 13 provides a YAML block to deliver diagnostics and encourages use of this, so accumulate our messages in a QTestCharBuffer instead of emitting them one by one. However, messages produced after a test has produced its test line belong to that test, but are too late to be included in its diagnostics block, so should be emitted immediately as before, albeit now with a type prefix. This at least separates such messages, from the end of one test, from messages produced early in the next. In the process, add a type-prefix to each, to make clear what type of message it was. Since the Yamlish supported by TAP consumers doesn't support a way to have many messages, use the extensions: top-level hash tag with a messages: sub-tag to gather our messages as a list. (This expands at least one expected output file significantly and substantially rewrites some others.) Add methods to QTestCharBuffer, and a helper function, to support this. Task-number: QTBUG-96844 Change-Id: If44a33da5879ed1670ef0980042599afd516f9d2 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Document some details of TAP's use of YAMLishEdward Welbourne2022-03-221-1/+67
| | | | | | | ... along with how QtTest soon shall be using it. Change-Id: I9c723ebdb8ee5be1c4152aa911ae7c846ea61b5c Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Use a cheaper is-empty test for a C-stringEdward Welbourne2022-03-221-1/+1
| | | | | | | | We only need to look at the first byte to determine whether the length is > 0; no need to count how many more bytes after it aren't '\0'. Change-Id: Id2db2a9a993086c0aee02c99396fcb346b1a348e Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* Avoid duplicate descriptions in TAP's incident YAMLEdward Welbourne2022-03-221-2/+2
| | | | | | | | | | | | | | When a B?Fail's description doesn't match the QVERIFY/QCOMPARE regexes, it got output as a comment in the YAML block after the test line had already reported it as a TODO comment. An empty description would also have lead to an empty comment in the YAML block. Condition this fallback output case on there being a description that hasn't yet been reported in the test line. Task-number: QTBUG-96844 Change-Id: Id7fe81d26ddb01da3d8003ada8fa590a5e1a166f Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Fix indentation of some continuation lines in qtaptestlogger{.cpp,_p.h}Edward Welbourne2022-03-221-2/+2
| | | | | Change-Id: I33dc05b61b07f077edc7c4816d43d0b53f5f4b43 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* TAP test logger: treat blacklisted XFail the same as XFailEdward Welbourne2022-03-221-1/+1
| | | | | | | | | | | In all places but one they were treated the same; fix that last one to match the rest. This removes one line from the YAML block for each blacklisted XFail test; in each case, this message duplicates the one on its "not ok ... # TODO..." test line. Change-Id: Iff2028afccd979db7f2c84e85d1b78541900008e Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* Document what QtTest's various loggers doEdward Welbourne2022-03-221-2/+12
| | | | | | | | | Provide basic (internal) documentation of each logging class and link the command-line documentation to pages relevant to the formats not defined by Qt. Change-Id: I3251dd1304203c6ab87dfe1f2dec0e9787ab69f8 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Tidy up in QTapTestLogger::addIncident()Edward Welbourne2021-12-091-9/+7
| | | | | | | | | | After we've checked m_wasExpectedFail, we might as well set it promptly, since one of our earlier checks is on the same condition. Explain what that check is about, fix a typo in another comment and make the local ok variable const, since it can be. Change-Id: Ie04c0ad4248e47b116d3e22046116d39cf2ac6df Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* TAP test logger: avoid dangling space on TODO and SKIP linesEdward Welbourne2021-12-091-6/+20
| | | | | | | | | If an incident has an empty description we produced output with a dangling space. Avoid triggering the inanity 'bot with those. Reorganise the code to save some duplication. Change-Id: I1dc29fa8ad4449a4584f11dddcf002e405cd9238 Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* Avoid duplicated code by reworking #if-ery around an if/elseEdward Welbourne2021-12-091-9/+4
| | | | | Change-Id: I08376d2dfd071989fab9ee18dbb90ab8670ed20f Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* TAP test logger: skip XFail results as well as pass after an XFailEdward Welbourne2021-12-091-1/+2
| | | | | | | | | | The rationale for skipping pass after XFail is "to emit a single test point for" the test; emitting several XFails violates that aim. Task-number: QTBUG-96844 Change-Id: Ia8626dfc2dded234b3aa530fc2dc2324f1e28400 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* Make Skip an incident in test loggingEdward Welbourne2021-12-091-9/+6
| | | | | | | | | | | | | | | Skip ends the test (albeit inconclusively). Rearrange the enums in the abstract logger, move code to handle skip between relevant function and tidy up various things that became simpler as a result. Also reorder the message enum, and its switches, to separate testlib's internals from the usual Qt messages, and put each group in ascending order of severity. Task-number: QTBUG-96844 Change-Id: I2c7a634b9f849830d64eafa750155e66e244b729 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* testlib: Don't report additional 'pass' test point for blacklisted XFailTor Arne Vestbø2021-08-051-3/+3
| | | | | | | | | | Regression after 9906cc57ed3eed64d534f43c677bb16e08561bb6. Pick-to: 6.2 Change-Id: I5566f70c66d248426c7a41b6de1cfb92f104cc64 Reviewed-by: Oliver Wolff <oliver.wolff@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* QTestLib: Fix class declarations/structureFriedemann Kleint2019-07-111-3/+1
| | | | | | | | | | | | - Remove virtual from functions declared as override - Use " = default" for trivial constructors/destructors - Remove all special functions from QTestLog Apply Fixits by Qt Creator with some amendments. Task-number: QTBUG-69413 Change-Id: I812b8116e5b4c927e4e5cee44e63bc705385d866 Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
* Remove QLatin1Literal usagesGiuseppe D'Angelo2019-06-101-12/+12
| | | | | | | | | | | | | That's an undocumented Qt 4/3/2 remnant, start remove usages. Fix incorrect include header in qclass_lib_map.h as a drive-by. Change-Id: I939be2621bc03e5c75f7e3f152546d3af6d37b91 Reviewed-by: Marc Mutz <marc.mutz@kdab.com> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* testlib: Add BXPASS and BXFAILOliver Wolff2019-01-291-2/+4
| | | | | | | | | | | | | | | Prioritize blacklisting over QEXPECT_FAIL so that a test that is blacklisted no longer fails if QEXPECT_FAIL returns true unexpectedly. To reflect this state properly, the two values of BXPASS and BXFAIL were added to testlib's output. [ChangeLog][Important Behavior Changes][QtTestLib] Blacklisting of tests will be taken into account for XPASS and XFAIL. A blacklisted test that causes an XPASS will no longer be a fail. Task-number: QTBUG-72928 Change-Id: Ia2232fdc714d405fa3fd9aea6c89eb2836bc5950 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* qtlite: Fix build libs with -no-feature-regularexpressionMikhail Svetkin2019-01-271-1/+10
| | | | | Change-Id: I427ff1f8f4986fbf466aba60a9d3de614c1e006f Reviewed-by: Lars Knoll <lars.knoll@qt.io>
* testlib: Add Test Anything Protocol (TAP) reporterTor Arne Vestbø2018-03-141-0/+254
The Test Anything Protocol (TAP), was originally Perl's simple text-based interface between testing modules and test harnesses, but has since been adopted by a large number of producers and consumers in many different languages, which allows colorizing and summarizing test results. The format is very simple: TAP version 13 ok 1 - test description not ok 2 - test description --- message: 'Failure message' severity: fail expected: 123 actual: 456 ... ok 3 - test description # SKIP 1..3 The specification [1] is very brief, so the implementation has been based on how typical consumers behave, especially when it comes to the undefined diagnostics block. [1] http://testanything.org/tap-version-13-specification.html Change-Id: I616e802ea380165c678510e940ddc6607d39c92d Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
generated by cgit v1.2.3 (git 2.25.1) at 2024-06-08 12:01:18 +0000