summaryrefslogtreecommitdiffstats
path: root/src/testlib/qjunittestlogger.cpp
Commit message (Collapse)AuthorAgeFilesLines
* Remove spurious class prefixes from subclass codeEdward Welbourne2021-12-091-5/+5
| | | | | | | | | The subclass's code can refer to the base's public enums without needing the (long) class prefix. Change-Id: I4824440d216695aee9345776faba2e510130425e Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* Make Skip an incident in test loggingEdward Welbourne2021-12-091-7/+6
| | | | | | | | | | | | | | | Skip ends the test (albeit inconclusively). Rearrange the enums in the abstract logger, move code to handle skip between relevant function and tidy up various things that became simpler as a result. Also reorder the message enum, and its switches, to separate testlib's internals from the usual Qt messages, and put each group in ascending order of severity. Task-number: QTBUG-96844 Change-Id: I2c7a634b9f849830d64eafa750155e66e244b729 Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io> Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
* testlib: Gracefully handle messages outside of test function in JUnit reporterTor Arne Vestbø2021-09-161-0/+3
| | | | | | | | Fixes: QTBUG-96543 Task-number: COIN-755 Pick-to: 6.2 Change-Id: I2334597319f3595f37f48c9811f667e57b99df28 Reviewed-by: Fabian Kosmale <fabian.kosmale@qt.io>
* testlib: Replace custom QTestCoreList with std::vectorTor Arne Vestbø2021-09-101-23/+17
| | | | | | | | | | | | | | | | | The custom linked list implementation was implemented using recursion, and as a result didn't handle long lists of test cases, exhausting the stack on e.g. Windows where the default stack is only 1MB. This was the case with e.g. the tst_QChar test that produces 20K test cases. Replacing with a std::vector should do nicely for our use-case. No attempt has been made at further reducing the complexity of QTestElement/QTestCoreElement/QTestElementAttribute. Pick-to: 6.2 Change-Id: Ie295f7cf937ec6abdc4606b6120818551ad285c7 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Don't write redundant log message information in JUnit reporterTor Arne Vestbø2021-08-101-55/+22
| | | | | | | | | | | | | The separation between <system-out> and <system-err> is sufficient, and we can't expect consumers to interpret our custom comment format. The type of the text node has been renamed to more accurately identify its purpose. Pick-to: 6.2 Change-Id: I63c8ff17529fc087e1b695698350a6711eb5e68d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Treat qCritical messages as system-err in JUnit reporterTor Arne Vestbø2021-08-101-0/+1
| | | | | | Pick-to: 6.2 Change-Id: Ia34fbc59dc430b7ef39a94bd6836d9675cde58f5 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Rename QAbstractTestLogger::QSystem to QCriticalTor Arne Vestbø2021-08-101-2/+2
| | | | | | | | | | | | | It's used to handle qCritical() messages, and corresponds to QtMsgType's QtCriticalMsg, so change its name to reflect what it really is. In the process, reclassify the -maxwarnings overflow message as a warning, rather than a critical message. Pick-to: 6.2 Change-Id: I87626117a547ae4498d5dc352b93bd6db8bfb332 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Move stdout/stderr reporting into individual <testcase> elementsTor Arne Vestbø2021-08-101-7/+17
| | | | | | | | | | | | | | | The original Ant JUnit reporter only writes <system-err> and <system-out> to the <testsuite>, but more modern reporters such as Maven Surefire scopes output to each individual <testcase>. This is also handled by both the Jenkins JUnit and xUnit plugins, e.g.: https://github.com/jenkinsci/junit-plugin/commit/145eb5c98 Pick-to: 6.2 Change-Id: I20c87276004a4e0910fc18e05e6ffa0f5e5a7b7c Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Produce <error> elements on fatal errors in JUnit reporterTor Arne Vestbø2021-08-101-53/+50
| | | | | | | | | Test errors represents unanticipated problems, e.g., an unhandled exception, or a problem with the implementation of the test. Pick-to: 6.2 Change-Id: I87219e7ffdea56862278f005de44526ad97545f0 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Don't report blacklisted tests as failures in JUnit reporterTor Arne Vestbø2021-08-061-2/+0
| | | | | | | | | | | | | We don't produce <failure> elements for them, so we shouldn't include them in the total count of failures. In the future we might produce <skipped> elements instead, but for now remove the incorrect increment. Pick-to: 6.2 Change-Id: Ia89a5cdaf79f6aa2f92be53180112c40ad6da2c6 Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Fix name of current test case element in JUnit reporterTor Arne Vestbø2021-08-061-9/+9
| | | | | | | Pick-to: 6.2 Change-Id: I0bd77e5f1d0540b87568a0c69a4b96442d4000eb Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Write failure details as <failure> content in JUnit reporterTor Arne Vestbø2021-08-061-1/+14
| | | | | | | Pick-to: 6.2 Change-Id: Ica48769e7dfcabdc4bc8f0ed058bc22e29a0b632 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io> Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
* testlib: Report skipped tests in JUnit reporter as <skipped> elementsTor Arne Vestbø2021-08-051-1/+11
| | | | | | | | | | | | | | | | | The Apache Ant and Surefire Maven specs document a <skipped> element that can be used to signify skipped test, with a corresponding total skipped test attribute on the <testsuite>. The element includes an optional message attribute, documented in the Surefire spec, and in the Ant source code, but not yet documented in the reverse-engineered Ant spec: https://github.com/windyroad/JUnit-Schema/pull/11 Pick-to: 6.2 Task-number: QTBUG-95424 Change-Id: Ib6417a41b9c328836f4017e6ebf7f7e9cd91288d Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Don't report JUnit errors attribute without any <error> elementsTor Arne Vestbø2021-08-051-1/+0
| | | | | | | | | | | The errors attribute on the <testsuite> element represents the number of <error> elements, but we do not produce any at the moment. Pick-to: 6.2 Task-number: QTBUG-95424 Change-Id: I7196d622a9a6bbb7e79ed2c2886984d539abb1da Reviewed-by: Edward Welbourne <edward.welbourne@qt.io> Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Reduce JUnit test duration reporting to millisecond precisionTor Arne Vestbø2021-08-051-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The original Ant JUnit reporter produced test durations via Double.toString(), supporting arbitrary precisions, and the de-facto schema declared them as xs:decimal. Sadly, the now popular Maven Surefire reporter limited the duration to millisecond precision, and hard-coded this into its schema as SUREFIRE_TIME: https://issues.apache.org/jira/browse/SUREFIRE-1533 Unfortunately this definition spread into tools such as the Jenkins xUnit plugin, which relies on the schema provided by Maven Surefire: https://issues.jenkins.io/browse/JENKINS-52152 As a result, anything that produces higher precision results will not validate in the Jenkins xUnit plugin. Other test frameworks have bitten the bullet and reduced their precision correspondingly, e.g.: https://github.com/catchorg/Catch2/issues/2221 https://github.com/catchorg/Catch2/commit/581c46249acf8389e9 We follow suit, and our JUnit XML output now validates against both the Jenkins JUnit and xUnit plugins, as well as the original Apache Ant de-facto schema. Pick-to: 6.2 Task-number: QTBUG-95424 Change-Id: I3097d10c03c2a29709960372301b29055d224e10 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Improve JUnit XML conformanceTor Arne Vestbø2021-08-051-64/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The JUnit test framework did not initially have any XML reporting facilities built in. Instead, the XML report was generated by the Apache Ant JUnit task: https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java Many users interacted with these reports via the Jenkins JUnit plugin, which provided graphical visualization of the test results: https://plugins.jenkins.io/junit/ Due to the lack of an official XML schema for the Apache Ant JUnit report there was some confusion about what the actual format was. People started documenting the de-facto format, both as produced by Ant, and as consumed by Jenkins: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd The XML produced by the Qt Test JUnit reporter was far from these schemas, causing issues when importing results into tools such as Jenkins, Allure2, or Test Center. The following changes have been made to improve conformance: - The 'timestamp' attribute on <testsuite> is is now in ISO 8601 local time, without any time zone specified - The 'hostname' attribute on <testsuite> is now included - The 'classname' attribute on <testcase> is now included - The non-standard 'result' attribute on <testcase> has been removed - The non-standard 'result' attribute on <failure> has been renamed to 'type' - The <system-out> element on <testsuite> is always included, even when empty - The non-standard 'tag' attribute on <failure> has been removed. Data-driven tests are now represented as individual <testcase> elements, e.g.: <testcase name="someTest(someData X)" ...> <testcase name="someTest(someData Y)" ...> <testcase name="someTest(someData Z)" ...> The resulting XML validates against both the de-facto Apache Ant 'JUnit 4' schema and the Jenkins JUnit plugin schema. Task-number: QTBUG-95424 Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* testlib: Simplify JUnit test loggerTor Arne Vestbø2021-08-051-45/+6
| | | | | | | | | | | | | - Use the right name for the attribute (AI_Message), rather than fixing it up in QTestJUnitStreamer. - Don't pretend that we're adding line and file information, only to discard it in QTestJUnitStreamer. - Don't pretend to add benchmark information, only to discard it in QTestJUnitStreamer. Pick-to: 6.2 Change-Id: Ib6eadc12300157216fe9c6e8bcfebd7eb8a3ea68 Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
* QTestlib/JUnit XML: Log most messages to standard output instead of error outputFriedemann Kleint2021-01-151-16/+24
| | | | | | | | | | | | | | | Add an enumeration for system-out and alog element for it. Redirect the messages types that are not warnings/errors to this element. For compatibility, write it out only if it is not empty. Rename enumerations and members accordingly. [ChangeLog][QtTestLib] In JUnit XML, output that is not a warning/error is now logged under <system-out> instead of <system-err>. Fixes: QTBUG-86540 Change-Id: I55598eafa7dafa486ac5a8221029c332ff47413b Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
* testlib: Add start time and test duration to JUnit XMLTor Arne Vestbø2020-07-241-0/+10
| | | | | | | As defined by https://llg.cubic.org/docs/junit/ Change-Id: Ic7683f3d49c529674f8467d591528d4a65d3add8 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
* testlib: Track current test suite in JUnit test loggerTor Arne Vestbø2020-07-221-27/+31
| | | | | | | | | Instead of deferring the creation of the test suite until logging stops, we create it up front, matching the logic of adding test elements on test function enter. Change-Id: I78b1ccdfde5493d78ef478d4b3c45d5a49358979 Reviewed-by: Volker Hilsheimer <volker.hilsheimer@qt.io>
* testlib: Clarify that our XUnit reporter is actually a JUnit reporterTor Arne Vestbø2020-02-031-0/+352
The reporter was probably named 'xunit' based on the historical use of xUnit to refer to testing frameworks derived from Smalltalk's SUnit. These frameworks typically added their own prefix, e.g. JUnit for Java, RUnit for R, etc. The most popular of these was the JUnit framework, and the corresponding XML output produced by the Ant built tool became somewhat of a de facto standard, which is probably why we chose to model our reporter after it. Nowadays however, naming it 'xunit' is problematic as there is actually a testing famework named xUnit.net, typically shortened to, you guessed it: xunit. Test report consumers will typically have a junit mode, and an xunit mode, and the latter could easily be mistaken for what testlib outputs, unless we clarify this. The clarification also allows us to safely extend our support for the JUnit XML format to incorporate some elements that are nowadays common, but where we are lagging behind the standard. [ChangeLog][QTestLib] The formerly named 'xunitxml' test reporter has been renamed to what it actually is: a JUnit test reporter, and is now triggered by passing -o junitxml to the test binary. Change-Id: Ieb20d3d2b5905c74e55b98174948cc70870c0ef9 Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io> Reviewed-by: Simon Hausmann <simon.hausmann@qt.io>
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-04 22:17:04 +0000