From 973e74399e807fb6f351a7d330c8c8b85b66bc5a Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Tor=20Arne=20Vestb=C3=B8?= <tor.arne.vestbo@qt.io>
Date: Thu, 29 Jul 2021 09:59:59 +0200
Subject: testlib: Improve JUnit XML conformance
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The JUnit test framework did not initially have any XML reporting
facilities built in. Instead, the XML report was generated by the
Apache Ant JUnit task:

 https://github.com/apache/ant/search?q=filename%3AXMLJUnitResultFormatter.java

Many users interacted with these reports via the Jenkins JUnit plugin,
which provided graphical visualization of the test results:

 https://plugins.jenkins.io/junit/

Due to the lack of an official XML schema for the Apache Ant JUnit
report there was some confusion about what the actual format was.
People started documenting the de-facto format, both as produced
by Ant, and as consumed by Jenkins:

 https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd
 https://github.com/junit-team/junit5/search?q=filename%3Ajenkins-junit.xsd

The XML produced by the Qt Test JUnit reporter was far from these
schemas, causing issues when importing results into tools such
as Jenkins, Allure2, or Test Center.

The following changes have been made to improve conformance:

  - The 'timestamp' attribute on <testsuite> is is now in ISO
    8601 local time, without any time zone specified
  - The 'hostname' attribute on <testsuite> is now included
  - The 'classname' attribute on <testcase> is now included
  - The non-standard 'result' attribute on <testcase> has
    been removed
  - The non-standard 'result' attribute on <failure> has
    been renamed to 'type'
  - The <system-out> element on <testsuite> is always included,
    even when empty
  - The non-standard 'tag' attribute on <failure> has been
    removed. Data-driven tests are now represented as individual
    <testcase> elements, e.g.:

      <testcase name="someTest(someData X)" ...>
      <testcase name="someTest(someData Y)" ...>
      <testcase name="someTest(someData Z)" ...>

The resulting XML validates against both the de-facto Apache Ant
'JUnit 4' schema and the Jenkins JUnit plugin schema.

Task-number: QTBUG-95424
Change-Id: I6fc9abedbfb319f2545b99b37d059b18c16776ff
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
---
 tests/auto/testlib/selftests/generate_expected_output.py | 1 +
 1 file changed, 1 insertion(+)

(limited to 'tests/auto/testlib/selftests/generate_expected_output.py')

diff --git a/tests/auto/testlib/selftests/generate_expected_output.py b/tests/auto/testlib/selftests/generate_expected_output.py
index fb1b7ee96c..7dd9aa4576 100755
--- a/tests/auto/testlib/selftests/generate_expected_output.py
+++ b/tests/auto/testlib/selftests/generate_expected_output.py
@@ -116,6 +116,7 @@ class Cleaner (object):
             (r'(Config: Using QtTest library).*', r'\1'), # txt
             (r'( *<QtBuild)>[^<]+</QtBuild>', r'\1/>'), # xml, lightxml
             (r'(<property name="QtBuild" value=")[^"]+"', r'\1"'), # junitxml
+            (r'(<testsuite .*? hostname=")[^"]+(".*>)', r'\1@HOSTNAME@\2'), # junit
             # Line numbers in source files:
             (r'(ASSERT: ("|&quot;).*("|&quot;) in file .*, line) \d+', r'\1 0'), # lightxml
             (r'(Loc: \[[^[\]()]+)\(\d+\)', r'\1(0)'), # txt
-- 
cgit v1.2.3