support pyside CI testing with filtering

PySide is currently in an unfinished state. Therefore, running all the
tests will never return a success.

To make this more useful, this script is controlled by a blacklist,
in order to keep record of the known-to-fail tests.

On success: zero error status
On Failure: nonzero (raises ValueError)

The test parser uses a blacklist file that is compatible to standard
Qt blacklist files.

macOS, Ubuntu and windows (thanks to fkleint) are supported.

The blacklist has a feature where new configurations are learned.
It first removes old versions of the same tests, before it inserts the new ones.

Blacklisting is now optional.
We support now py2, py3, qt5, qt5.6 etc.

Some consideration about collapsing the data are added.
The parser has now a sub-command "test" that is mandatory.

Support for moving the build_dir to another location without loosing the tests so far.

Implement BPASS.
Support CTEST_OUTPUT_ON_FAILURE

Change-Id: If66d3cf1733a3b0c8a47fe9b8c6aec8ea430c699
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>

1 files changed, 156 insertions, 0 deletions

diff --git a/build_history/blacklist.txt b/build_history/blacklist.txt
new file mode 100644
index 000000000..f7548e3d9
--- /dev/null
+++ b/build_history/blacklist.txt
@@ -0,0 +1,156 @@
+[pysidetest::modelview_test]
+    darwin py3
+[pysidetest::qvariant_test]
+    darwin py3
+[pysidetest::utils_test]
+    win32
+[signals::disconnect_test]
+    linux2
+    darwin py2
+    win32
+[signals::multiple_connections_test]
+    linux2
+    darwin
+    win32
+[signals::ref06_test]
+    linux2
+    darwin
+    win32
+[signals::short_circuit_test]
+    linux2
+    darwin
+    win32
+[signals::signal_signature_test]
+    linux2
+    darwin
+    win32
+[signals::static_metaobject_test]
+    linux2
+    darwin
+    win32
+[QtCore::bug_829]
+    win32
+[QtCore::bug_1063]
+    linux2
+    darwin
+    win32
+[QtCore::deepcopy_test]
+    linux2
+    darwin
+    win32
+[QtCore::qfile_test]
+    win32
+    darwin py3
+[QtCore::qflags_test]
+    win32
+    darwin py3
+[QtCore::qobject_connect_notify_test]
+    linux2
+    darwin
+    win32
+[QtCore::qthread_test]
+    linux2
+[QtCore::qtextstream_test]
+    linux2
+    darwin
+    win32
+[QtCore::repr_test]
+    linux2
+    darwin
+    win32
+[QtGui::qmatrix_test]
+    win32
+[QtWidgets::bug_576]
+    linux2
+    win32
+    darwin py3
+[QtWidgets::bug_722]
+    linux2
+    darwin
+    win32
+[QtWidgets::bug_919]
+    linux2
+[QtWidgets::qstandarditemmodel_test]
+    linux2
+    darwin
+    win32
+[QtWidgets::returnquadruplesofnumbers_test]
+    linux2
+    darwin
+    win32
+[QtTest::touchevent_test]
+    linux2
+    darwin
+    win32
+[QtMultimedia::audio_test]
+    linux2
+    darwin
+    win32
+[QtQml::bug_451]
+    linux2
+    darwin
+    win32
+[QtQml::bug_456]
+    linux2
+    darwin
+    win32
+[QtQml::bug_557]
+    linux2
+    darwin
+    win32
+[QtQml::bug_726]
+    linux2
+    darwin
+    win32
+[QtQml::bug_814]
+    linux2
+    darwin
+    win32
+[QtQml::bug_825]
+    linux2
+    darwin
+    win32
+[QtQml::bug_847]
+    linux2
+    darwin
+    win32
+[QtQml::bug_915]
+    linux2
+    darwin
+    win32
+[QtQml::bug_926]
+    linux2
+    darwin
+    win32
+[QtQml::bug_951]
+    linux2
+    darwin
+    win32
+[QtQml::bug_995]
+    linux2
+    darwin
+    win32
+[QtQml::bug_997]
+    linux2
+    darwin
+    win32
+[QtQml::qqmlnetwork_test]
+    linux2
+    darwin
+    win32
+[QtQml::qquickview_test]
+    linux2
+    darwin
+    win32
+[QtQml::registertype]
+    linux2
+    darwin
+    win32
+[QtScript::qscriptvalue_test]
+    linux2
+    darwin
+    win32
+[QtScriptTools::debugger_test]
+    linux2
+    darwin
+    win32