Run ES5 tests on Linux/x86-64

This requires including the test suite as a submodule by default and
bumping the module to a new sha1 that removes some particularly long
paths.

Also force the test suite to run under a US locale, as tests like
ch15/15.5/15.5.4/15.5.4.7/S15.5.4.7_A1_T11 cannot deal with locale
dependent output for date formatting.

The test harness now returns a non-zero exit code when a test fails.
This is used by the QTestLib wrapper to determine success or failure.

The tests with JIT are run, but the tests with the interpreter are
omitted at the moment until the last failure is fixed.

The tests add about 10-15 minutes extra time to the total time it takes
to run tests on Linux in the CI.

Change-Id: Id01fd3b41350f9c9a6ce9e43236f51f7f0fb71c8
Reviewed-by: Robin Burchell <robin.burchell@crimson.no>

1 files changed, 31 insertions, 0 deletions

diff --git a/tests/auto/qml/ecmascripttests/TestExpectations b/tests/auto/qml/ecmascripttests/TestExpectations
new file mode 100644
index 0000000000..49f107452a
--- /dev/null
+++ b/tests/auto/qml/ecmascripttests/TestExpectations
@@ -0,0 +1,31 @@
+# wrong tests
+# uses octal number
+15.2.3.6-2-17-1 failing
+
+# these fail after the update to Unicode 6.3.
+# the reason is that u+180e changed type from whitespace to control
+S9.3.1_A2
+S9.3.1_A3_T1
+S9.3.1_A3_T2
+S15.1.2.2_A2_T10
+S15.1.2.3_A2_T10
+15.5.4.20-3-2
+15.5.4.20-3-3
+15.5.4.20-3-4
+15.5.4.20-3-5
+15.5.4.20-3-6
+
+10.4.3-1-106 failing
+11.2.3-3_3 failing
+S13_A15_T4 failing
+S15.4.4.3_A1_T1 failing
+S15.4.4.3_A3_T1 failing
+S15.5.4.11_A5_T1 failing
+S15.2.4.4_A14 failing
+
+# Function declarations in conditionals. We allow them, because the real
+# world requires them.
+Sbp_12.5_A9_T3 failing
+Sbp_12.6.1_A13_T3 failing
+Sbp_12.6.2_A13_T3 failing
+Sbp_12.6.4_A13_T3 failing