| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Added tests for repeated skips and failures (from within void lambdas,
to simulate skips and failures from within event handlers). These
exhibit yet more ways to count more than one outcome for a test. The
new QTest::failOnWarning() can also provoke more than one failure from
a single test, and several existing selftests exhibited various ways
for the Totals line's counts to add up to more than the number of
actual tests run.
Fixed counting so that only the first decisive incident is counted.
Tests can still report later failure or skipping, but only the first
is counted.
Added a currentTestState in qtestlog.cpp, by which it keeps track of
whether the test has resolved to a result, and clearCurrentTestState()
by which other code can reset that at the end of each test. This
brought to light various places where test-end clean-up was not being
handled - due to failure or skipping in a *_data() method or init, or
a skip in cleanup.
Fixes: QTBUG-95661
Change-Id: I5d24a37a53d3db225fa602649d8aad8f5ed6c1ad
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
|
| |
|
|
|
|
|
|
|
|
| |
Include counting of test types, to catch two more cases where totals
don't add up.
Task-number: QTBUG-95661
Change-Id: I9fe5424bc6652c61a065bf2889333e2ed9437c81
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
This reverts commit 904617dfb83f39a6a379635b64fea6fcd00f241a and makes
clear to future readers why that wouldn't be a sensible change.
Update the test's data, eliminating a case of duplicate counting that
was caused by the reverted commit.
Task-number: QTBUG-95661
Pick-to: 5.12 5.15 6.1 6.2
Change-Id: Ice6d3ab06ca171e0d6eb0fac757f1ab774e229f0
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
33d7f76f0e847d7e0fb00dd6056e7bba45b8b1e7 attempted to exercise more
of the blacklisting code, but in doing so didn't account for the
fact that the features it added didn't actually work.
Adding a global blacklisting of * will result in blacklisting
all tests, regardless of any inverse conditions "!*" later on,
such as for the 'pass' test. This would have been evident by
looking at the expectation files, which still showed the test
as a BPASS.
The broken feature should be fixed, but in the meantime let's
clean up the test so it doesn't misrepresent reality.
At the same time we also remove the messages() tests, which was
copied straight out of tst_silent, but doesn't serve any purpose
for the blacklist selftest.
Pick-to: 6.2
Change-Id: I20e4693300aad5e9ff5f17ad22e3dfcb5c49fd0e
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We try our best to pass on the file location of a failure, including for
fatal errors, but the reporting or logging machinery should not assume
there is one.
By passing on nullptr for the file location we allow the logging backends
to decide how to handle the situation, e.g. by not emitting extra fields
for failure location.
This effectively reverts c25687fa0b6e4be043e1f8c92c093d8b06ca06c4,
in favor of relying on the backends to cope with null filename,
which they already did.
As qFatal uses QMessageLogger, which by default disables file/line
information in release builds, we need to explicitly enable this in
our self-tests, to get uniform test results. Similarly, we disable
file/line info from testlib itself, as reporting Qt internal file
and line information for user diagnostics is less useful. The odd
one out there is qtestdata.cpp, which still ends up in test output
due to using QTEST_ASSERT instead of qFatal for its diagnostics.
Cleaning up that, and unifying how we report testlib issues to the
user, is left for another day.
Pick-to: 6.2
Change-Id: Ib9451b8eed86fe3ade4a4dcaf0037e1a3450321c
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
|
| |
|
|
|
| |
Change-Id: Ifcd1155f7627f6a9f8b379fb9e87425f9d9f2877
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Prioritize blacklisting over QEXPECT_FAIL so that a test that is
blacklisted no longer fails if QEXPECT_FAIL returns true unexpectedly. To
reflect this state properly, the two values of BXPASS and BXFAIL were
added to testlib's output.
[ChangeLog][Important Behavior Changes][QtTestLib] Blacklisting of tests
will be taken into account for XPASS and XFAIL. A blacklisted test that
causes an XPASS will no longer be a fail.
Task-number: QTBUG-72928
Change-Id: Ia2232fdc714d405fa3fd9aea6c89eb2836bc5950
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In selftests.qrc, imposed alphabetic order (on stem of name, then on
suffix, effectively treating . as sorting before any letter) while
removing old tests and adding new tests and data. Updated all non-csv
files and added many missing files. (Not clear on csv support status;
the script seems to have dropped it after 5.6, but the test still uses
it.)
Left expected_crashes* alone (no new files added, no update to old) as
I don't get results resembling those anticipated.
Omitted printdatatagswithglobaltags, printdatatags due to dangling
hspace on output lines, which upset sanity-bot. A change to the test
cpp is needed to make it viable to skip that dangling hspace.
Change-Id: Iab3fb626c44a91c249b2fb626c12c75ea0317098
Reviewed-by: Frederik Gladhorn <frederik.gladhorn@qt.io>
|
|
|
This is for example useful when looking for a possible BLACKLIST
file while doing a shadow build.
Add autotests for blacklist.
Change-Id: I41d3939d31d21d10187fefcb82604736d911b6ad
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@theqtcompany.com>
Reviewed-by: Simon Hausmann <simon.hausmann@theqtcompany.com>
|
