summaryrefslogtreecommitdiffstats
path: root/src/testlib/doc/src/qttest-best-practices.qdoc
diff options
context:
space:
mode:
Diffstat (limited to 'src/testlib/doc/src/qttest-best-practices.qdoc')
-rw-r--r--src/testlib/doc/src/qttest-best-practices.qdoc139
1 files changed, 83 insertions, 56 deletions
diff --git a/src/testlib/doc/src/qttest-best-practices.qdoc b/src/testlib/doc/src/qttest-best-practices.qdoc
index 7143e644fd..cea5b3a8de 100644
--- a/src/testlib/doc/src/qttest-best-practices.qdoc
+++ b/src/testlib/doc/src/qttest-best-practices.qdoc
@@ -1,29 +1,5 @@
-/****************************************************************************
-**
-** Copyright (C) 2019 The Qt Company Ltd.
-** Contact: https://www.qt.io/licensing/
-**
-** This file is part of the documentation of the Qt Toolkit.
-**
-** $QT_BEGIN_LICENSE:FDL$
-** Commercial License Usage
-** Licensees holding valid commercial Qt licenses may use this file in
-** accordance with the commercial license agreement provided with the
-** Software or, alternatively, in accordance with the terms contained in
-** a written agreement between you and The Qt Company. For licensing terms
-** and conditions see https://www.qt.io/terms-conditions. For further
-** information use the contact form at https://www.qt.io/contact-us.
-**
-** GNU Free Documentation License Usage
-** Alternatively, this file may be used under the terms of the GNU Free
-** Documentation License version 1.3 as published by the Free Software
-** Foundation and appearing in the file included in the packaging of
-** this file. Please review the following information to ensure
-** the GNU Free Documentation License version 1.3 requirements
-** will be met: https://www.gnu.org/licenses/fdl-1.3.html.
-** $QT_END_LICENSE$
-**
-****************************************************************************/
+// Copyright (C) 2019 The Qt Company Ltd.
+// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only
/*!
\page qttest-best-practices.qdoc
@@ -132,7 +108,11 @@
Do not simply number the test-case, or use bug-tracking identifiers. Someone
reading the test output will have no idea what the numbers or identifiers
mean. You can add a comment on the test-row that mentions the bug-tracking
- identifier, when relevant.
+ identifier, when relevant. It's best to avoid spacing characters and
+ characters that may be significant to command-line shells on which you may
+ want to run tests. This makes it easier to specify the test and tag on \l{Qt
+ Test Command Line Arguments}{the command-line} to your test program - for
+ example, to limit a test run to just one test-case.
\section2 Write Self-contained Test Functions
@@ -174,17 +154,24 @@
\section2 Use Data-driven Testing
- Data-driven tests make it easier to add new tests for boundary conditions
- found in later bug reports.
+ \l{Chapter 2: Data Driven Testing}{Data-driven tests} make it easier to add
+ new tests for boundary conditions found in later bug reports.
Using a data-driven test rather than testing several items in sequence in
a test saves repetition of very similar code and ensures later cases are
tested even when earlier ones fail. It also encourages systematic and
uniform testing, because the same tests are applied to each data sample.
+ When a test is data-driven, you can specify its data-tag along with the
+ test-function name, as \c{function:tag}, on the command-line of the test to
+ run the test on just one specific test-case, rather than all test-cases of
+ the function. This can be used for either a global data tag or a local tag,
+ identifying a row from the function's own data; you can even combine them as
+ \c{function:global:local}.
+
\section2 Use Coverage Tools
- Use a coverage tool such as \l {Froglogic Coco Code Coverage} or \l {gcov}
+ Use a coverage tool such as \l {Coco} or \l {gcov}
to help write tests that cover as many statements, branches, and conditions
as possible in the function or class being tested. The earlier this is done
in the development cycle for a new feature, the easier it will be to catch
@@ -193,14 +180,19 @@
\section2 Select Appropriate Mechanisms to Exclude Tests
It is important to select the appropriate mechanism to exclude inapplicable
- tests: \l QSKIP(), using conditional statements to exclude parts of a test
- function, or not building the test for a particular platform.
+ tests.
+
+ Use \l QSKIP() to handle cases where a whole test function is found at
+ run-time to be inapplicable in the current test environment. When just a
+ part of a test function is to be skipped, a conditional statement can be
+ used, optionally with a \c qDebug() call to report the reason for skipping
+ the inapplicable part.
- Use QSKIP() to handle cases where a whole test function is found at run-time
- to be inapplicable in the current test environment. When just a part of a
- test function is to be skipped, a conditional statement can be used,
- optionally with a \c qDebug() call to report the reason for skipping the
- inapplicable part.
+ When there are known test failures that should eventually be fixed,
+ \l QEXPECT_FAIL is recommended, as it supports running the rest of the
+ test, when possible. It also verifies that the issue still exists, and
+ lets the code's maintainer know if they unwittingly fix it, a benefit
+ which is gained even when using the \l {QTest::}{Abort} flag.
Test functions or data rows of a data-driven test can be limited to
particular platforms, or to particular features being enabled using
@@ -215,16 +207,28 @@
attempt to run a slot that is not implemented. In the second case, the
test will not attempt to run a test slot even though it should.
- If an entire test program is inapplicable for a specific platform or
- unless a particular feature is enabled, the best approach is to use the
- parent directory's \c .pro file to avoid building the test. For example,
- if the \c tests/auto/gui/someclass test is not valid for \macOS, add the
- following line to \c tests/auto/gui.pro:
+ If an entire test program is inapplicable for a specific platform or unless
+ a particular feature is enabled, the best approach is to use the parent
+ directory's build configuration to avoid building the test. For example, if
+ the \c tests/auto/gui/someclass test is not valid for \macOS, wrap its
+ inclusion as a subdirectory in \c{tests/auto/gui/CMakeLists.txt} in a
+ platform check:
+
+ \badcode
+ if(NOT APPLE)
+ add_subdirectory(someclass)
+ endif
+ \endcode
+
+ or, if using \c qmake, add the following line to \c tests/auto/gui.pro:
\badcode
mac*: SUBDIRS -= someclass
\endcode
+ See also \l {Chapter 6: Skipping Tests with QSKIP}
+ {Skipping Tests with QSKIP}.
+
\section2 Avoid Q_ASSERT
The \l Q_ASSERT macro causes a program to abort whenever the asserted
@@ -356,24 +360,47 @@
helpful test output:
\list
- \li \l {Explicitly Ignore Expected Warnings}
+ \li \l {Test for Warnings}
\li \l {Avoid Printing Debug Messages from Autotests}
\li \l {Write Well-structured Diagnostic Code}
\endlist
- \section2 Explicitly Ignore Expected Warnings
-
- If a test is expected to cause Qt to output a warning or debug message
- on the console, it should call \l QTest::ignoreMessage() to filter that
- message out of the test output and to fail the test if the message is
- not output.
-
- If such a message is only output when Qt is built in debug mode, use
- \l QLibraryInfo::isDebugBuild() to determine whether the Qt libraries
- were built in debug mode. Using \c{#ifdef QT_DEBUG} is not enough, as
- it will only tell you whether the test was built in debug mode, and
- that does not guarantee that the Qt libraries were also built in debug
- mode.
+ \section2 Test for Warnings
+
+ Just as when building your software, if test output is cluttered with
+ warnings you will find it harder to notice a warning that really is a clue
+ to the emergence of a bug. It is thus prudent to regularly check your test
+ logs for warnings, and other extraneous output, and investigate the
+ causes. When they are signs of a bug, you can make warnings trigger test
+ failure.
+
+ When the code under test \e should produce messages, such as warnings
+ about misguided use, it is also important to test that it \e does produce
+ them when so used. You can test for expected messages from the code under
+ test, produced by \l qWarning(), \l qDebug(), \l qInfo() and friends,
+ using \l QTest::ignoreMessage(). This will verify that the message is
+ produced and filter it out of the output of the test run. If the message
+ is not produced, the test will fail.
+
+ If an expected message is only output when Qt is built in debug mode, use
+ \l QLibraryInfo::isDebugBuild() to determine whether the Qt libraries were
+ built in debug mode. Using \c{#ifdef QT_DEBUG} is not enough, as it will
+ only tell you whether \e{the test} was built in debug mode, and that does
+ not guarantee that the \e{Qt libraries} were also built in debug mode.
+
+ Your tests can (since Qt 6.3) verify that they do not trigger calls to
+ \l qWarning() by calling \l QTest::failOnWarning(). This takes the warning
+ message to test for or a \l QRegularExpression to match against warnings; if
+ a matching warning is produced, it will be reported and cause the test to
+ fail. For example, a test that should produce no warnings at all can
+ \c{QTest::failOnWarning(QRegularExpression(u".*"_s))}, which will match any
+ warning at all.
+
+ You can also set the environment variable \c QT_FATAL_WARNINGS to cause
+ warnings to be treated as fatal errors. See \l qWarning() for details; this
+ is not specific to autotests. If warnings would otherwise be lost in vast
+ test logs, the occasional run with this environment variable set can help
+ you to find and eliminate any that do arise.
\section2 Avoid Printing Debug Messages from Autotests