diff options
Diffstat (limited to 'src/testlib/doc/src/qttest-best-practices.qdoc')
| -rw-r--r-- | src/testlib/doc/src/qttest-best-practices.qdoc | 104 |
1 files changed, 74 insertions, 30 deletions
diff --git a/src/testlib/doc/src/qttest-best-practices.qdoc b/src/testlib/doc/src/qttest-best-practices.qdoc index 57f07d4a16..cea5b3a8de 100644 --- a/src/testlib/doc/src/qttest-best-practices.qdoc +++ b/src/testlib/doc/src/qttest-best-practices.qdoc @@ -108,7 +108,11 @@ Do not simply number the test-case, or use bug-tracking identifiers. Someone reading the test output will have no idea what the numbers or identifiers mean. You can add a comment on the test-row that mentions the bug-tracking - identifier, when relevant. + identifier, when relevant. It's best to avoid spacing characters and + characters that may be significant to command-line shells on which you may + want to run tests. This makes it easier to specify the test and tag on \l{Qt + Test Command Line Arguments}{the command-line} to your test program - for + example, to limit a test run to just one test-case. \section2 Write Self-contained Test Functions @@ -150,8 +154,8 @@ \section2 Use Data-driven Testing - Data-driven tests make it easier to add new tests for boundary conditions - found in later bug reports. + \l{Chapter 2: Data Driven Testing}{Data-driven tests} make it easier to add + new tests for boundary conditions found in later bug reports. Using a data-driven test rather than testing several items in sequence in a test saves repetition of very similar code and ensures later cases are @@ -167,7 +171,7 @@ \section2 Use Coverage Tools - Use a coverage tool such as \l {Froglogic Coco Code Coverage} or \l {gcov} + Use a coverage tool such as \l {Coco} or \l {gcov} to help write tests that cover as many statements, branches, and conditions as possible in the function or class being tested. The earlier this is done in the development cycle for a new feature, the easier it will be to catch @@ -176,14 +180,19 @@ \section2 Select Appropriate Mechanisms to Exclude Tests It is important to select the appropriate mechanism to exclude inapplicable - tests: \l QSKIP(), using conditional statements to exclude parts of a test - function, or not building the test for a particular platform. + tests. + + Use \l QSKIP() to handle cases where a whole test function is found at + run-time to be inapplicable in the current test environment. When just a + part of a test function is to be skipped, a conditional statement can be + used, optionally with a \c qDebug() call to report the reason for skipping + the inapplicable part. - Use QSKIP() to handle cases where a whole test function is found at run-time - to be inapplicable in the current test environment. When just a part of a - test function is to be skipped, a conditional statement can be used, - optionally with a \c qDebug() call to report the reason for skipping the - inapplicable part. + When there are known test failures that should eventually be fixed, + \l QEXPECT_FAIL is recommended, as it supports running the rest of the + test, when possible. It also verifies that the issue still exists, and + lets the code's maintainer know if they unwittingly fix it, a benefit + which is gained even when using the \l {QTest::}{Abort} flag. Test functions or data rows of a data-driven test can be limited to particular platforms, or to particular features being enabled using @@ -198,16 +207,28 @@ attempt to run a slot that is not implemented. In the second case, the test will not attempt to run a test slot even though it should. - If an entire test program is inapplicable for a specific platform or - unless a particular feature is enabled, the best approach is to use the - parent directory's \c .pro file to avoid building the test. For example, - if the \c tests/auto/gui/someclass test is not valid for \macOS, add the - following line to \c tests/auto/gui.pro: + If an entire test program is inapplicable for a specific platform or unless + a particular feature is enabled, the best approach is to use the parent + directory's build configuration to avoid building the test. For example, if + the \c tests/auto/gui/someclass test is not valid for \macOS, wrap its + inclusion as a subdirectory in \c{tests/auto/gui/CMakeLists.txt} in a + platform check: + + \badcode + if(NOT APPLE) + add_subdirectory(someclass) + endif + \endcode + + or, if using \c qmake, add the following line to \c tests/auto/gui.pro: \badcode mac*: SUBDIRS -= someclass \endcode + See also \l {Chapter 6: Skipping Tests with QSKIP} + {Skipping Tests with QSKIP}. + \section2 Avoid Q_ASSERT The \l Q_ASSERT macro causes a program to abort whenever the asserted @@ -339,24 +360,47 @@ helpful test output: \list - \li \l {Explicitly Ignore Expected Warnings} + \li \l {Test for Warnings} \li \l {Avoid Printing Debug Messages from Autotests} \li \l {Write Well-structured Diagnostic Code} \endlist - \section2 Explicitly Ignore Expected Warnings - - If a test is expected to cause Qt to output a warning or debug message - on the console, it should call \l QTest::ignoreMessage() to filter that - message out of the test output and to fail the test if the message is - not output. - - If such a message is only output when Qt is built in debug mode, use - \l QLibraryInfo::isDebugBuild() to determine whether the Qt libraries - were built in debug mode. Using \c{#ifdef QT_DEBUG} is not enough, as - it will only tell you whether the test was built in debug mode, and - that does not guarantee that the Qt libraries were also built in debug - mode. + \section2 Test for Warnings + + Just as when building your software, if test output is cluttered with + warnings you will find it harder to notice a warning that really is a clue + to the emergence of a bug. It is thus prudent to regularly check your test + logs for warnings, and other extraneous output, and investigate the + causes. When they are signs of a bug, you can make warnings trigger test + failure. + + When the code under test \e should produce messages, such as warnings + about misguided use, it is also important to test that it \e does produce + them when so used. You can test for expected messages from the code under + test, produced by \l qWarning(), \l qDebug(), \l qInfo() and friends, + using \l QTest::ignoreMessage(). This will verify that the message is + produced and filter it out of the output of the test run. If the message + is not produced, the test will fail. + + If an expected message is only output when Qt is built in debug mode, use + \l QLibraryInfo::isDebugBuild() to determine whether the Qt libraries were + built in debug mode. Using \c{#ifdef QT_DEBUG} is not enough, as it will + only tell you whether \e{the test} was built in debug mode, and that does + not guarantee that the \e{Qt libraries} were also built in debug mode. + + Your tests can (since Qt 6.3) verify that they do not trigger calls to + \l qWarning() by calling \l QTest::failOnWarning(). This takes the warning + message to test for or a \l QRegularExpression to match against warnings; if + a matching warning is produced, it will be reported and cause the test to + fail. For example, a test that should produce no warnings at all can + \c{QTest::failOnWarning(QRegularExpression(u".*"_s))}, which will match any + warning at all. + + You can also set the environment variable \c QT_FATAL_WARNINGS to cause + warnings to be treated as fatal errors. See \l qWarning() for details; this + is not specific to autotests. If warnings would otherwise be lost in vast + test logs, the occasional run with this environment variable set can help + you to find and eliminate any that do arise. \section2 Avoid Printing Debug Messages from Autotests |