diff options
Diffstat (limited to 'src/testlib/doc/src')
| -rw-r--r-- | src/testlib/doc/src/qt-webpages.qdoc | 9 | ||||
| -rw-r--r-- | src/testlib/doc/src/qt6-changes.qdoc | 2 | ||||
| -rw-r--r-- | src/testlib/doc/src/qttest-best-practices.qdoc | 104 | ||||
| -rw-r--r-- | src/testlib/doc/src/qttestlib-manual.qdoc | 534 | ||||
| -rw-r--r-- | src/testlib/doc/src/qttestlib-tutorial1.qdoc | 76 | ||||
| -rw-r--r-- | src/testlib/doc/src/qttestlib-tutorial2.qdoc | 132 | ||||
| -rw-r--r-- | src/testlib/doc/src/qttestlib-tutorial3.qdoc | 75 | ||||
| -rw-r--r-- | src/testlib/doc/src/qttestlib-tutorial4.qdoc | 95 | ||||
| -rw-r--r-- | src/testlib/doc/src/qttestlib-tutorial5.qdoc | 57 | ||||
| -rw-r--r-- | src/testlib/doc/src/qttestlib-tutorial6.qdoc | 50 |
10 files changed, 621 insertions, 513 deletions
diff --git a/src/testlib/doc/src/qt-webpages.qdoc b/src/testlib/doc/src/qt-webpages.qdoc index 0a691bb63b..611f3795ba 100644 --- a/src/testlib/doc/src/qt-webpages.qdoc +++ b/src/testlib/doc/src/qt-webpages.qdoc @@ -1,13 +1,8 @@ // Copyright (C) 2019 The Qt Company Ltd. // SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only /*! - \externalpage https://blog.qt.io/blog/2008/12/05/qtestlib-now-with-nice-graphs-pointing-upwards/ - \title qtestlib-tools Announcement -*/ - -/*! - \externalpage https://www.froglogic.com/coco/ - \title Froglogic Coco Code Coverage + \externalpage https://www.qt.io/product/quality-assurance/coco + \title Coco */ /*! diff --git a/src/testlib/doc/src/qt6-changes.qdoc b/src/testlib/doc/src/qt6-changes.qdoc index 03a5c8db03..77e7b04afd 100644 --- a/src/testlib/doc/src/qt6-changes.qdoc +++ b/src/testlib/doc/src/qt6-changes.qdoc @@ -5,7 +5,7 @@ \page testlib-changes-qt6.html \title Changes to Qt Test \ingroup changes-qt-5-to-6 - \brief Migrate Qt Test to Qt 6. + \brief Touch-related functionality changes. Qt 6 is a result of the conscious effort to make the framework more efficient and easy to use. diff --git a/src/testlib/doc/src/qttest-best-practices.qdoc b/src/testlib/doc/src/qttest-best-practices.qdoc index 57f07d4a16..cea5b3a8de 100644 --- a/src/testlib/doc/src/qttest-best-practices.qdoc +++ b/src/testlib/doc/src/qttest-best-practices.qdoc @@ -108,7 +108,11 @@ Do not simply number the test-case, or use bug-tracking identifiers. Someone reading the test output will have no idea what the numbers or identifiers mean. You can add a comment on the test-row that mentions the bug-tracking - identifier, when relevant. + identifier, when relevant. It's best to avoid spacing characters and + characters that may be significant to command-line shells on which you may + want to run tests. This makes it easier to specify the test and tag on \l{Qt + Test Command Line Arguments}{the command-line} to your test program - for + example, to limit a test run to just one test-case. \section2 Write Self-contained Test Functions @@ -150,8 +154,8 @@ \section2 Use Data-driven Testing - Data-driven tests make it easier to add new tests for boundary conditions - found in later bug reports. + \l{Chapter 2: Data Driven Testing}{Data-driven tests} make it easier to add + new tests for boundary conditions found in later bug reports. Using a data-driven test rather than testing several items in sequence in a test saves repetition of very similar code and ensures later cases are @@ -167,7 +171,7 @@ \section2 Use Coverage Tools - Use a coverage tool such as \l {Froglogic Coco Code Coverage} or \l {gcov} + Use a coverage tool such as \l {Coco} or \l {gcov} to help write tests that cover as many statements, branches, and conditions as possible in the function or class being tested. The earlier this is done in the development cycle for a new feature, the easier it will be to catch @@ -176,14 +180,19 @@ \section2 Select Appropriate Mechanisms to Exclude Tests It is important to select the appropriate mechanism to exclude inapplicable - tests: \l QSKIP(), using conditional statements to exclude parts of a test - function, or not building the test for a particular platform. + tests. + + Use \l QSKIP() to handle cases where a whole test function is found at + run-time to be inapplicable in the current test environment. When just a + part of a test function is to be skipped, a conditional statement can be + used, optionally with a \c qDebug() call to report the reason for skipping + the inapplicable part. - Use QSKIP() to handle cases where a whole test function is found at run-time - to be inapplicable in the current test environment. When just a part of a - test function is to be skipped, a conditional statement can be used, - optionally with a \c qDebug() call to report the reason for skipping the - inapplicable part. + When there are known test failures that should eventually be fixed, + \l QEXPECT_FAIL is recommended, as it supports running the rest of the + test, when possible. It also verifies that the issue still exists, and + lets the code's maintainer know if they unwittingly fix it, a benefit + which is gained even when using the \l {QTest::}{Abort} flag. Test functions or data rows of a data-driven test can be limited to particular platforms, or to particular features being enabled using @@ -198,16 +207,28 @@ attempt to run a slot that is not implemented. In the second case, the test will not attempt to run a test slot even though it should. - If an entire test program is inapplicable for a specific platform or - unless a particular feature is enabled, the best approach is to use the - parent directory's \c .pro file to avoid building the test. For example, - if the \c tests/auto/gui/someclass test is not valid for \macOS, add the - following line to \c tests/auto/gui.pro: + If an entire test program is inapplicable for a specific platform or unless + a particular feature is enabled, the best approach is to use the parent + directory's build configuration to avoid building the test. For example, if + the \c tests/auto/gui/someclass test is not valid for \macOS, wrap its + inclusion as a subdirectory in \c{tests/auto/gui/CMakeLists.txt} in a + platform check: + + \badcode + if(NOT APPLE) + add_subdirectory(someclass) + endif + \endcode + + or, if using \c qmake, add the following line to \c tests/auto/gui.pro: \badcode mac*: SUBDIRS -= someclass \endcode + See also \l {Chapter 6: Skipping Tests with QSKIP} + {Skipping Tests with QSKIP}. + \section2 Avoid Q_ASSERT The \l Q_ASSERT macro causes a program to abort whenever the asserted @@ -339,24 +360,47 @@ helpful test output: \list - \li \l {Explicitly Ignore Expected Warnings} + \li \l {Test for Warnings} \li \l {Avoid Printing Debug Messages from Autotests} \li \l {Write Well-structured Diagnostic Code} \endlist - \section2 Explicitly Ignore Expected Warnings - - If a test is expected to cause Qt to output a warning or debug message - on the console, it should call \l QTest::ignoreMessage() to filter that - message out of the test output and to fail the test if the message is - not output. - - If such a message is only output when Qt is built in debug mode, use - \l QLibraryInfo::isDebugBuild() to determine whether the Qt libraries - were built in debug mode. Using \c{#ifdef QT_DEBUG} is not enough, as - it will only tell you whether the test was built in debug mode, and - that does not guarantee that the Qt libraries were also built in debug - mode. + \section2 Test for Warnings + + Just as when building your software, if test output is cluttered with + warnings you will find it harder to notice a warning that really is a clue + to the emergence of a bug. It is thus prudent to regularly check your test + logs for warnings, and other extraneous output, and investigate the + causes. When they are signs of a bug, you can make warnings trigger test + failure. + + When the code under test \e should produce messages, such as warnings + about misguided use, it is also important to test that it \e does produce + them when so used. You can test for expected messages from the code under + test, produced by \l qWarning(), \l qDebug(), \l qInfo() and friends, + using \l QTest::ignoreMessage(). This will verify that the message is + produced and filter it out of the output of the test run. If the message + is not produced, the test will fail. + + If an expected message is only output when Qt is built in debug mode, use + \l QLibraryInfo::isDebugBuild() to determine whether the Qt libraries were + built in debug mode. Using \c{#ifdef QT_DEBUG} is not enough, as it will + only tell you whether \e{the test} was built in debug mode, and that does + not guarantee that the \e{Qt libraries} were also built in debug mode. + + Your tests can (since Qt 6.3) verify that they do not trigger calls to + \l qWarning() by calling \l QTest::failOnWarning(). This takes the warning + message to test for or a \l QRegularExpression to match against warnings; if + a matching warning is produced, it will be reported and cause the test to + fail. For example, a test that should produce no warnings at all can + \c{QTest::failOnWarning(QRegularExpression(u".*"_s))}, which will match any + warning at all. + + You can also set the environment variable \c QT_FATAL_WARNINGS to cause + warnings to be treated as fatal errors. See \l qWarning() for details; this + is not specific to autotests. If warnings would otherwise be lost in vast + test logs, the occasional run with this environment variable set can help + you to find and eliminate any that do arise. \section2 Avoid Printing Debug Messages from Autotests diff --git a/src/testlib/doc/src/qttestlib-manual.qdoc b/src/testlib/doc/src/qttestlib-manual.qdoc index c99c947253..1b6f534045 100644 --- a/src/testlib/doc/src/qttestlib-manual.qdoc +++ b/src/testlib/doc/src/qttestlib-manual.qdoc @@ -109,6 +109,47 @@ For more examples, refer to the \l{Qt Test Tutorial}. + \section1 Increasing Test Function Timeout + + QtTest limits the run-time of each test to catch infinite loops and similar + bugs. By default, any test function call will be interrupted after five + minutes. For data-driven tests, this applies to each call with a distinct + data-tag. This timeout can be configured by setting the \c QTEST_FUNCTION_TIMEOUT + environment variable to the maximum number of milliseconds that is acceptable + for a single call to take. If a test takes longer than the configured timeout, + it is interrupted, and \c qFatal() is called. As a result, the test aborts by + default, as if it had crashed. + + To set \c QTEST_FUNCTION_TIMEOUT from the command line on Linux or macOS, enter: + + \badcode + QTEST_FUNCTION_TIMEOUT=900000 + export QTEST_FUNCTION_TIMEOUT + \endcode + + On Windows: + \badcode + SET QTEST_FUNCTION_TIMEOUT=900000 + \endcode + + Then run the test inside this environment. + + Alternatively, you can set the environment variable programmatically in the + test code itself, for example by calling, from the + \l{Creating a Test}{initMain()} special method of your test class: + + \badcode + qputenv("QTEST_FUNCTION_TIMEOUT", "900000"); + \endcode + + To calculate a suitable value for the timeout, see how long the test usually + takes and decide how much longer it can take without that being a symptom of + some problem. Convert that longer time to milliseconds to get the timeout value. + For example, if you decide that a test that takes several minutes could + reasonably take up to twenty minutes, for example on a slow machine, + multiply \c{20 * 60 * 1000 = 1200000} and set the environment variable to + \c 1200000 instead of the \c 900000 above. + \if !defined(qtforpython) \section1 Building a Test @@ -224,7 +265,7 @@ \li \c -o \e{filename,format} \br Writes output to the specified file, in the specified format (one of \c txt, \c csv, \c junitxml, \c xml, \c lightxml, \c teamcity - or \c tap). The special filename \c - may be used to log to + or \c tap). Use the special filename \c{-} (hyphen) to log to standard output. \li \c -o \e filename \br Writes output to the specified file. @@ -307,6 +348,16 @@ Disables the crash handler on Unix platforms. On Windows, it re-enables the Windows Error Reporting dialog, which is turned off by default. This is useful for debugging crashes. + \li \c -repeat \e n \br + Run the testsuite n times or until the test fails. Useful for finding + flaky tests. If negative, the tests are repeated forever. This is intended + as a developer tool, and is only supported with the plain text logger. + \li \c -skipblacklisted \br + Skip the blacklisted tests. This option is intended to allow more accurate + measurement of test coverage by preventing blacklisted tests from inflating + coverage statistics. When not measuring test coverage, it is recommended to + execute blacklisted tests to reveal any changes in their results, such as + a new crash or the issue that caused blacklisting being resolved. \li \c -platform \e name \br This command line argument applies to all Qt applications, but might be @@ -362,8 +413,14 @@ Setting this variable to a non-zero value will cause a failure in an autotest to immediately abort the entire autotest. This is useful to e.g. debug an unstable or intermittent failure in a test, by - launching the test in a debugger. Support for this variable has been + launching the test in a debugger. Support for this variable was added in Qt 6.1. + \li \c QTEST_THROW_ON_FAIL (since 6.8) \br + Setting this variable to a non-zero value will cause QCOMPARE()/QVERIFY() + etc to throw on failure (as opposed to just returning from the + immediately-surrounding function scope). + \li \c QTEST_THROW_ON_SKIP (since 6.8) \br + Same as \c QTEST_THROW_ON_FAIL, except affecting QSKIP(). \endlist \section1 Creating a Benchmark @@ -532,478 +589,5 @@ \li \l {Chapter 6: Skipping Tests with QSKIP}{Skipping Tests} \endlist - \note You can build and execute the tests from each chapter using the - available source code, which is linked to at the end of each chapter. - -*/ - - -/*! - \example tutorial1 - - \previouspage {Qt Test Tutorial}{Qt Test Tutorial Overview} - \nextpage {Chapter 2: Data Driven Testing}{Chapter 2} - - \title Chapter 1: Writing a Unit Test - \brief How to write a unit test. - - This first chapter demonstrates how to write a simple unit test and how to - run the test case as a stand-alone executable. - - \section1 Writing a Test - - Let's assume you want to test the behavior of our QString class. - First, you need a class that contains your test functions. This class - has to inherit from QObject: - - \snippet tutorial1/testqstring.cpp 0 - - \note You need to include the QTest header and declare the test functions as - private slots so the test framework finds and executes it. - - Then you need to implement the test function itself. The - implementation could look like this: - - \snippet code/doc_src_qtestlib.cpp 8 - - The \l QVERIFY() macro evaluates the expression passed as its - argument. If the expression evaluates to true, the execution of - the test function continues. Otherwise, a message describing the - failure is appended to the test log, and the test function stops - executing. - - But if you want a more verbose output to the test log, you should - use the \l QCOMPARE() macro instead: - - \snippet tutorial1/testqstring.cpp 1 - - If the strings are not equal, the contents of both strings are - appended to the test log, making it immediately visible why the - comparison failed. - - \section1 Preparing the Stand-Alone Executable - - Finally, to make our test case a stand-alone executable, the - following two lines are needed: - - \snippet tutorial1/testqstring.cpp 2 - - The \l QTEST_MAIN() macro expands to a simple \c main() - method that runs all the test functions. Note that if both the - declaration and the implementation of our test class are in a \c - .cpp file, we also need to include the generated moc file to make - Qt's introspection work. - - \section1 Building the Executable - - \include {building-examples.qdocinc} {building the executable} {tutorial1} - - \note If you're using windows, replace \c make with \c - nmake or whatever build tool you use. - - \section1 Running the Executable - - Running the resulting executable should give you the following - output: - - \snippet code/doc_src_qtestlib.qdoc 10 - - Congratulations! You just wrote and executed your first unit test - using the Qt Test framework. -*/ - -/*! - \example tutorial2 - - \previouspage {Chapter 1: Writing a Unit Test}{Chapter 1} - \nextpage {Chapter 3: Simulating Gui Events}{Chapter 3} - - \title Chapter 2: Data Driven Testing - \brief How to create data driven tests. - - This chapter demonstrates how to execute a test multiple times with - different test data. - - So far, we have hard coded the data we wanted to test into our - test function. If we add more test data, the function might look like - this: - - \snippet code/doc_src_qtestlib.cpp 11 - - To prevent the function from being cluttered with repetitive code, Qt Test - supports adding test data to a test function. All we need is to add another - private slot to our test class: - - \snippet tutorial2/testqstring.cpp 0 - - \section1 Writing the Data Function - - A test function's associated data function has \c _data appended to its - name. Our data function looks like this: - - \snippet tutorial2/testqstring.cpp 1 - - First, we define the two elements of our test table using the \l - QTest::addColumn() function: a test string and the - expected result of applying the QString::toUpper() function to - that string. - - Then, we add some data to the table using the \l - QTest::newRow() function. Each set of data will become a - separate row in the test table. - - \l QTest::newRow() takes one argument: a name that will be associated - with the data set and used in the test log to identify the data set. - Then, we stream the data set into the new table row. First an arbitrary - string, and then the expected result of applying the - QString::toUpper() function to that string. - - You can think of the test data as a two-dimensional table. In - our case, it has two columns called \c string and \c result and - three rows. In addition, a name and an index are associated - with each row: - - \table - \header - \li index - \li name - \li string - \li result - \row - \li 0 - \li all lower - \li "hello" - \li HELLO - \row - \li 1 - \li mixed - \li "Hello" - \li HELLO - \row - \li 2 - \li all upper - \li "HELLO" - \li HELLO - \endtable - - When data is streamed into the row, each datum is asserted to match - the type of the column whose value it supplies. If any assertion fails, - the test is aborted. - - \section1 Rewriting the Test Function - - Our test function can now be rewritten: - - \snippet tutorial2/testqstring.cpp 2 - - The TestQString::toUpper() function will be executed three times, - once for each entry in the test table that we created in the - associated TestQString::toUpper_data() function. - - First, we fetch the two elements of the data set using the \l - QFETCH() macro. \l QFETCH() takes two arguments: The data type of - the element and the element name. Then, we perform the test using - the \l QCOMPARE() macro. - - This approach makes it very easy to add new data to the test - without modifying the test itself. - - \section1 Preparing the Stand-Alone Executable - - And again, to make our test case a stand-alone executable, - the following two lines are needed: - - \snippet tutorial2/testqstring.cpp 3 - - As before, the QTEST_MAIN() macro expands to a simple main() - method that runs all the test functions, and since both the - declaration and the implementation of our test class are in a .cpp - file, we also need to include the generated moc file to make Qt's - introspection work. - - \section1 Building the Executable - - \include {building-examples.qdocinc} {building the executable} {tutorial2} - - \section1 Running the Executable - - Running the resulting executable should give you the following - output: - - \snippet code/doc_src_qtestlib.qdoc 11 -*/ - -/*! - \example tutorial3 - - \previouspage {Chapter 2: Data Driven Testing}{Chapter 2} - \nextpage {Chapter 4: Replaying GUI Events}{Chapter 4} - - \title Chapter 3: Simulating GUI Events - \brief Howe to simulate GUI events. - - Qt Test features some mechanisms to test graphical user - interfaces. Instead of simulating native window system events, - Qt Test sends internal Qt events. That means there are no - side-effects on the machine the tests are running on. - - This chapter demonstrates how to write a simple GUI test. - - \section1 Writing a GUI Test - - This time, let's assume you want to test the behavior of our - QLineEdit class. As before, you will need a class that contains - your test function: - - \snippet tutorial3/testgui.cpp 0 - - The only difference is that you need to include the Qt GUI class - definitions in addition to the QTest namespace. - - \snippet tutorial3/testgui.cpp 1 - - In the implementation of the test function, we first create a - QLineEdit. Then, we simulate writing "hello world" in the line edit - using the \l QTest::keyClicks() function. - - \note The widget must also be shown in order to correctly test keyboard - shortcuts. - - QTest::keyClicks() simulates clicking a sequence of keys on a - widget. Optionally, a keyboard modifier can be specified as well - as a delay (in milliseconds) of the test after each key click. In - a similar way, you can use the QTest::keyClick(), - QTest::keyPress(), QTest::keyRelease(), QTest::mouseClick(), - QTest::mouseDClick(), QTest::mouseMove(), QTest::mousePress() - and QTest::mouseRelease() functions to simulate the associated - GUI events. - - Finally, we use the \l QCOMPARE() macro to check if the line edit's - text is as expected. - - \section1 Preparing the Stand-Alone Executable - - As before, to make our test case a stand-alone executable, the - following two lines are needed: - - \snippet tutorial3/testgui.cpp 2 - - The QTEST_MAIN() macro expands to a simple main() method that - runs all the test functions, and since both the declaration and - the implementation of our test class are in a .cpp file, we also - need to include the generated moc file to make Qt's introspection - work. - - \section1 Building the Executable - - \include {building-examples.qdocinc} {building the executable} {tutorial3} - - \section1 Running the Executable - - Running the resulting executable should give you the following - output: - - \snippet code/doc_src_qtestlib.qdoc 12 -*/ - -/*! - \example tutorial4 - - \previouspage {Chapter 3: Simulating GUI Events}{Chapter 3} - \nextpage {Chapter 5: Writing a Benchmark}{Chapter 5} - - \title Chapter 4: Replaying GUI Events - \brief How to replay GUI events. - - In this chapter, we will show how to simulate a GUI event, - and how to store a series of GUI events as well as replay them on - a widget. - - The approach to storing a series of events and replaying them is - quite similar to the approach explained in \l {Chapter 2: - Data Driven Testing}{chapter 2}. All you need to do is to add a data - function to your test class: - - \snippet tutorial4/testgui.cpp 0 - - \section1 Writing the Data Function - - As before, a test function's associated data function carries the - same name, appended by \c{_data}. - - \snippet tutorial4/testgui.cpp 1 - - First, we define the elements of the table using the - QTest::addColumn() function: A list of GUI events, and the - expected result of applying the list of events on a QWidget. Note - that the type of the first element is \l QTestEventList. - - A QTestEventList can be populated with GUI events that can be - stored as test data for later usage, or be replayed on any - QWidget. - - In our current data function, we create two \l - {QTestEventList} elements. The first list consists of a single click to - the 'a' key. We add the event to the list using the - QTestEventList::addKeyClick() function. Then we use the - QTest::newRow() function to give the data set a name, and - stream the event list and the expected result into the table. - - The second list consists of two key clicks: an 'a' with a - following 'backspace'. Again we use the - QTestEventList::addKeyClick() to add the events to the list, and - QTest::newRow() to put the event list and the expected - result into the table with an associated name. - - \section1 Rewriting the Test Function - - Our test can now be rewritten: - - \snippet tutorial4/testgui.cpp 2 - - The TestGui::testGui() function will be executed two times, - once for each entry in the test data that we created in the - associated TestGui::testGui_data() function. - - First, we fetch the two elements of the data set using the \l - QFETCH() macro. \l QFETCH() takes two arguments: the data type of - the element and the element name. Then we create a QLineEdit, and - apply the list of events on that widget using the - QTestEventList::simulate() function. - - Finally, we use the QCOMPARE() macro to check if the line edit's - text is as expected. - - \section1 Preparing the Stand-Alone Executable - - As before, to make our test case a stand-alone executable, - the following two lines are needed: - - \snippet tutorial4/testgui.cpp 3 - - The QTEST_MAIN() macro expands to a simple main() method that - runs all the test functions, and since both the declaration and - the implementation of our test class are in a .cpp file, we also - need to include the generated moc file to make Qt's introspection - work. - - \section1 Building the Executable - - \include {building-examples.qdocinc} {building the executable} {tutorial4} - - \section1 Running the Executable - - Running the resulting executable should give you the following - output: - - \snippet code/doc_src_qtestlib.qdoc 13 -*/ - -/*! - \example tutorial5 - - \previouspage {Chapter 4: Replaying GUI Events}{Chapter 4} - \nextpage {Chapter 6: Skipping Tests with QSKIP}{Chapter 6} - - \title Chapter 5: Writing a Benchmark - \brief How to write a benchmark. - - This final demonstrates how to write benchmarks using Qt Test. - - \section1 Writing a Benchmark - To create a benchmark we extend a test function with a QBENCHMARK macro. - A benchmark test function will then typically consist of setup code and - a QBENCHMARK macro that contains the code to be measured. This test - function benchmarks QString::localeAwareCompare(). - - \snippet tutorial5/benchmarking.cpp 0 - - Setup can be done at the beginning of the function. At this point, the clock - is not running. The code inside the QBENCHMARK macro will be - measured, and possibly repeated several times in order to get an - accurate measurement. - - Several \l {testlib-benchmarking-measurement}{back-ends} are available - and can be selected on the command line. - - \section1 Data Functions - - Data functions are useful for creating benchmarks that compare - multiple data inputs, for example locale aware compare against standard - compare. - - \snippet tutorial5/benchmarking.cpp 1 - - The test function then uses the data to determine what to benchmark. - - \snippet tutorial5/benchmarking.cpp 2 - - The "if (useLocaleCompare)" switch is placed outside the QBENCHMARK - macro to avoid measuring its overhead. Each benchmark test function - can have one active QBENCHMARK macro. - - \section1 External Tools - - Tools for handling and visualizing test data are available as part of - the \l {qtestlib-tools} project. - These include a tool for comparing performance data obtained from test - runs and a utility to generate Web-based graphs of performance data. - - See the \l{qtestlib-tools Announcement}{qtestlib-tools announcement} - for more information on these tools and a simple graphing example. - - \section1 Building the Executable - - \include {building-examples.qdocinc} {building the executable} {tutorial5} - - \section1 Running the Executable - - Running the resulting executable should give you the following - output: - - \snippet code/doc_src_qtestlib.qdoc 14 -*/ -/*! - \page qttestlib-tutorial6.html - - \previouspage {Chapter 5: Writing a Benchmark}{Chapter 5} - - \title Chapter 6: Skipping Tests with QSKIP - \brief How to skip tests in certain cases. - - \section2 Using QSKIP(\a description) in a test function - - If the QSKIP() macro is called from a test function, it stops - the execution of the test without adding a failure to the test log. - It can be used to skip tests that are certain to fail. The text in - the QSKIP \a description parameter is appended to the test log, - and explains why the test was not carried out. - - QSKIP can be used to skip testing when the implementation is not yet - complete or not supported on a certain platform. When there are known - failures, QEXPECT_FAIL is recommended, as it supports running the rest - of the test, when possible. - - Example of QSKIP in a test function: - - \snippet code/doc_src_qtqskip_snippet.cpp 0 - - In a data-driven test, each call to QSKIP() skips only the current - row of test data. If the data-driven test contains an unconditional - call to QSKIP, it produces a skip message for each row of test data. - - \section2 Using QSKIP in a _data function - - If called from a _data function, the QSKIP() macro stops - execution of the _data function. This prevents execution of the - associated test function. - - See below for an example: - - \snippet code/doc_src_qtqskip.cpp 1 - - \section2 Using QSKIP from initTestCase() or initTestCase_data() - If called from \c initTestCase() or \c initTestCase_data(), the - QSKIP() macro will skip all test and _data functions. */ diff --git a/src/testlib/doc/src/qttestlib-tutorial1.qdoc b/src/testlib/doc/src/qttestlib-tutorial1.qdoc new file mode 100644 index 0000000000..41d9264c0d --- /dev/null +++ b/src/testlib/doc/src/qttestlib-tutorial1.qdoc @@ -0,0 +1,76 @@ +// Copyright (C) 2023 The Qt Company Ltd. +// Copyright (C) 2016 Intel Corporation. +// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only + +/*! + \page qttestlib-tutorial1-example.html + \previouspage {Qt Test Tutorial}{Qt Test Tutorial Overview} + \nextpage {Chapter 2: Data Driven Testing}{Chapter 2} + + \title Chapter 1: Writing a Unit Test + \brief How to write a unit test. + + This first chapter demonstrates how to write a simple unit test and how to + run the test case as a stand-alone executable. + + \section1 Writing a Test + + Let's assume you want to test the behavior of our QString class. + First, you need a class that contains your test functions. This class + has to inherit from QObject: + + \snippet tutorial1/testqstring.cpp 0 + + \note You need to include the QTest header and declare the test functions as + private slots so the test framework finds and executes it. + + Then you need to implement the test function itself. The + implementation could look like this: + + \snippet code/doc_src_qtestlib.cpp 8 + + The \l QVERIFY() macro evaluates the expression passed as its + argument. If the expression evaluates to true, the execution of + the test function continues. Otherwise, a message describing the + failure is appended to the test log, and the test function stops + executing. + + But if you want a more verbose output to the test log, you should + use the \l QCOMPARE() macro instead: + + \snippet tutorial1/testqstring.cpp 1 + + If the strings are not equal, the contents of both strings are + appended to the test log, making it immediately visible why the + comparison failed. + + \section1 Preparing the Stand-Alone Executable + + Finally, to make our test case a stand-alone executable, the + following two lines are needed: + + \snippet tutorial1/testqstring.cpp 2 + + The \l QTEST_MAIN() macro expands to a simple \c main() + method that runs all the test functions. Note that if both the + declaration and the implementation of our test class are in a \c + .cpp file, we also need to include the generated moc file to make + Qt's introspection work. + + \section1 Building the Executable + + \include {building-examples.qdocinc} {building the executable} {tutorial1} + + \note If you're using windows, replace \c make with \c + nmake or whatever build tool you use. + + \section1 Running the Executable + + Running the resulting executable should give you the following + output: + + \snippet code/doc_src_qtestlib.qdoc 10 + + Congratulations! You just wrote and executed your first unit test + using the Qt Test framework. +*/ diff --git a/src/testlib/doc/src/qttestlib-tutorial2.qdoc b/src/testlib/doc/src/qttestlib-tutorial2.qdoc new file mode 100644 index 0000000000..bd828b3963 --- /dev/null +++ b/src/testlib/doc/src/qttestlib-tutorial2.qdoc @@ -0,0 +1,132 @@ +// Copyright (C) 2023 The Qt Company Ltd. +// Copyright (C) 2016 Intel Corporation. +// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only + +/*! + \page qttestlib-tutorial2-example.html + \previouspage {Chapter 1: Writing a Unit Test}{Chapter 1} + \nextpage {Chapter 3: Simulating Gui Events}{Chapter 3} + + \title Chapter 2: Data Driven Testing + \brief How to create data driven tests. + + This chapter demonstrates how to execute a test multiple times with + different test data. + + So far, we have hard coded the data we wanted to test into our + test function. If we add more test data, the function might look like + this: + + \snippet code/doc_src_qtestlib.cpp 11 + + To prevent the function from being cluttered with repetitive code, Qt Test + supports adding test data to a test function. All we need is to add another + private slot to our test class: + + \snippet tutorial2/testqstring.cpp 0 + + \section1 Writing the Data Function + + A test function's associated data function has \c _data appended to its + name. Our data function looks like this: + + \snippet tutorial2/testqstring.cpp 1 + + First, we define the two elements of our test table using the \l + QTest::addColumn() function: a test string and the + expected result of applying the QString::toUpper() function to + that string. + + Then, we add some data to the table using the \l QTest::newRow() + function. We can also use \l QTest::addRow() if we need to format some data + in the row name, for example when generating many data rows iteratively. + Each row of data will become a separate row in the test table. + + \l QTest::newRow() takes one argument: a name that will be associated with + the data set and used in the test log to identify the data row. \l + QTest::addRow() takes a (\c{printf}-style) format string followed by the + parameters to be represented in place of the formatting tokens in the format + string. Then, we stream the data set into the new table row. First an + arbitrary string, and then the expected result of applying the + QString::toUpper() function to that string. + + You can think of the test data as a two-dimensional table. In + our case, it has two columns called \c string and \c result and + three rows. In addition, a name and an index are associated + with each row: + + \table + \header + \li index + \li name + \li string + \li result + \row + \li 0 + \li all-lower + \li "hello" + \li HELLO + \row + \li 1 + \li mixed + \li "Hello" + \li HELLO + \row + \li 2 + \li all-upper + \li "HELLO" + \li HELLO + \endtable + + When data is streamed into the row, each datum is asserted to match + the type of the column whose value it supplies. If any assertion fails, + the test is aborted. + + The names of rows and columns, in a given test function's data table, should + be unique: if two rows share a name, or two columns share a name, a warning + will (since Qt 6.5) be produced. See \l qWarning() for how you can cause + warnings to be treated as errors and \l {Test for Warnings} for how to get + your tests clear of other warnings. + + \section1 Rewriting the Test Function + + Our test function can now be rewritten: + + \snippet tutorial2/testqstring.cpp 2 + + The TestQString::toUpper() function will be executed three times, + once for each entry in the test table that we created in the + associated TestQString::toUpper_data() function. + + First, we fetch the two elements of the data set using the \l + QFETCH() macro. \l QFETCH() takes two arguments: The data type of + the element and the element name. Then, we perform the test using + the \l QCOMPARE() macro. + + This approach makes it very easy to add new data to the test + without modifying the test itself. + + \section1 Preparing the Stand-Alone Executable + + And again, to make our test case a stand-alone executable, + the following two lines are needed: + + \snippet tutorial2/testqstring.cpp 3 + + As before, the QTEST_MAIN() macro expands to a simple main() + method that runs all the test functions, and since both the + declaration and the implementation of our test class are in a .cpp + file, we also need to include the generated moc file to make Qt's + introspection work. + + \section1 Building the Executable + + \include {building-examples.qdocinc} {building the executable} {tutorial2} + + \section1 Running the Executable + + Running the resulting executable should give you the following + output: + + \snippet code/doc_src_qtestlib.qdoc 11 +*/ diff --git a/src/testlib/doc/src/qttestlib-tutorial3.qdoc b/src/testlib/doc/src/qttestlib-tutorial3.qdoc new file mode 100644 index 0000000000..2b7fe25c96 --- /dev/null +++ b/src/testlib/doc/src/qttestlib-tutorial3.qdoc @@ -0,0 +1,75 @@ +// Copyright (C) 2023 The Qt Company Ltd. +// Copyright (C) 2016 Intel Corporation. +// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only + +/*! + \page qttestlib-tutorial3-example.html + \previouspage {Chapter 2: Data Driven Testing}{Chapter 2} + \nextpage {Chapter 4: Replaying GUI Events}{Chapter 4} + + \title Chapter 3: Simulating GUI Events + \brief How to simulate GUI events. + + Qt Test features some mechanisms to test graphical user + interfaces. Instead of simulating native window system events, + Qt Test sends internal Qt events. That means there are no + side-effects on the machine the tests are running on. + + This chapter demonstrates how to write a simple GUI test. + + \section1 Writing a GUI Test + + This time, let's assume you want to test the behavior of our + QLineEdit class. As before, you will need a class that contains + your test function: + + \snippet tutorial3/testgui.cpp 0 + + The only difference is that you need to include the Qt GUI class + definitions in addition to the QTest namespace. + + \snippet tutorial3/testgui.cpp 1 + + In the implementation of the test function, we first create a + QLineEdit. Then, we simulate writing "hello world" in the line edit + using the \l QTest::keyClicks() function. + + \note The widget must also be shown in order to correctly test keyboard + shortcuts. + + QTest::keyClicks() simulates clicking a sequence of keys on a + widget. Optionally, a keyboard modifier can be specified as well + as a delay (in milliseconds) of the test after each key click. In + a similar way, you can use the QTest::keyClick(), + QTest::keyPress(), QTest::keyRelease(), QTest::mouseClick(), + QTest::mouseDClick(), QTest::mouseMove(), QTest::mousePress() + and QTest::mouseRelease() functions to simulate the associated + GUI events. + + Finally, we use the \l QCOMPARE() macro to check if the line edit's + text is as expected. + + \section1 Preparing the Stand-Alone Executable + + As before, to make our test case a stand-alone executable, the + following two lines are needed: + + \snippet tutorial3/testgui.cpp 2 + + The QTEST_MAIN() macro expands to a simple main() method that + runs all the test functions, and since both the declaration and + the implementation of our test class are in a .cpp file, we also + need to include the generated moc file to make Qt's introspection + work. + + \section1 Building the Executable + + \include {building-examples.qdocinc} {building the executable} {tutorial3} + + \section1 Running the Executable + + Running the resulting executable should give you the following + output: + + \snippet code/doc_src_qtestlib.qdoc 12 +*/ diff --git a/src/testlib/doc/src/qttestlib-tutorial4.qdoc b/src/testlib/doc/src/qttestlib-tutorial4.qdoc new file mode 100644 index 0000000000..d5a0121f67 --- /dev/null +++ b/src/testlib/doc/src/qttestlib-tutorial4.qdoc @@ -0,0 +1,95 @@ +// Copyright (C) 2022 The Qt Company Ltd. +// Copyright (C) 2016 Intel Corporation. +// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only + +/*! + \page qttestlib-tutorial4-example.html + \previouspage {Chapter 3: Simulating GUI Events}{Chapter 3} + \nextpage {Chapter 5: Writing a Benchmark}{Chapter 5} + + \title Chapter 4: Replaying GUI Events + \brief How to replay GUI events. + + In this chapter, we will show how to simulate a GUI event, + and how to store a series of GUI events as well as replay them on + a widget. + + The approach to storing a series of events and replaying them is + quite similar to the approach explained in \l {Chapter 2: + Data Driven Testing}{chapter 2}. All you need to do is to add a data + function to your test class: + + \snippet tutorial4/testgui.cpp 0 + + \section1 Writing the Data Function + + As before, a test function's associated data function carries the + same name, appended by \c{_data}. + + \snippet tutorial4/testgui.cpp 1 + + First, we define the elements of the table using the + QTest::addColumn() function: A list of GUI events, and the + expected result of applying the list of events on a QWidget. Note + that the type of the first element is \l QTestEventList. + + A QTestEventList can be populated with GUI events that can be + stored as test data for later usage, or be replayed on any + QWidget. + + In our current data function, we create two \l + {QTestEventList} elements. The first list consists of a single click to + the 'a' key. We add the event to the list using the + QTestEventList::addKeyClick() function. Then we use the + QTest::newRow() function to give the data set a name, and + stream the event list and the expected result into the table. + + The second list consists of two key clicks: an 'a' with a + following 'backspace'. Again we use the + QTestEventList::addKeyClick() to add the events to the list, and + QTest::newRow() to put the event list and the expected + result into the table with an associated name. + + \section1 Rewriting the Test Function + + Our test can now be rewritten: + + \snippet tutorial4/testgui.cpp 2 + + The TestGui::testGui() function will be executed two times, + once for each entry in the test data that we created in the + associated TestGui::testGui_data() function. + + First, we fetch the two elements of the data set using the \l + QFETCH() macro. \l QFETCH() takes two arguments: the data type of + the element and the element name. Then we create a QLineEdit, and + apply the list of events on that widget using the + QTestEventList::simulate() function. + + Finally, we use the QCOMPARE() macro to check if the line edit's + text is as expected. + + \section1 Preparing the Stand-Alone Executable + + As before, to make our test case a stand-alone executable, + the following two lines are needed: + + \snippet tutorial4/testgui.cpp 3 + + The QTEST_MAIN() macro expands to a simple main() method that + runs all the test functions, and since both the declaration and + the implementation of our test class are in a .cpp file, we also + need to include the generated moc file to make Qt's introspection + work. + + \section1 Building the Executable + + \include {building-examples.qdocinc} {building the executable} {tutorial4} + + \section1 Running the Executable + + Running the resulting executable should give you the following + output: + + \snippet code/doc_src_qtestlib.qdoc 13 +*/ diff --git a/src/testlib/doc/src/qttestlib-tutorial5.qdoc b/src/testlib/doc/src/qttestlib-tutorial5.qdoc new file mode 100644 index 0000000000..7569019b4e --- /dev/null +++ b/src/testlib/doc/src/qttestlib-tutorial5.qdoc @@ -0,0 +1,57 @@ +// Copyright (C) 2023 The Qt Company Ltd. +// Copyright (C) 2016 Intel Corporation. +// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only + +/*! + \page qttestlib-tutorial5-example.html + \previouspage {Chapter 4: Replaying GUI Events}{Chapter 4} + \nextpage {Chapter 6: Skipping Tests with QSKIP}{Chapter 6} + + \title Chapter 5: Writing a Benchmark + \brief How to write a benchmark. + + This chapter demonstrates how to write benchmarks using Qt Test. + + \section1 Writing a Benchmark + To create a benchmark we extend a test function with a QBENCHMARK macro. + A benchmark test function will then typically consist of setup code and + a QBENCHMARK macro that contains the code to be measured. This test + function benchmarks QString::localeAwareCompare(). + + \snippet tutorial5/benchmarking.cpp 0 + + Setup can be done at the beginning of the function. At this point, the clock + is not running. The code inside the QBENCHMARK macro will be + measured, and possibly repeated several times in order to get an + accurate measurement. + + Several \l {testlib-benchmarking-measurement}{back-ends} are available + and can be selected on the command line. + + \section1 Data Functions + + Data functions are useful for creating benchmarks that compare + multiple data inputs, for example locale aware compare against standard + compare. + + \snippet tutorial5/benchmarking.cpp 1 + + The test function then uses the data to determine what to benchmark. + + \snippet tutorial5/benchmarking.cpp 2 + + The \c{if (useLocaleCompare)} switch is placed outside the QBENCHMARK + macro to avoid measuring its overhead. Each benchmark test function + can have one active QBENCHMARK macro. + + \section1 Building the Executable + + \include {building-examples.qdocinc} {building the executable} {tutorial5} + + \section1 Running the Executable + + Running the resulting executable should give you the following + output: + + \snippet code/doc_src_qtestlib.qdoc 14 +*/ diff --git a/src/testlib/doc/src/qttestlib-tutorial6.qdoc b/src/testlib/doc/src/qttestlib-tutorial6.qdoc new file mode 100644 index 0000000000..602ca0b28e --- /dev/null +++ b/src/testlib/doc/src/qttestlib-tutorial6.qdoc @@ -0,0 +1,50 @@ +// Copyright (C) 2023 The Qt Company Ltd. +// Copyright (C) 2016 Intel Corporation. +// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only + +/*! + \page qttestlib-tutorial6.html + + \previouspage {Chapter 5: Writing a Benchmark}{Chapter 5} + + \title Chapter 6: Skipping Tests with QSKIP + \brief How to skip tests in certain cases. + + \section2 Using QSKIP(\a description) in a test function + + If the \l QSKIP() macro is called from a test function, it stops + the execution of the test without adding a failure to the test log. + It can be used to skip tests that are certain to fail. The text in + the QSKIP \a description parameter is appended to the test log, + and should explain why the test was not carried out. + + QSKIP can be used to skip testing when the implementation is not yet + complete or not supported on a certain platform. When there are known + failures, \l QEXPECT_FAIL is recommended, as it supports running the rest + of the test, when possible. + + Example of QSKIP in a test function: + + \snippet code/doc_src_qtqskip_snippet.cpp 0 + + In a data-driven test, each call to QSKIP() skips only the current + row of test data. If the data-driven test contains an unconditional + call to QSKIP, it produces a skip message for each row of test data. + + \section2 Using QSKIP in a _data function + + If called from a _data function, the QSKIP() macro stops + execution of the _data function. This prevents execution of the + associated test function. + + See below for an example: + + \snippet code/doc_src_qtqskip.cpp 1 + + \section2 Using QSKIP from initTestCase() or initTestCase_data() + + If called from \c initTestCase() or \c initTestCase_data(), the + QSKIP() macro will skip all test and _data functions. + + \sa {Select Appropriate Mechanisms to Exclude Tests} +*/ |