diff options
Diffstat (limited to 'src/testlib/doc/src/qttestlib-manual.qdoc')
-rw-r--r-- | src/testlib/doc/src/qttestlib-manual.qdoc | 490 |
1 files changed, 62 insertions, 428 deletions
diff --git a/src/testlib/doc/src/qttestlib-manual.qdoc b/src/testlib/doc/src/qttestlib-manual.qdoc index 4c67c34d23..1b6f534045 100644 --- a/src/testlib/doc/src/qttestlib-manual.qdoc +++ b/src/testlib/doc/src/qttestlib-manual.qdoc @@ -1,4 +1,4 @@ -// Copyright (C) 2019 The Qt Company Ltd. +// Copyright (C) 2022 The Qt Company Ltd. // Copyright (C) 2016 Intel Corporation. // SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only @@ -109,6 +109,47 @@ For more examples, refer to the \l{Qt Test Tutorial}. + \section1 Increasing Test Function Timeout + + QtTest limits the run-time of each test to catch infinite loops and similar + bugs. By default, any test function call will be interrupted after five + minutes. For data-driven tests, this applies to each call with a distinct + data-tag. This timeout can be configured by setting the \c QTEST_FUNCTION_TIMEOUT + environment variable to the maximum number of milliseconds that is acceptable + for a single call to take. If a test takes longer than the configured timeout, + it is interrupted, and \c qFatal() is called. As a result, the test aborts by + default, as if it had crashed. + + To set \c QTEST_FUNCTION_TIMEOUT from the command line on Linux or macOS, enter: + + \badcode + QTEST_FUNCTION_TIMEOUT=900000 + export QTEST_FUNCTION_TIMEOUT + \endcode + + On Windows: + \badcode + SET QTEST_FUNCTION_TIMEOUT=900000 + \endcode + + Then run the test inside this environment. + + Alternatively, you can set the environment variable programmatically in the + test code itself, for example by calling, from the + \l{Creating a Test}{initMain()} special method of your test class: + + \badcode + qputenv("QTEST_FUNCTION_TIMEOUT", "900000"); + \endcode + + To calculate a suitable value for the timeout, see how long the test usually + takes and decide how much longer it can take without that being a symptom of + some problem. Convert that longer time to milliseconds to get the timeout value. + For example, if you decide that a test that takes several minutes could + reasonably take up to twenty minutes, for example on a slow machine, + multiply \c{20 * 60 * 1000 = 1200000} and set the environment variable to + \c 1200000 instead of the \c 900000 above. + \if !defined(qtforpython) \section1 Building a Test @@ -224,7 +265,7 @@ \li \c -o \e{filename,format} \br Writes output to the specified file, in the specified format (one of \c txt, \c csv, \c junitxml, \c xml, \c lightxml, \c teamcity - or \c tap). The special filename \c - may be used to log to + or \c tap). Use the special filename \c{-} (hyphen) to log to standard output. \li \c -o \e filename \br Writes output to the specified file. @@ -307,6 +348,16 @@ Disables the crash handler on Unix platforms. On Windows, it re-enables the Windows Error Reporting dialog, which is turned off by default. This is useful for debugging crashes. + \li \c -repeat \e n \br + Run the testsuite n times or until the test fails. Useful for finding + flaky tests. If negative, the tests are repeated forever. This is intended + as a developer tool, and is only supported with the plain text logger. + \li \c -skipblacklisted \br + Skip the blacklisted tests. This option is intended to allow more accurate + measurement of test coverage by preventing blacklisted tests from inflating + coverage statistics. When not measuring test coverage, it is recommended to + execute blacklisted tests to reveal any changes in their results, such as + a new crash or the issue that caused blacklisting being resolved. \li \c -platform \e name \br This command line argument applies to all Qt applications, but might be @@ -362,8 +413,14 @@ Setting this variable to a non-zero value will cause a failure in an autotest to immediately abort the entire autotest. This is useful to e.g. debug an unstable or intermittent failure in a test, by - launching the test in a debugger. Support for this variable has been + launching the test in a debugger. Support for this variable was added in Qt 6.1. + \li \c QTEST_THROW_ON_FAIL (since 6.8) \br + Setting this variable to a non-zero value will cause QCOMPARE()/QVERIFY() + etc to throw on failure (as opposed to just returning from the + immediately-surrounding function scope). + \li \c QTEST_THROW_ON_SKIP (since 6.8) \br + Same as \c QTEST_THROW_ON_FAIL, except affecting QSKIP(). \endlist \section1 Creating a Benchmark @@ -520,9 +577,8 @@ \title Qt Test Tutorial - This tutorial gives a short introduction to how to use some of the - features of the Qt Test framework. It is divided into five - chapters: + This tutorial introduces some of the features of the Qt Test framework. It + is divided into six chapters: \list 1 \li \l {Chapter 1: Writing a Unit Test}{Writing a Unit Test} @@ -533,427 +589,5 @@ \li \l {Chapter 6: Skipping Tests with QSKIP}{Skipping Tests} \endlist -*/ - - -/*! - \example tutorial1 - - \nextpage {Chapter 2: Data Driven Testing}{Chapter 2} - - \title Chapter 1: Writing a Unit Test - \brief How to write a unit test. - - In this first chapter we will see how to write a simple unit test - for a class, and how to execute it. - - \section1 Writing a Test - - Let's assume you want to test the behavior of our QString class. - First, you need a class that contains your test functions. This class - has to inherit from QObject: - - \snippet tutorial1/testqstring.cpp 0 - - \note You need to include the QTest header and declare the test functions as - private slots so the test framework finds and executes it. - - Then you need to implement the test function itself. The - implementation could look like this: - - \snippet code/doc_src_qtestlib.cpp 8 - - The \l QVERIFY() macro evaluates the expression passed as its - argument. If the expression evaluates to true, the execution of - the test function continues. Otherwise, a message describing the - failure is appended to the test log, and the test function stops - executing. - - But if you want a more verbose output to the test log, you should - use the \l QCOMPARE() macro instead: - - \snippet tutorial1/testqstring.cpp 1 - - If the strings are not equal, the contents of both strings are - appended to the test log, making it immediately visible why the - comparison failed. - - Finally, to make our test case a stand-alone executable, the - following two lines are needed: - - \snippet tutorial1/testqstring.cpp 2 - - The \l QTEST_MAIN() macro expands to a simple \c main() - method that runs all the test functions. Note that if both the - declaration and the implementation of our test class are in a \c - .cpp file, we also need to include the generated moc file to make - Qt's introspection work. - - \section1 Executing a Test - - Now that we finished writing our test, we want to execute - it. Assuming that our test was saved as \c testqstring.cpp in an - empty directory, we build the test using \c qmake to create a - project and generate a makefile. - - \snippet code/doc_src_qtestlib.qdoc 9 - - \note If you're using windows, replace \c make with \c - nmake or whatever build tool you use. - - Running the resulting executable should give you the following - output: - - \snippet code/doc_src_qtestlib.qdoc 10 - - Congratulations! You just wrote and executed your first unit test - using the Qt Test framework. -*/ - -/*! - \example tutorial2 - - \previouspage {Chapter 1: Writing a Unit Test}{Chapter 1} - \nextpage {Chapter 3: Simulating Gui Events}{Chapter 3} - - \title Chapter 2: Data Driven Testing - \brief How to create data driven tests. - - In this chapter we will demonstrate how to execute a test - multiple times with different test data. - - So far, we have hard coded the data we wanted to test into our - test function. If we add more test data, the function might look like - this: - - \snippet code/doc_src_qtestlib.cpp 11 - - To prevent that the function ends up being cluttered by repetitive - code, Qt Test supports adding test data to a test function. All - we need is to add another private slot to our test class: - - \snippet tutorial2/testqstring.cpp 0 - - \section1 Writing the Data Function - - A test function's associated data function carries the same name, - appended by \c{_data}. Our data function looks like this: - - \snippet tutorial2/testqstring.cpp 1 - - First, we define the two elements of our test table using the \l - QTest::addColumn() function: a test string, and the - expected result of applying the QString::toUpper() function to - that string. - - Then we add some data to the table using the \l - QTest::newRow() function. Each set of data will become a - separate row in the test table. - - \l QTest::newRow() takes one argument: a name that will be associated - with the data set and used in the test log to identify the data set. - Then we stream the data set into the new table row. First an arbitrary - string, and then the expected result of applying the - QString::toUpper() function to that string. - - You can think of the test data as a two-dimensional table. In - our case, it has two columns called \c string and \c result and - three rows. In addition a name as well as an index is associated - with each row: - - \table - \header - \li index - \li name - \li string - \li result - \row - \li 0 - \li all lower - \li "hello" - \li HELLO - \row - \li 1 - \li mixed - \li "Hello" - \li HELLO - \row - \li 2 - \li all upper - \li "HELLO" - \li HELLO - \endtable - - When data is streamed into the row, each datum is asserted to match - the type of the column whose value it supplies. If any assertion fails, - the test is aborted. - - \section1 Rewriting the Test Function - - Our test function can now be rewritten: - - \snippet tutorial2/testqstring.cpp 2 - - The TestQString::toUpper() function will be executed three times, - once for each entry in the test table that we created in the - associated TestQString::toUpper_data() function. - - First, we fetch the two elements of the data set using the \l - QFETCH() macro. \l QFETCH() takes two arguments: The data type of - the element and the element name. Then we perform the test using - the \l QCOMPARE() macro. - - This approach makes it very easy to add new data to the test - without modifying the test itself. - - And again, to make our test case a stand-alone executable, - the following two lines are needed: - - \snippet tutorial2/testqstring.cpp 3 - - As before, the QTEST_MAIN() macro expands to a simple main() - method that runs all the test functions, and since both the - declaration and the implementation of our test class are in a .cpp - file, we also need to include the generated moc file to make Qt's - introspection work. -*/ - -/*! - \example tutorial3 - - \previouspage {Chapter 2: Data Driven Testing}{Chapter 2} - \nextpage {Chapter 4: Replaying GUI Events}{Chapter 4} - - \title Chapter 3: Simulating GUI Events - \brief Howe to simulate GUI events. - - Qt Test features some mechanisms to test graphical user - interfaces. Instead of simulating native window system events, - Qt Test sends internal Qt events. That means there are no - side-effects on the machine the tests are running on. - - In this chapter we will see how to write a simple GUI test. - - \section1 Writing a GUI Test - - This time, let's assume you want to test the behavior of our - QLineEdit class. As before, you will need a class that contains - your test function: - - \snippet tutorial3/testgui.cpp 0 - - The only difference is that you need to include the Qt GUI class - definitions in addition to the QTest namespace. - - \snippet tutorial3/testgui.cpp 1 - - In the implementation of the test function we first create a - QLineEdit. Then we simulate writing "hello world" in the line edit - using the \l QTest::keyClicks() function. - - \note The widget must also be shown in order to correctly test keyboard - shortcuts. - - QTest::keyClicks() simulates clicking a sequence of keys on a - widget. Optionally, a keyboard modifier can be specified as well - as a delay (in milliseconds) of the test after each key click. In - a similar way, you can use the QTest::keyClick(), - QTest::keyPress(), QTest::keyRelease(), QTest::mouseClick(), - QTest::mouseDClick(), QTest::mouseMove(), QTest::mousePress() - and QTest::mouseRelease() functions to simulate the associated - GUI events. - - Finally, we use the \l QCOMPARE() macro to check if the line edit's - text is as expected. - - As before, to make our test case a stand-alone executable, the - following two lines are needed: - - \snippet tutorial3/testgui.cpp 2 - - The QTEST_MAIN() macro expands to a simple main() method that - runs all the test functions, and since both the declaration and - the implementation of our test class are in a .cpp file, we also - need to include the generated moc file to make Qt's introspection - work. -*/ - -/*! - \example tutorial4 - - \previouspage {Chapter 3: Simulating GUI Events}{Chapter 3} - \nextpage {Chapter 5: Writing a Benchmark}{Chapter 5} - - \title Chapter 4: Replaying GUI Events - \brief How to replay GUI events. - - In this chapter, we will show how to simulate a GUI event, - and how to store a series of GUI events as well as replay them on - a widget. - - The approach to storing a series of events and replaying them is - quite similar to the approach explained in \l {Chapter 2: - Data Driven Testing}{chapter 2}. All you need to do is to add a data - function to your test class: - - \snippet tutorial4/testgui.cpp 0 - - \section1 Writing the Data Function - - As before, a test function's associated data function carries the - same name, appended by \c{_data}. - - \snippet tutorial4/testgui.cpp 1 - - First, we define the elements of the table using the - QTest::addColumn() function: A list of GUI events, and the - expected result of applying the list of events on a QWidget. Note - that the type of the first element is \l QTestEventList. - - A QTestEventList can be populated with GUI events that can be - stored as test data for later usage, or be replayed on any - QWidget. - - In our current data function, we create two \l - {QTestEventList} elements. The first list consists of a single click to - the 'a' key. We add the event to the list using the - QTestEventList::addKeyClick() function. Then we use the - QTest::newRow() function to give the data set a name, and - stream the event list and the expected result into the table. - - The second list consists of two key clicks: an 'a' with a - following 'backspace'. Again we use the - QTestEventList::addKeyClick() to add the events to the list, and - QTest::newRow() to put the event list and the expected - result into the table with an associated name. - - \section1 Rewriting the Test Function - - Our test can now be rewritten: - - \snippet tutorial4/testgui.cpp 2 - - The TestGui::testGui() function will be executed two times, - once for each entry in the test data that we created in the - associated TestGui::testGui_data() function. - - First, we fetch the two elements of the data set using the \l - QFETCH() macro. \l QFETCH() takes two arguments: the data type of - the element and the element name. Then we create a QLineEdit, and - apply the list of events on that widget using the - QTestEventList::simulate() function. - - Finally, we use the QCOMPARE() macro to check if the line edit's - text is as expected. - - As before, to make our test case a stand-alone executable, - the following two lines are needed: - - \snippet tutorial4/testgui.cpp 3 - - The QTEST_MAIN() macro expands to a simple main() method that - runs all the test functions, and since both the declaration and - the implementation of our test class are in a .cpp file, we also - need to include the generated moc file to make Qt's introspection - work. -*/ - -/*! - \example tutorial5 - - \previouspage {Chapter 4: Replaying GUI Events}{Chapter 4} - \nextpage {Chapter 6: Skipping Tests with QSKIP}{Chapter 6} - - \title Chapter 5: Writing a Benchmark - \brief How to write a benchmark. - - In this final chapter we will demonstrate how to write benchmarks - using Qt Test. - - \section1 Writing a Benchmark - To create a benchmark we extend a test function with a QBENCHMARK macro. - A benchmark test function will then typically consist of setup code and - a QBENCHMARK macro that contains the code to be measured. This test - function benchmarks QString::localeAwareCompare(). - - \snippet tutorial5/benchmarking.cpp 0 - - Setup can be done at the beginning of the function, the clock is not - running at this point. The code inside the QBENCHMARK macro will be - measured, and possibly repeated several times in order to get an - accurate measurement. - - Several \l {testlib-benchmarking-measurement}{back-ends} are available - and can be selected on the command line. - - \section1 Data Functions - - Data functions are useful for creating benchmarks that compare - multiple data inputs, for example locale aware compare against standard - compare. - - \snippet tutorial5/benchmarking.cpp 1 - - The test function then uses the data to determine what to benchmark. - - \snippet tutorial5/benchmarking.cpp 2 - - The "if (useLocaleCompare)" switch is placed outside the QBENCHMARK - macro to avoid measuring its overhead. Each benchmark test function - can have one active QBENCHMARK macro. - - \section1 External Tools - - Tools for handling and visualizing test data are available as part of - the \l {qtestlib-tools} project. - These include a tool for comparing performance data obtained from test - runs and a utility to generate Web-based graphs of performance data. - - See the \l{qtestlib-tools Announcement}{qtestlib-tools announcement} - for more information on these tools and a simple graphing example. - -*/ -/*! - \page qttestlib-tutorial6.html - - \previouspage {Chapter 5: Writing a Benchmark}{Chapter 5} - - \title Chapter 6: Skipping Tests with QSKIP - \brief How to skip tests in certain cases. - - \section2 Using QSKIP(\a description) in a test function - - If the QSKIP() macro is called from a test function, it stops - the execution of the test without adding a failure to the test log. - It can be used to skip tests that are certain to fail. The text in - the QSKIP \a description parameter is appended to the test log, - and explains why the test was not carried out. - - QSKIP can be used to skip testing when the implementation is not yet - complete or not supported on a certain platform. When there are known - failures, QEXPECT_FAIL is recommended, as it supports running the rest - of the test, when possible. - - Example of QSKIP in a test function: - - \snippet code/doc_src_qtqskip_snippet.cpp 0 - - In a data-driven test, each call to QSKIP() skips only the current - row of test data. If the data-driven test contains an unconditional - call to QSKIP, it produces a skip message for each row of test data. - - \section2 Using QSKIP in a _data function - - If called from a _data function, the QSKIP() macro stops - execution of the _data function. This prevents execution of the - associated test function. - - See below for an example: - - \snippet code/doc_src_qtqskip.cpp 1 - - \section2 Using QSKIP from initTestCase() or initTestCase_data() - If called from \c initTestCase() or \c initTestCase_data(), the - QSKIP() macro will skip all test and _data functions. */ |