diff options
Diffstat (limited to 'src/testlib/doc/src/qttestlib-manual.qdoc')
| -rw-r--r-- | src/testlib/doc/src/qttestlib-manual.qdoc | 578 |
1 files changed, 112 insertions, 466 deletions
diff --git a/src/testlib/doc/src/qttestlib-manual.qdoc b/src/testlib/doc/src/qttestlib-manual.qdoc index 9073cc7c88..9f650496bb 100644 --- a/src/testlib/doc/src/qttestlib-manual.qdoc +++ b/src/testlib/doc/src/qttestlib-manual.qdoc @@ -1,30 +1,6 @@ -/**************************************************************************** -** -** Copyright (C) 2019 The Qt Company Ltd. -** Copyright (C) 2016 Intel Corporation. -** Contact: https://www.qt.io/licensing/ -** -** This file is part of the documentation of the Qt Toolkit. -** -** $QT_BEGIN_LICENSE:FDL$ -** Commercial License Usage -** Licensees holding valid commercial Qt licenses may use this file in -** accordance with the commercial license agreement provided with the -** Software or, alternatively, in accordance with the terms contained in -** a written agreement between you and The Qt Company. For licensing terms -** and conditions see https://www.qt.io/terms-conditions. For further -** information use the contact form at https://www.qt.io/contact-us. -** -** GNU Free Documentation License Usage -** Alternatively, this file may be used under the terms of the GNU Free -** Documentation License version 1.3 as published by the Free Software -** Foundation and appearing in the file included in the packaging of -** this file. Please review the following information to ensure -** the GNU Free Documentation License version 1.3 requirements -** will be met: https://www.gnu.org/licenses/fdl-1.3.html. -** $QT_END_LICENSE$ -** -****************************************************************************/ +// Copyright (C) 2022 The Qt Company Ltd. +// Copyright (C) 2016 Intel Corporation. +// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only /*! \page qtest-overview.html @@ -133,6 +109,47 @@ For more examples, refer to the \l{Qt Test Tutorial}. + \section1 Increasing Test Function Timeout + + QtTest limits the run-time of each test to catch infinite loops and similar + bugs. By default, any test function call will be interrupted after five + minutes. For data-driven tests, this applies to each call with a distinct + data-tag. This timeout can be configured by setting the \c QTEST_FUNCTION_TIMEOUT + environment variable to the maximum number of milliseconds that is acceptable + for a single call to take. If a test takes longer than the configured timeout, + it is interrupted, and \c qFatal() is called. As a result, the test aborts by + default, as if it had crashed. + + To set \c QTEST_FUNCTION_TIMEOUT from the command line on Linux or macOS, enter: + + \badcode + QTEST_FUNCTION_TIMEOUT=900000 + export QTEST_FUNCTION_TIMEOUT + \endcode + + On Windows: + \badcode + SET QTEST_FUNCTION_TIMEOUT=900000 + \endcode + + Then run the test inside this environment. + + Alternatively, you can set the environment variable programmatically in the + test code itself, for example by calling, from the + \l{Creating a Test}{initMain()} special method of your test class: + + \badcode + qputenv("QTEST_FUNCTION_TIMEOUT", "900000"); + \endcode + + To calculate a suitable value for the timeout, see how long the test usually + takes and decide how much longer it can take without that being a symptom of + some problem. Convert that longer time to milliseconds to get the timeout value. + For example, if you decide that a test that takes several minutes could + reasonably take up to twenty minutes, for example on a slow machine, + multiply \c{20 * 60 * 1000 = 1200000} and set the environment variable to + \c 1200000 instead of the \c 900000 above. + \if !defined(qtforpython) \section1 Building a Test @@ -153,6 +170,12 @@ All labeled targets will be run when \c {test} target is called on the command line. + \note On Android, if you have one connected device or emulator, tests will + run on that device. If you have more than one device connected, set the + environment variable \c {ANDROID_DEVICE_SERIAL} to the + \l {Android: Query for devices}{ADB serial number} of the device that + you want to run tests on. + There are several other advantages with CMake. For example, the result of a test run can be published on a web server using CDash with virtually no effort. @@ -222,9 +245,9 @@ \snippet code/doc_src_qtestlib.qdoc 4 Runs the \c toUpper test function with all available test data, - and the \c toInt test function with the test data called \c + and the \c toInt test function with the test data row called \c zero (if the specified test data doesn't exist, the associated test - will fail). + will fail and the available data tags are reported). \snippet code/doc_src_qtestlib.qdoc 5 @@ -240,26 +263,28 @@ \list \li \c -o \e{filename,format} \br - Writes output to the specified file, in the specified format (one of - \c txt, \c xml, \c lightxml, \c junitxml or \c tap). The special filename \c - - may be used to log to standard output. + Writes output to the specified file, in the specified format (one + of \c txt, \c csv, \c junitxml, \c xml, \c lightxml, \c teamcity + or \c tap). Use the special filename \c{-} (hyphen) to log to + standard output. \li \c -o \e filename \br Writes output to the specified file. \li \c -txt \br Outputs results in plain text. + \li \c -csv \br + Outputs results as comma-separated values (CSV) suitable for + import into spreadsheets. This mode is only suitable for + benchmarks, since it suppresses normal pass/fail messages. + \li \c -junitxml \br + Outputs results as a \l{JUnit XML} document. \li \c -xml \br Outputs results as an XML document. \li \c -lightxml \br Outputs results as a stream of XML tags. - \li \c -junitxml \br - Outputs results as an JUnit XML document. - \li \c -csv \br - Outputs results as comma-separated values (CSV). This mode is only suitable for - benchmarks, since it suppresses normal pass/fail messages. \li \c -teamcity \br - Outputs results in TeamCity format. + Outputs results in \l{TeamCity} format. \li \c -tap \br - Outputs results in Test Anything Protocol (TAP) format. + Outputs results in \l{Test Anything Protocol} (TAP) format. \endlist The first version of the \c -o option may be repeated in order to log @@ -323,6 +348,16 @@ Disables the crash handler on Unix platforms. On Windows, it re-enables the Windows Error Reporting dialog, which is turned off by default. This is useful for debugging crashes. + \li \c -repeat \e n \br + Run the testsuite n times or until the test fails. Useful for finding + flaky tests. If negative, the tests are repeated forever. This is intended + as a developer tool, and is only supported with the plain text logger. + \li \c -skipblacklisted \br + Skip the blacklisted tests. This option is intended to allow more accurate + measurement of test coverage by preventing blacklisted tests from inflating + coverage statistics. When not measuring test coverage, it is recommended to + execute blacklisted tests to reveal any changes in their results, such as + a new crash or the issue that caused blacklisting being resolved. \li \c -platform \e name \br This command line argument applies to all Qt applications, but might be @@ -378,8 +413,14 @@ Setting this variable to a non-zero value will cause a failure in an autotest to immediately abort the entire autotest. This is useful to e.g. debug an unstable or intermittent failure in a test, by - launching the test in a debugger. Support for this variable has been + launching the test in a debugger. Support for this variable was added in Qt 6.1. + \li \c QTEST_THROW_ON_FAIL (since 6.8) \br + Setting this variable to a non-zero value will cause QCOMPARE()/QVERIFY() + etc to throw on failure (as opposed to just returning from the + immediately-surrounding function scope). + \li \c QTEST_THROW_ON_SKIP (since 6.8) \br + Same as \c QTEST_THROW_ON_FAIL, except affecting QSKIP(). \endlist \section1 Creating a Benchmark @@ -477,7 +518,7 @@ You can define \c{initTestCase_data()} to set up a global test data table. Each test is run once for each row in the global test data table. When the - test function itself \l{Chapter 2: Data-driven Testing}{is data-driven}, + test function itself \l{Chapter 2: Data Driven Testing}{is data-driven}, it is run for each local data row, for each global data row. So, if there are \c g rows in the global data table and \c d rows in the test's own data-table, the number of runs of this test is \c g times \c d. @@ -500,6 +541,32 @@ each locale provided by \c {initTestCase_data()}: \snippet code/src_qtestlib_qtestcase_snippet.cpp 31 + + On the command-line of a test you can pass the name of a function (with no + test-class-name prefix) to run only that one function's tests. If the test + class has global data, or the function is data-driven, you can append a data + tag, after a colon, to run only that tag's data-set for the function. To + specify both a global tag and a tag specific to the test function, combine + them with a colon between, putting the global data tag first. For example + + \snippet code/doc_src_qtestlib.qdoc 6 + + will run the \c zero test-case of the \c roundTripInt() test above (assuming + its \c TestQLocale class has been compiled to an executable \c testqlocale) + in each of the locales specified by \c initTestCase_data(), while + + \snippet code/doc_src_qtestlib.qdoc 7 + + will run all three test-cases of \c roundTripInt() only in the C locale and + + \snippet code/doc_src_qtestlib.qdoc 8 + + will only run the \c zero test-case in the C locale. + + Providing such fine-grained control over which tests are to be run can make + it considerably easier to debug a problem, as you only need to step through + the one test-case that has been seen to fail. + */ /*! @@ -510,9 +577,8 @@ \title Qt Test Tutorial - This tutorial gives a short introduction to how to use some of the - features of the Qt Test framework. It is divided into five - chapters: + This tutorial introduces some of the features of the Qt Test framework. It + is divided into six chapters: \list 1 \li \l {Chapter 1: Writing a Unit Test}{Writing a Unit Test} @@ -523,427 +589,7 @@ \li \l {Chapter 6: Skipping Tests with QSKIP}{Skipping Tests} \endlist -*/ - - -/*! - \example tutorial1 - - \nextpage {Chapter 2: Data Driven Testing}{Chapter 2} - - \title Chapter 1: Writing a Unit Test - \brief How to write a unit test. - - In this first chapter we will see how to write a simple unit test - for a class, and how to execute it. - - \section1 Writing a Test - - Let's assume you want to test the behavior of our QString class. - First, you need a class that contains your test functions. This class - has to inherit from QObject: - - \snippet tutorial1/testqstring.cpp 0 - - \note You need to include the QTest header and declare the test functions as - private slots so the test framework finds and executes it. - - Then you need to implement the test function itself. The - implementation could look like this: - - \snippet code/doc_src_qtestlib.cpp 8 - - The \l QVERIFY() macro evaluates the expression passed as its - argument. If the expression evaluates to true, the execution of - the test function continues. Otherwise, a message describing the - failure is appended to the test log, and the test function stops - executing. - - But if you want a more verbose output to the test log, you should - use the \l QCOMPARE() macro instead: - - \snippet tutorial1/testqstring.cpp 1 - - If the strings are not equal, the contents of both strings are - appended to the test log, making it immediately visible why the - comparison failed. - - Finally, to make our test case a stand-alone executable, the - following two lines are needed: - - \snippet tutorial1/testqstring.cpp 2 - - The \l QTEST_MAIN() macro expands to a simple \c main() - method that runs all the test functions. Note that if both the - declaration and the implementation of our test class are in a \c - .cpp file, we also need to include the generated moc file to make - Qt's introspection work. - - \section1 Executing a Test - - Now that we finished writing our test, we want to execute - it. Assuming that our test was saved as \c testqstring.cpp in an - empty directory, we build the test using qmake to create a project - and generate a makefile. - - \snippet code/doc_src_qtestlib.qdoc 9 - - \note If you're using windows, replace \c make with \c - nmake or whatever build tool you use. - - Running the resulting executable should give you the following - output: - - \snippet code/doc_src_qtestlib.qdoc 10 - - Congratulations! You just wrote and executed your first unit test - using the Qt Test framework. -*/ - -/*! - \example tutorial2 - - \previouspage {Chapter 1: Writing a Unit Test}{Chapter 1} - \nextpage {Chapter 3: Simulating Gui Events}{Chapter 3} - - \title Chapter 2: Data Driven Testing - \brief How to create data driven tests. - - In this chapter we will demonstrate how to execute a test - multiple times with different test data. - - So far, we have hard coded the data we wanted to test into our - test function. If we add more test data, the function might look like - this: - - \snippet code/doc_src_qtestlib.cpp 11 - - To prevent that the function ends up being cluttered by repetitive - code, Qt Test supports adding test data to a test function. All - we need is to add another private slot to our test class: - - \snippet tutorial2/testqstring.cpp 0 - - \section1 Writing the Data Function - - A test function's associated data function carries the same name, - appended by \c{_data}. Our data function looks like this: - - \snippet tutorial2/testqstring.cpp 1 - - First, we define the two elements of our test table using the \l - QTest::addColumn() function: a test string, and the - expected result of applying the QString::toUpper() function to - that string. - - Then we add some data to the table using the \l - QTest::newRow() function. Each set of data will become a - separate row in the test table. - - \l QTest::newRow() takes one argument: a name that will be associated - with the data set and used in the test log to identify the data set. - Then we stream the data set into the new table row. First an arbitrary - string, and then the expected result of applying the - QString::toUpper() function to that string. - - You can think of the test data as a two-dimensional table. In - our case, it has two columns called \c string and \c result and - three rows. In addition a name as well as an index is associated - with each row: - - \table - \header - \li index - \li name - \li string - \li result - \row - \li 0 - \li all lower - \li "hello" - \li HELLO - \row - \li 1 - \li mixed - \li "Hello" - \li HELLO - \row - \li 2 - \li all upper - \li "HELLO" - \li HELLO - \endtable - - When data is streamed into the row, each datum is asserted to match - the type of the column whose value it supplies. If any assertion fails, - the test is aborted. - - \section1 Rewriting the Test Function - - Our test function can now be rewritten: - - \snippet tutorial2/testqstring.cpp 2 - - The TestQString::toUpper() function will be executed three times, - once for each entry in the test table that we created in the - associated TestQString::toUpper_data() function. - - First, we fetch the two elements of the data set using the \l - QFETCH() macro. \l QFETCH() takes two arguments: The data type of - the element and the element name. Then we perform the test using - the \l QCOMPARE() macro. - - This approach makes it very easy to add new data to the test - without modifying the test itself. - - And again, to make our test case a stand-alone executable, - the following two lines are needed: - - \snippet tutorial2/testqstring.cpp 3 - - As before, the QTEST_MAIN() macro expands to a simple main() - method that runs all the test functions, and since both the - declaration and the implementation of our test class are in a .cpp - file, we also need to include the generated moc file to make Qt's - introspection work. -*/ - -/*! - \example tutorial3 - - \previouspage {Chapter 2: Data Driven Testing}{Chapter 2} - \nextpage {Chapter 4: Replaying GUI Events}{Chapter 4} - - \title Chapter 3: Simulating GUI Events - \brief Howe to simulate GUI events. - - Qt Test features some mechanisms to test graphical user - interfaces. Instead of simulating native window system events, - Qt Test sends internal Qt events. That means there are no - side-effects on the machine the tests are running on. - - In this chapter we will see how to write a simple GUI test. - - \section1 Writing a GUI Test - - This time, let's assume you want to test the behavior of our - QLineEdit class. As before, you will need a class that contains - your test function: - - \snippet tutorial3/testgui.cpp 0 - - The only difference is that you need to include the Qt GUI class - definitions in addition to the QTest namespace. - - \snippet tutorial3/testgui.cpp 1 - - In the implementation of the test function we first create a - QLineEdit. Then we simulate writing "hello world" in the line edit - using the \l QTest::keyClicks() function. - - \note The widget must also be shown in order to correctly test keyboard - shortcuts. - - QTest::keyClicks() simulates clicking a sequence of keys on a - widget. Optionally, a keyboard modifier can be specified as well - as a delay (in milliseconds) of the test after each key click. In - a similar way, you can use the QTest::keyClick(), - QTest::keyPress(), QTest::keyRelease(), QTest::mouseClick(), - QTest::mouseDClick(), QTest::mouseMove(), QTest::mousePress() - and QTest::mouseRelease() functions to simulate the associated - GUI events. - - Finally, we use the \l QCOMPARE() macro to check if the line edit's - text is as expected. - - As before, to make our test case a stand-alone executable, the - following two lines are needed: - - \snippet tutorial3/testgui.cpp 2 - - The QTEST_MAIN() macro expands to a simple main() method that - runs all the test functions, and since both the declaration and - the implementation of our test class are in a .cpp file, we also - need to include the generated moc file to make Qt's introspection - work. -*/ - -/*! - \example tutorial4 - - \previouspage {Chapter 3: Simulating GUI Events}{Chapter 3} - \nextpage {Chapter 5: Writing a Benchmark}{Chapter 5} - - \title Chapter 4: Replaying GUI Events - \brief How to replay GUI events. - - In this chapter, we will show how to simulate a GUI event, - and how to store a series of GUI events as well as replay them on - a widget. - - The approach to storing a series of events and replaying them is - quite similar to the approach explained in \l {Chapter 2: - Data Driven Testing}{chapter 2}. All you need to do is to add a data - function to your test class: - - \snippet tutorial4/testgui.cpp 0 - - \section1 Writing the Data Function - - As before, a test function's associated data function carries the - same name, appended by \c{_data}. - - \snippet tutorial4/testgui.cpp 1 - - First, we define the elements of the table using the - QTest::addColumn() function: A list of GUI events, and the - expected result of applying the list of events on a QWidget. Note - that the type of the first element is \l QTestEventList. - - A QTestEventList can be populated with GUI events that can be - stored as test data for later usage, or be replayed on any - QWidget. - - In our current data function, we create two \l - {QTestEventList} elements. The first list consists of a single click to - the 'a' key. We add the event to the list using the - QTestEventList::addKeyClick() function. Then we use the - QTest::newRow() function to give the data set a name, and - stream the event list and the expected result into the table. - - The second list consists of two key clicks: an 'a' with a - following 'backspace'. Again we use the - QTestEventList::addKeyClick() to add the events to the list, and - QTest::newRow() to put the event list and the expected - result into the table with an associated name. - - \section1 Rewriting the Test Function - - Our test can now be rewritten: - - \snippet tutorial4/testgui.cpp 2 - - The TestGui::testGui() function will be executed two times, - once for each entry in the test data that we created in the - associated TestGui::testGui_data() function. - - First, we fetch the two elements of the data set using the \l - QFETCH() macro. \l QFETCH() takes two arguments: the data type of - the element and the element name. Then we create a QLineEdit, and - apply the list of events on that widget using the - QTestEventList::simulate() function. - - Finally, we use the QCOMPARE() macro to check if the line edit's - text is as expected. - - As before, to make our test case a stand-alone executable, - the following two lines are needed: - - \snippet tutorial4/testgui.cpp 3 - - The QTEST_MAIN() macro expands to a simple main() method that - runs all the test functions, and since both the declaration and - the implementation of our test class are in a .cpp file, we also - need to include the generated moc file to make Qt's introspection - work. -*/ - -/*! - \example tutorial5 - - \previouspage {Chapter 4: Replaying GUI Events}{Chapter 4} - \nextpage {Chapter 6: Skipping Tests with QSKIP}{Chapter 6} - - \title Chapter 5: Writing a Benchmark - \brief How to write a benchmark. - - In this final chapter we will demonstrate how to write benchmarks - using Qt Test. - - \section1 Writing a Benchmark - To create a benchmark we extend a test function with a QBENCHMARK macro. - A benchmark test function will then typically consist of setup code and - a QBENCHMARK macro that contains the code to be measured. This test - function benchmarks QString::localeAwareCompare(). - - \snippet tutorial5/benchmarking.cpp 0 - - Setup can be done at the beginning of the function, the clock is not - running at this point. The code inside the QBENCHMARK macro will be - measured, and possibly repeated several times in order to get an - accurate measurement. - - Several \l {testlib-benchmarking-measurement}{back-ends} are available - and can be selected on the command line. - - \section1 Data Functions - - Data functions are useful for creating benchmarks that compare - multiple data inputs, for example locale aware compare against standard - compare. - - \snippet tutorial5/benchmarking.cpp 1 - - The test function then uses the data to determine what to benchmark. - - \snippet tutorial5/benchmarking.cpp 2 - - The "if (useLocaleCompare)" switch is placed outside the QBENCHMARK - macro to avoid measuring its overhead. Each benchmark test function - can have one active QBENCHMARK macro. - - \section1 External Tools - - Tools for handling and visualizing test data are available as part of - the \l {qtestlib-tools} project. - These include a tool for comparing performance data obtained from test - runs and a utility to generate Web-based graphs of performance data. - - See the \l{qtestlib-tools Announcement}{qtestlib-tools announcement} - for more information on these tools and a simple graphing example. - -*/ -/*! - \page qttestlib-tutorial6.html - - \previouspage {Chapter 5: Writing a Benchmark}{Chapter 5} - - \title Chapter 6: Skipping Tests with QSKIP - \brief How to skip tests in certain cases. - - \section2 Using QSKIP(\a description) in a test function - - If the QSKIP() macro is called from a test function, it stops - the execution of the test without adding a failure to the test log. - It can be used to skip tests that are certain to fail. The text in - the QSKIP \a description parameter is appended to the test log, - and explains why the test was not carried out. - - QSKIP can be used to skip testing when the implementation is not yet - complete or not supported on a certain platform. When there are known - failures, it is recommended to use QEXPECT_FAIL, so that the test is - always completely executed. - - Example of QSKIP in a test function: - - \snippet code/doc_src_qtqskip_snippet.cpp 0 - - In a data-driven test, each call to QSKIP() skips only the current - row of test data. If the data-driven test contains an unconditional - call to QSKIP, it produces a skip message for each row of test data. - - \section2 Using QSKIP in a _data function - - If called from a _data function, the QSKIP() macro stops - execution of the _data function. This prevents execution of the - associated test function. - - See below for an example: - - \snippet code/doc_src_qtqskip.cpp 1 - - \section2 Using QSKIP from initTestCase() or initTestCase_data() + \note You can build and execute the tests from each chapter using the + available source code, which is linked to at the end of each chapter. - If called from \c initTestCase() or \c initTestCase_data(), the - QSKIP() macro will skip all test and _data functions. */ |