path: root/doc/qtcreator/src/howto/creator-only/creator-autotest.qdoc

/****************************************************************************
**
** Copyright (C) 2019 The Qt Company Ltd.
** Contact: https://www.qt.io/licensing/
**
** This file is part of the Qt Creator documentation.
**
** Commercial License Usage
** Licensees holding valid commercial Qt licenses may use this file in
** accordance with the commercial license agreement provided with the
** Software or, alternatively, in accordance with the terms contained in
** a written agreement between you and The Qt Company. For licensing terms
** and conditions see https://www.qt.io/terms-conditions. For further
** information use the contact form at https://www.qt.io/contact-us.
**
** GNU Free Documentation License Usage
** Alternatively, this file may be used under the terms of the GNU Free
** Documentation License version 1.3 as published by the Free Software
** Foundation and appearing in the file included in the packaging of
** this file. Please review the following information to ensure
** the GNU Free Documentation License version 1.3 requirements
** will be met: https://www.gnu.org/licenses/fdl-1.3.html.
**
****************************************************************************/

/*!
    \previouspage creator-ctf-visualizer.html
    \page creator-autotest.html
    \nextpage creator-advanced.html

    \title Running Autotests

    \QC integrates the \l{Qt Test} framework,
    \l{https://github.com/google/googletest}{Google C++ Testing Framework}, and
    \l{https://www.boost.org/doc/libs/1_70_0/libs/test/doc/html/index.html}
    {Boost.Test} for unit testing applications and libraries. You can use \QC to
    create, build, and run Qt tests, Qt Quick tests (QML-based Qt tests), Google
    tests, and Boost tests for your projects.

    \image qtcreator-autotests.png

    \section1 Creating Tests

    You can use a wizard to create projects that contain tests.

    \section2 Creating Qt and Qt Quick Tests

    To create a Qt or Qt Quick test:

    \list 1
        \li Select \uicontrol File > \uicontrol {New File or Project} >
            \uicontrol {Other Project} > \uicontrol {Auto Test Project} >
            \uicontrol Choose to create a project with boilerplate code for a
            Qt test or a Qt Quick test.

        \li In the \uicontrol {Project and Test Information} dialog, specify
            settings for the project and test:

            \list 1

                \li In the \uicontrol {Test framework} field, select
                    \uicontrol {Qt Test} or \uicontrol {Qt Quick Test}.

                \li For a Qt test, select the \uicontrol {GUI Application} check
                     box to create a Qt application.

                \li In the \uicontrol {Test case name} field, enter a name for
                    the test case.

                \li For a Qt test, select the \uicontrol {Requires \QApplication}
                    check box to add the include statement for QApplication to
                    the main.cpp file of the project.

                \li Select the \uicontrol {Generate initialization and cleanup
                    code} checkbox to add functions to your test that are
                    executed by the testing framework to initialize and clean
                    up the test.

                \li In the \uicontrol {Build system} field, select the build
                    system to use for building the project: qmake, CMake, or
                    Qbs.

            \endlist

    \endlist

    \QC creates the test in the specified project directory. Edit the .cpp file
    to add private slots for each test function in your test. For more information
    about creating Qt tests, see \l{Creating a Test}.

    \section2 Creating Google Tests

    To create a Google test:

    \list 1
        \li Select \uicontrol File > \uicontrol {New File or Project} >
            \uicontrol {Other Project} > \uicontrol {Auto Test Project} >
            \uicontrol Choose to create a project with boilerplate code for a
            Google test.

        \li In the \uicontrol {Project and Test Information} dialog, specify
            settings for the project and test:

            \list 1

                \li In the \uicontrol {Test framework} field, select
                    \uicontrol {Google Test}.

                \li In the \uicontrol {Test suite name} field, enter a name for
                    the test suite.

                \li In the \uicontrol {Test case name} field, enter a name for
                    the test case.

                \li Select the \uicontrol {Enable C++ 11} check box to
                    support C++ 11 features in the test.

                \li In the \uicontrol {Google test repository} field, select
                    a directory that contains a clone of the googletest
                    repository.

                    To use an installed Google C++ Testing framework instead,
                    see \l{Setting Up the Google C++ Testing Framework}.

                \li In the \uicontrol {Build system} field, select the build
                    system to use for building the project: qmake, CMake, or
                    Qbs.

            \endlist

    \endlist

    \QC creates the test in the specified project directory.
    For more information about creating Google tests, see the
    \l{https://github.com/google/googletest/blob/master/googletest/docs/primer.md}
    {Google Test Primer}.

    \section2 Creating Boost Tests

    To build and run Boost tests, you must have the Boost.Test installed on the
    development host. Typically, it is installed when you install Boost. You can
    download Boost from \l{https://www.boost.org/}{Boost.org}.

    If Boost libraries can be found by the used compiler and build system, you
    do not need to specify the include directory when creating the test.

    To create a Boost test:

    \list 1
        \li Select \uicontrol File > \uicontrol {New File or Project} >
            \uicontrol {Other Project} > \uicontrol {Auto Test Project} >
            \uicontrol Choose to create a project with boilerplate code for a
            Boost test.
        \li In the \uicontrol {Project and Test Information} dialog, specify
            settings for the project and test:
            \list 1
                \li In the \uicontrol {Test framework} field, select
                    \uicontrol {Boost Test}.
                \li In the \uicontrol {Test suite name} field, enter a name for
                    the test suite.
                \li In the \uicontrol {Test case name} field, enter a name for
                    the test case.
                \li In the \uicontrol {Boost include dir (optional)} field,
                    enter the path to the directory that contains files needed
                    by Boost.Test, such as \e version.hpp and a subfolder called
                    \e test that contains the test header files.
                \li In the \uicontrol {Build system} field, select the build
                    system to use for building the project: qmake, CMake, or
                    Qbs.
            \endlist
    \endlist

    \QC creates the test in the specified project directory.
    For more information about creating Boost tests, see
    \l{https://www.boost.org/doc/libs/1_70_0/libs/test/doc/html/index.html}
    {Boost.Test}.

    \section1 Setting Up the Google C++ Testing Framework

    To build and run Google tests, you must have the Google C++ Testing
    framework installed and configured on the development host. You can either
    clone it from Git Hub or install it from an installation package.

    To configure a project to use a cloned Google testing framework, edit the
    \c INCLUDEPATH variable in the project file (.pro) to include the source
    and \c include folders of Google Test's \c googletest and \c googlemock.
    Usually, you need to add the following subfolders:

    \list
        \li \c googletest
        \li \c googlemock
        \li \c googletest/include
        \li \c googlemock/include
    \endlist

    You also need to add the necessary files to the \c SOURCES variable. For
    example:

    \list
        \li \c googletest/src/gtest-all.cc
        \li \c googlemock/src/gmock-all.cc
    \endlist

    To configure a project to use an installed Google testing framework package,
    add the following include paths to the .pro file:

    \list
        \li \c <googletest_install_path>/include/gtest
        \li \c <googletest_install_path>/include/gmock
    \endlist

    Then add linker options to be able to find the libraries and to link against
    them. For example, for qmake based projects, you typically need to add the
    following values to the .pro file:

    \list
        \li \c {LIBS += -lgtest -L<path_to_gtest_lib>}
        \li \c {LIBS += -lgmock -L<path_to_gmock_lib>}
    \endlist

    \section1 Building and Running Tests

    To build and run tests:

    \list 1

        \li Open a project that contains tests.

        \li In the \uicontrol Tests view, select the tests to run.

        \li In the \uicontrol {Test Results} output pane, select:

            \list
                \li \inlineimage run_small.png
                    (\uicontrol {Run All Tests}) to run all tests.
                 \li \inlineimage qtcreator-run-selected-tests.png
                    (\uicontrol {Run Selected Tests}) to run the selected tests.
                 \li \inlineimage qtcreator-run-tests-in-current-file.png
                    (\uicontrol {Run Tests for Current File}) to run the tests
                    in the file currently open in the code editor.
            \endlist

            \note By default, \QC builds a project before deploying and running
            it.

    \endlist

    If a test takes more than a minute to execute, the default timeout might
    stop the test execution. To increase the timeout, select \uicontrol Tools >
    \uicontrol Options > \uicontrol {Testing} > \uicontrol General.

    \section2 Selecting Tests to Run

    The \uicontrol Tests view shows all the tests found for the currently active
    test frameworks in the current project. Select the test cases to run.

    \image qtcreator-tests-view.png

    If a Qt Quick test case does not have a name, it is marked
    \uicontrol Unnamed in the list. Unnamed test cases are executed when you
    select \uicontrol {Run All Tests}. You cannot select or deselect them.

    \QC scans the project for tests when you open the project and updates the
    test list for the currently active test frameworks when you edit tests.
    To refresh the view, select \uicontrol {Rescan Tests} in the context menu.

    To group related test cases for an active test framework, select
    \uicontrol Tools > \uicontrol Options > \uicontrol {Testing} >
    \uicontrol General, and then select the \uicontrol Group check box next to
    the framework name in the \uicontrol {Active Test Frameworks} list.
    By default, tests are grouped based on the directory where they are located.

    To group Google tests by using a GTest filter, select
    \uicontrol {Google Test}. In the \uicontrol {Group mode} field,
    select \uicontrol {GTest Filter} and specify the filter to use in the
    \uicontrol {Active filter} field. For more information about GTest
    filters, see
    \l{https://github.com/google/googletest/blob/master/googletest/docs/advanced.md#running-a-subset-of-the-tests}
    {Running a Subset of the Tests}.

    \image qtcreator-autotests-options.png

    In some special setups, \QC cannot deduce which executable or run
    configuration it should use. If \QC repeatedly asks you to select the
    tests to run when trying to execute tests, you can enable it to cache
    your choices and use them were appropriate. The cached information is
    cleared when you switch to another project, close the current one, or
    select \uicontrol {Reset Cached Choices}.

    To show or hide init and cleanup or data functions in the \uicontrol Tests
    view, select \inlineimage filtericon.png
    (\uicontrol {Filter Test Tree}), and then select \uicontrol {Show Init and
    Cleanup Functions} or \uicontrol {Show Data Functions}. Double-click a
    function in the list to open its source code in the code editor.

    The test cases are listed in alphabetic order. To list them in the order in
    which they are defined in the source code, select \inlineimage leafsort.png
    (\uicontrol {Sort Naturally}).

    \section2 Running and Debugging Tests from Code Editor

    You can run and debug tests in the file currently open in the code editor.
    To run all tests in the open file, select \uicontrol Tools >
    \uicontrol Tests > \uicontrol {Run Tests for Current File}.

    To run the test currently selected in the open file, select
    \uicontrol {Run Test Under Cursor} in the context menu.

    To debug the currently selected test, select
    \uicontrol {Debug Test Under Cursor} in the context menu.

    \section2 Specifying Settings for Running Qt Tests

    The code inside a benchmark test is measured, and possibly also repeated
    several times in order to get an accurate measurement. This depends on the
    measurement back-end that you can select in the
    \uicontrol {Benchmark Metrics} group in \uicontrol Tools >
    \uicontrol Options > \uicontrol {Testing} > \uicontrol {Qt Test}:
    walltime, CPU tick counter, event counter, Valgrind Callgrind, and Linux
    Perf. For more information, see \l{Creating a Benchmark}.

    \image qtcreator-autotests-options-qt.png

    To receive verbose output when running benchmarks, select the
    \uicontrol {Verbose benchmarks} check box.

    To allow the debugger to interrupt Qt tests on assertions, select the
    \uicontrol {Disable crash handler while debugging} check box.

    To record information about signals and slots in the test log, select the
    \uicontrol {Log signals and slots} check box.

    \section2 Specifying Settings for Running Google Tests

    To specify settings for running Google tests, select \uicontrol Tools >
    \uicontrol Options > \uicontrol {Testing} > \uicontrol {Google Test}.

    \image qtcreator-autotests-options-google.png

    To run disabled tests, select the \uicontrol {Run disabled tests} check box.

    To run several iterations of the tests, select the \uicontrol {Repeat tests}
    check box and enter the number of times the tests should be run in the
    \uicontrol Iterations field. To make sure that the tests are independent and
    repeatable, you can run them in a different order each time by selecting the
    \uicontrol {Shuffle tests} check box.

    To turn failures into debugger breakpoints, select the
    \uicontrol {Break on failure while debugging} check box. To turn assertion
    failures into C++ exceptions, select the \uicontrol {Throw on failure} check
    box.

    \section2 Specifying Settings for Running Boost Tests

    \list 1
        \li To specify settings for running Boost tests, select \uicontrol Tools
            > \uicontrol Options > \uicontrol {Testing} >
            \uicontrol {Boost Test}.
            \image qtcreator-autotests-options-boost.png
        \li In the \uicontrol {Log format} field, select the error report
            format to specify the type of events you want recorded in the
            test report.
        \li In the \uicontrol {Report level} field, select the verbosity level
            of the test result report. Select \uicontrol No if you do not want
            a report.
        \li Select the \uicontrol Randomize check box to execute the tests in
            a random order, using the seed specified in the \uicontrol Seed
            field for initializing the randomizer.
        \li Select the \uicontrol {Catch system errors} check box to catch
            system errors.
        \li Select the \uicontrol {Floating point exceptions} check box to
            detect floating point exceptions.
        \li Select the \uicontrol {Detect memory leaks} check box to detect
            memory leaks.
    \endlist

    \section1 Viewing Test Output

    The test results are displayed in the \uicontrol {Test Results} output pane
    in XML format. XML can be parsed more easily and reliably than plain text.

    However, if a Qt test crashes, it might not produce complete XML code that
    can be parsed, which might lead to information loss. The lost information
    might be retrievable when viewing the results as plain text.
    To view the results of Qt tests as plain text, select \uicontrol Tools >
    \uicontrol Options > \uicontrol {Testing} > \uicontrol {Qt Test}, and
    then deselect the \uicontrol {Use XML output} check box. Then select the
    \inlineimage text.png
    (\uicontrol {Switch Between Visual and Text Display}) button in the
    \uicontrol {Test Results} output pane to switch to the text display.

    The following table lists the messages that the \uicontrol {Test Results}
    output pane displays:

    \table
        \header
            \li Result
            \li Description
        \row
            \li BENCH
            \li Benchmark test.
        \row
            \li DEBUG
            \li Debug message.
        \row
            \li FAIL
            \li Test case failed. Double-click the line for more information.
        \row
            \li FATAL
            \li A fatal error occurred that stops the test case from being run,
                for example.
        \row
            \li INFO
            \li Informative message.
        \row
            \li INTERNAL
            \li Internal message.
        \row
            \li PASS
            \li Test case passed.
        \row
            \li SKIP
            \li Test case was skipped.
        \row
            \li SYSTEM
            \li An error message received from or influenced by the OS.
        \row
            \li XFAIL
            \li Test case is expected to fail, so it is marked by using the
                QEXPECT_FAIL macro. If the test case passes instead, an
                unexpected pass (XPASS) is written to the test log.
        \row
            \li XPASS
            \li Test case passed even though it was expected to fail.
        \row
            \li WARN
            \li Warning message.

    \endtable

    Since Qt 5.4, you can provide a BLACKLIST file for tests. It is mainly used
    internally by the Qt CI system.

    \table
        \header
            \li Result
            \li Description
        \row
            \li BFAIL
            \li Blacklisted test case failed.
        \row
            \li BPASS
            \li Blacklisted test case passed.
        \row
            \li BXFAIL
            \li Blacklisted test case failed but was marked to be expected to fail.
        \row
            \li BXPASS
            \li Blacklisted test case passed even though it was expected to fail.

    \endtable

    To view only messages of a particular type, select
    \inlineimage filtericon.png
    (\uicontrol {Filter Test Results}), and then select the types of messages to
    show. To show all messages, select \uicontrol {Check All Filters}. To
    deselect all message types, select \uicontrol {Uncheck All Filters}.

    By default, test result output is limited to 100,000 characters. The output
    pane is automatically scrolled down when new results are added. To display
    full results, select \uicontrol Tools > \uicontrol Options >
    \uicontrol {Testing} > \uicontrol General, and then deselect the
    \uicontrol {Limit result output} check box. To disable automatic scrolling,
    deselect the \uicontrol {Automatically scroll results} check box.

    Test results can be grouped by the executable path that was used to run the
    tests. This is useful if you have multiple test executables and run them all
    at once. To enable this functionality you need to select the
    \uicontrol {Group results by application} check box.

    Internal messages and run configuration warnings for deduced configurations
    are omitted by default. To view them, deselect the \uicontrol {Omit internal
    messages} and \uicontrol {Omit run configuration warnings} check boxes.
*/