path: root/doc/src/qtuitest_manual.qdoc

/*!

\page qtuitest_manual.html
\title QtUiTest Manual

The QtUiTest framework is a tool for automatic and semi-automatic User Interface/System level testing of Qt based applications and libraries. QtUiTest provides functionality commonly found in system test frameworks but with a focus on maintainable cross-platform tests.

Contents:
\tableofcontents

\section1 Introduction

QtUiTest is, by design, quite similar to the \l{QTest}{QTestLib API} that is used for Unit level testing. In contrast to QTestLib however, QtUiTest based tests are written in QtScript. Please refer to the \l{QTestLib Manual} if you are looking for a Unit test framework.

QtUiTest is a scripting test language plus some basic test execution tools and as such does not provide a complete Integrated Development Environment (IDE) to write and execute tests. Tests can be written with any standard editor or development IDE. Tests can be executed from the command line or in continuous integration test systems.

\section1 The Scripting Language

\section2 Language design philosophy

QtUiTest aims to provide a test scripting language that is:
\list
\o simple, reliable and easy to learn,
\o resilient to UI changes,
\o transparent to device changes (i.e. a touchscreen and keyboard based device should run the same test),
\o suitable for performance testing.
\endlist

QtUiTest scripts are written from a User perspective. When a User uses a GUI application all he/she does is apply a few concepts that form the basis of using a GUI application. For instance, a button can be clicked on and then 'the text on the button' will happen.

It is irrelevant to the user that the actual UI implementation uses a button. The UI could just as well have been an html page with a hyperlink in it with a similar text as on the button. In both cases, the user will 'select' the text to initiate a certain action from the application. So, today the implementation can be a button, tomorrow it can be a hyperlink, the day after tomorrow it can be something that hasn't been invented yet: the principle remains the same.

So conceptually a user does not 'click' keys, but rather uses the concept of a button to 'select' an action or hits keys on a real or virtual keyboard to 'enter' text. The user wants something to happen and has an understanding of what needs to be done to "get there".

On a new system a user first needs to learn the User Manual. The more the system behaves in a way the user already knows, and the more the system provides functionality in a way the user can predict, the more "Intuitive" and "Usable" the system is experienced by the user.

System testing is, in our view, very much related to these fundamental issues and just like a real user, QtUiTest understands the UI concepts that form the foundation of a UI application and uses a simple syntax to enable the tester to describe it's Use Cases.

Next to being able to write tests in a language that is as close as possible to the user experience it is also important to write maintainable code. Automating tests is well known to be major investment, and tests being broken for every small UI change, or tests that need to be modified for each slightly different device (or Operating System) are a major problem. QtUiTest is designed in such a way that the script will result in good readable and understandable code which reduces the need for additional documentation, as well as tests that are platform agnostic.

\list
\o The four most important commands are \l{QSystemTest::}{select()}, \l{QSystemTest::}{enter()}, \l{QSystemTest::}{compare()} and \l{QSystemTest::}{verify()}.
\o Every command is making verification steps along the way. For instance, \l{QSystemTest::}{enter()} will perform a verification at the end to ensure that the text has been entered correctly.
\o There is no need to explicitly code key clicks or mouse clicks. The usage of key/mouse clicks typically results in low level code that is difficult to understand and should only be used in exceptional cases.
\o The test can be executed equally well with a keyboard based and/or a mouse/touchscreen based system.
\o There is usually no need for explicit synchronisation steps such as a \l{QSystemTest::}{wait()}: QtUiTest understands that certain actions may result in a background task and will wait automatically.
\o QtUiTest uses the Labels that are visible on the screen to find the editable 'buddy fields'.
\o QtUiTest is forgiving to UI changes. For instance, the user "Name" field may start as a simple line edit, then may become a full text edit, and finally become a combobox but all this time the test will pass without as much as a single complaint: as long as a field labeled "Name" exists and a text can be successfully entered into a buddy field the test is a success from user perspective.
\o Test Data and Test Logic are separated.
\endlist

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

\section1 Using QtUiTest

\section2 Creating a Test

Each System Test is stored in its own file ending with the \c .js extension.

Within the test script, a \c testcase object is created containing several test functions.
Within each test function, the \l {QtUiTest API Reference}{QtUiTest API} scripting commands may be used, as well as
standard commands provided by \l{QtScript Module}{QtScript}.

Helper functions and global variables can be written outside of the testcase object.

For example:

\code
testcase = {
    testFunction1: function() {
        startApplication("Dog Walker");
        select("Walk the dog...", optionsMenu());
        compare( getText(), "Dog was walked." );
    },

    testFunction2: function() {
        var i = 1;
        helper(i);
    }
}

function helper(num) {
    verify( num > 0 );
    ...
}
\endcode

The project files for System tests are usually very simple, for example:
\code
    CONFIG+=systemtest
    SOURCES+=sys_mytest.js
\endcode
Adding "systemtest" to CONFIG enables the use of the "make test" command.

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

\section3 Writing a Manual System Test

Writing a Manual System test is typically the first step in many test automation efforts. It is far better to document and run through the steps manually to check that an idea is executable (and repeatable) before time is invested in automating all steps.

The downside about Manual tests is that the test is usually 'somewhere else', i.e. in a text document or spreadsheet that isn't necessarily saved in the same version control system as the source code it tests, or in a format that can be easily merged by the version control system.

QtUiTest integrates the Manual test process into the Automation process and makes it possible to seamlessly transition from a full Manual to a Semi-Automatic to a Full-Automatic test. All work is done in the same text file, so the intermediate steps are all recorded in the version control system for easy retrieval, merging and tracking.

A Manual test is written by using the prompt() command:
\code
    my_manual_test: function()
    {
        prompt( "* Do something\n"+
                "* Do something else\n"+
                "* Verify that the result is as expected");
    }
\endcode

When a System test is executed it can be a combination of fully automated and semi-automated test functions. The automated tests simply execute without user intervention and result in a PASS/FAIL/etc. A Manual test will be executed until a prompt is detected at which time a Dialog is shown on the Desktop (host) machine containing the specified prompt text. In the dialog, the stars (*) will be replaced with "1:", "2:", etc to indicate the steps that need to be taken in sequence. The advantage of using stars (*) in the code is that lines can be inserted/removed without having to renumber the remaining steps until the end of the prompt.

The Dialog will also contain "Pass" and "Fail" buttons as well as a "Record" button that puts the test system into recording mode. The Pass and Fail buttons obviously must be used to indicate the result of the manual step.

It is perfectly fine, and to be expected, that prompts() will be combined with automated steps. For instance, one could launch an application and then select a few menu options before arriving at the section of the test that requires manual intervention.

For instance:
\code
    startApplication("Dog Walker");
    select("Walk the dog...");
    prompt( "* Verify that the dog is having a good time");
\endcode

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

\section2 Running a Test

System tests can usually be run using the command "make test". Alternatively, the QtUiTest Script Execution engine (qtuitestrunner) can be called directly:
\code
    qtuitestrunner <scriptname> <options>
\endcode
These commands should be run on the desktop machine (host environment).


\section3 Running tests in a Continuous Integration System

QtUiTest tests can be executed from the command line of a desktop (Host) machine and as such are perfectly suited to be executed in Continuous Integration Test Systems. The output format of the test, the testresults, can be printed to the screen or saved into a file in a number of formats (such as XML) to allow easy post processing by test-reporting and defect-tracking systems.

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

\section2 QtUiTest Tutorial

Please refer to the \l {QtUiTest Tutorial} for a detailed example about writing QtUiTest test cases.

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

\section1 QtUiTest API Overview

    To write a System Test, create a new file containing a test script, which defines a "testcase" object with one or more test functions.

    For example:
    \code
        testcase = {
            testFunction1: function() {
                ...
            },
            testFunction2: function() {
                ...
            }
        }
    \endcode

    The testfunctions can be written with a combination of standard QtScript and special QtUiTest commands, which are grouped below.

    \list
    \o \l {System Architecture} - briefly describes the architecture of QtUiTest.
    \o \l {Query Paths} - specifying widgets and objects to access.
    \o \l {Querying Objects} - getting information from widgets or objects.
    \o \l {Learn Mode} - dynamically generating test data.
    \o \l {Application Management} - starting and stopping applications.
    \o \l {File Management} - accessing files on the test system.
    \o \l {Time Management} - functionality to manage the system time on the device.
    \o \l {Key Simulation} - simulating entering text, selection of items from the menus or lists, and generation of arbitrary key events.
    \o \l {Mouse / Touchscreen Simulation} - simulating user interaction via a mouse or touchscreen.
    \o \l {Other} - some miscellaneous functionality.
    \endlist

    \section2 System Architecture

    The QtUiTest framework is designed around a script execution engine (qtuitestrunner) that is executed on the desktop (host machine) and a set of QtUiTest
    plugins that are loaded by the System Under Test (SUT). The plugins are running in the SUT process space, i.e. may be running on a device, and communicate with the System Test script execution engine via a socket based communication protocol.

    Each test command is sent as a query to the SUT, and the script runner will be blocked until a reply is received from the SUT.

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

    \section2 Query Paths

    Most QtUiTest functions use 'query paths' to identify the object on which the query needs to be performed. A query path is a string value, and is one of the following:
        \list
        \o a Label
        \o a Signature
        \endlist
    \endlist

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

    \section3 Labels
    A \bold{Label} is a (usually static) text description that is associated with an editable field. In some cases, a widget is
    self-labelling (for example, the text property of a QPushButton), but often the label is seperate widget (usually a QLabel).
    The Label and Edit widgets form a "buddy pair". This relationship can be hardcoded in the application, but this is not
    mandatory.

    For labels that have no buddy widget set, QtUiTest resolves the buddy relationships based on relative positions of the
    label and edit widgets on screen. By default, for Left-to-Right languages, labels are expected to be to the left of their
    corresponding edit widgets (and to the right for Right-to-Left languages). However, this can be changed by using the
    setLabelOrientation() function.

    For example, the code below will enter the text "dog walker" in a text field next to a Label with text "Occupation".
    \code
        enter( "dog walker", "Occupation" );
    \endcode
    In words the example says: "enter the text 'dog walker' into a field labeled 'Occupation'.

    A special case of a Label is an empty string. This will resolve to the widget that currently has focus, or (for touchscreen
    devices) the first field in the dialog. Note that for tests that should run both on keyboard and touchscreen devices it
    may be risky to rely on this feature. Always explicitly specifying a Label is the most safe and portable thing to do.

    Using a Label to identify a field is the preferred mechanism for identifying fields. It is reliable, easy to write, easy
    to understand and easy to debug.

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

    \section3 Signatures
    In some cases it is not possible to use a label to uniquely identify a widget. In these cases it is necessary to use a
    \bold{Signature} to identify the widget.

    A Signature is a string that uniquely identifies an object. Signatures are generated by the system at run-time; the signature
    of a widget will be different each time a test is run, so it is not possible to hard-code a signature value in a test case.

    The signature() function can be used to return the signature of an object based on its position relative to another widget.
    \code
        enter( "dog walker", signature("Label",1));
    \endcode
    In this case, the signature() function returns the first editable field to the right of or below the field associated with label "Label".

    \section3 Querying Widgets By Property
    Widgets can also be specified using the findByProperty() function. This finds all widgets that match the specified property values, sorted by position (top left to bottom right). The values returned are returned as a QStringList of signatures. In addition to the properties supported by the widget, "className" and "inherits" are also supported. The "className" matches if the specified value matches the class name of the object; "inherits" matches if the class of the object inherits the specified class name.

    For example:
    \code
        var toolButtons = findByProperty( { className: "QToolButton" } );
        ...
        select (toolButtons[3]);
        var addNewPage = findByProperty( { tooltip: "Add new page" } );
    \endcode

    Multiple property value may be specified, for example:
    \code
        var password = findByProperty( { className: "QLineEdit", echoMode: 2 } );
    \endcode

    Using findByProperty requires some knowledge of the inner workings of the application under test, and as such breaks some of the principles of black-box testing. It is recommended that, where possible, logic using findByProperty should be wrapped in a utility function, which can be called by the test case.


\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

    \section2 Learn Mode

    Some testcases require the existence of non-volatile test data, such as screen snapshots. To simplify the process of data gathering, the test system supports the
    concept of a learn mode. When a test is run in learn mode, certain actions will result in the generation of test data, which will be presented to the user to manually
    verify once, then stored for use in subsequent test runs. A system test is run in learn mode by supplying the \c -learn or \c -learn-all command line options.

    The system test framework will store generated test data in the \c testdata subdirectory of the directory containing the test script.

    NOTE: It is up to the tester to ensure that new learned elements are added to a version control system.

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

    \section2 Application Management

    \list
    \o \l{QSystemTest::}{startApplication()} - start an application.
    \endlist

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

    \section2 File Management

    It is possible to access files on the System Under Test (even if this is another device) from within a system test.

    \list
    \o \l{QSystemTest::}{putFile()}, \l{QSystemTest::}{putData()} - copy data to the test system.
    \o \l{QSystemTest::}{deletePath()} - delete files from the test system.
    \o \l{QSystemTest::}{getDirectoryEntries()} - read the contents of a directory on the test system.
    \endlist

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

    \section2 Querying Objects

    The system test framework provides powerful methods of obtaining information on
    particular objects and widgets in the system.

    \list
    \o \l{QSystemTest::}{isVisible()}, \l{QSystemTest::}{isChecked()}, \l{QSystemTest::}{isEnabled()} - retrieve simple information from widgets.
    \o \l{QSystemTest::}{getText()}, \l{QSystemTest::}{getSelectedText()} - retrieve a textual representation of widgets.
    \o \l{QSystemTest::}{getProperty()}, \l{QSystemTest::}{setProperty()}, \l{QSystemTest::}{invokeMethod()} - access Qt properties, signals and slots on any QObject.
    \endlist

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

    \section2 Text Entry and Navigation

    The system test framework provides several ways of entering text into widgets and for navigating through applications.

    \list
    \o \l{QSystemTest::}{enter()} - navigate to a widget and enter any text.
    \o \l{QSystemTest::}{select()} - choose a particular item from a list, combo box, menu or navigate to a particular widget or tab.
    \endlist

    These functions are input method independant and will have the same syntax for both Keyboard and Mouse/Touchscreen based devices and are therefore the preferred commands to use in tests.

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

    \section2 Key Simulation

    The system test framework provides several ways of generating arbitrary key events.

    \list
    \o \l{QSystemTest::}{keyPress()}, \l{QSystemTest::}{keyRelease()}, \l{QSystemTest::}{keyRelease()}, \l{QSystemTest::}{keyClickHold()} - press/release/click/hold any key.
    \endlist

    Note that Key simulation should be used in exceptional cases only since it makes the test specific for keyboard based devices. Other functions such as select() are the preferred mechanism to write tests because these will work both for Keyboard and Mouse/Touchscreen based devices.

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

    \section2 Mouse / Touchscreen Simulation

    The system test framework provides methods to simulate input from a mouse or touchscreen.

    \list
    \o \l{QSystemTest::}{mousePress()}, \l{QSystemTest::}{mouseRelease()}, \l{QSystemTest::}{mouseClick()}, \l{QSystemTest::}{mouseClickHold()} - press/release/click with mouse or touchscreen.
    \endlist

    Note that Mouse simulation should be used in exceptional cases only since it makes the test specific for mouse/touchscreen based devices. Other functions such as select() are the preferred mechanism to write tests because these will work both for Keyboard and Mouse/Touchscreen based devices.

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

    \section2 Other

    Some functionality does not fall into the aforementioned categories:

    \list
        \o \l{QSystemTest::}{setSetting()}, \l{QSystemTest::}{getSetting()} - access any settings stored within QSettings objects.
    \endlist

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

    \section1 Extending QtUiTest for Custom Widgets

QtUiTest's high-level commands require knowledge of the widgets they interact with.  For example, \l{QSystemTest::}{select()} must know how to select items from both a combobox and a list view.

When testing an application which uses custom widgets, some additional steps may be required in order to use high-level QtUiTest commands.  The logic for interacting with custom widgets must be implemented either in the application under test or in a plugin.

See \l{Creating a QtUiTest Test Widget} for more details.

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

    \section1 QtUiTest API Reference

For a complete description of the QtUiTest API please refer to \l{QSystemTest}{QSystemTest API}.

\l{QtUiTest Manual#introduction}{Back to QtUiTest Manual}

\endif

*/