path: root/doc/src/qtuitest_tutorial.qdoc

/*!
    \page qtuitest-tutorial.html
    \contentspage QtUiTest Manual
    \nextpage {Chapter 1: Writing a System Test}{Chapter 1}

    \title QtUiTest Tutorial

    This tutorial gives a short introduction on the creation of System Tests using the QtUiTest framework. It is divided into the following chapters:

    \list 1
    \o \l {Chapter 1: Writing a System Test}{Writing a System Test}
    \o \l {Chapter 2: Input, Output and Widget Navigation}{Input, Output and Widget Navigation}
    \o \l {Chapter 3: Data Driven Testing}{Data Driven Testing}
    \o \l {Chapter 4: Putting It All Together}{Putting It All Together}
    \endlist
*/

/*!
    \page qtuitest-tutorial1.html
    \contentspage {QtUiTest Tutorial}{Contents}
    \nextpage {Chapter 2: Input, Output and Widget Navigation}{Chapter 2}

    \title Chapter 1: Writing a System Test

    In this first chapter we will demonstrate how to write a simple system test, and how to execute it.

    \section1 Setting up the environment

    Tests are saved in directories, one directory per test. As a convention we keep the name of the directory the same as the test name although this isn't a requirement. Tests are further stored in a 'tests' directory immediately under the application or library that is being tested. So let's create our first test directory, and we'll be writing a test for the addressbook application.

\code
    cd src/applications/addressbook
    mkdir -p tests/sys_demo
\endcode

    Note that we write and save tests in the \c source tree, and we can execute them from the source as well as the build tree.

    \section1 Writing the Test

    New system tests are defined by creating a new QtScript file (named ending with .js by convention), containing a \c{testcase} object with one or more test functions:
    So cd into \c tests/my_demo and create a file named \c sys_demo.js with the following content:
    \code
        testcase = {
            testFunction1: function() {
                print( "Hello World" );
            },
            testFunction2: function() {
                print( "Testfunction 2" );
            }
        }
    \endcode

    The example above shows the creation of a testcase object containing two test functions, \c testFunction1 and \c testFunction2. Obviously not much exciting is going to happen when we run this test. But we'll get to that a little later.

    The second bit that needs to be done is tell the build system about our new project. For this we need to create a project file named \c qbuild.pro. The contents of the file is as follows:
    \code
        CONFIG+=systemtest
        TARGET=sys_demo
        SOURCES+=sys_demo.js
    \endcode

    \section2 Running the test for the first time
    We can simply run the test as follows:
    \code
        cd <your_src_location>/src/applications/addressbook/tests/sys_demo
        make test
    \endcode

    The test will be built and executed and results in something like this:
    \code
1 echo '#!/bin/sh' > <your_build_dir>/src/applications/addressbook/tests/sys_demo/sys_demo
2 echo 'exec <your_build_dir>/bin/qtuitestrunner <your_src_location>/src/applications/addressbook/tests/sys_demo/sys_demo.js "$@"' >> <your_build_dir>/src/applications/addressbook/tests/sys_demo/sys_demo
3 chmod 755 <your_build_dir>/src/applications/addressbook/tests/sys_demo/sys_demo
4 export QPEHOME=$(mktemp -d /tmp/qtopia_maketest_XXXXXX); <your_build_dir>/src/applications/addressbook/tests/sys_demo/sys_demo $ARGS; rm -rf $QPEHOME
5 ********* Start testing of sys_demo *********
6 Config: Using QTest library 4.4.2, Qt 4.4.2
7 PASS   : sys_demo::initTestCase()
8 QDEBUG : sys_demo::testFunction1() Hello World
9 PASS   : sys_demo::testFunction1()
10 QDEBUG : sys_demo::testFunction2() Testfunction 2
11 PASS   : sys_demo::testFunction2()
12 PASS   : sys_demo::cleanupTestCase()
13 Totals: 4 passed, 0 failed, 0 skipped
14 ********* Finished testing of sys_demo *********
\endcode

In the above snippet line numbers have been added for clarity. Lines 1-4 are a bit of magic to create a subdirectory for the test in the build tree, and to create an 'executable' for the test. Once the executable is created, it is executed and the results are printed to stdout (or to a file if we use extra command line parameters).

Line 5-14 are the result of the test being execute.
Line 5 simply marks the start of the test.
Line 6 shows the version numbers of Qt and Qtest that are used.
Line 7 is the result of a special function named \c initTestCase. Note that we haven't written an initTestCase yet, so this function is empty (and shouldn't fail).
Once the test is initialized (with initTestCase) the testrunner will start executing all testfunctions, in principle in the order in which they are discovered in the file.
Line 8 is the result of the print statement we did in testFunction1. Note also that the output clearly marks this information as being a part of \c sys_demo::testFunction1.
Line 9 is the final test result for testFunction1. Since we did nothing else but a print statement it PASSes the test.
Line 10 and 11 show the results for testFunction2.
Line 12 shows the result for the counterpart to initTestCase: after all testfunctions have been executed the system takes a moment to clean up any garbage that has been created during test execution. Just like it's init counterpart is cleanupTestCase \c always called, and called only once, at the end of a test. Since we haven't actually defined anything yet the cleanup is empty and we'd expect it to PASS.
Line 13 shows the accumulated totals for the test and
Line 14 finally marks the end of the test.

In this example we have called \c {qbuild test} from the source tree, but we could have done the same, and with the same result from the build tree. This is because we're calling qbuild which knows everything about the actual configuration we're testing, as well as where to find it.

    \section2 Special functions

    As with \l{QTestLib Manual}{QTestLib} unit tests, special reserved functions include the four \l{QTestLib Manual#creating-a-test}{init/cleanup functions}. These behave identically to their unit test counterparts but will be explained here for completeness.

    \table 80%
    \header \o function \o description
    \row \o initTestCase \o called once immediately after the test is started. If initTestCase fails, which could happen if you add verification steps in the function, no test will be executed. In normal circumstances, each test will be executed after initTestCase is finished.
    \row \o init \o called immediately \c before a test function is executed. If a test function has multiple test data sets (to be discussed later) then \c init() will also be called multiple times. When init() fails, the test function will be skipped.
    \row \o cleanup \o called immediately after a test function has finished executing. cleanup() is guaranteed to be called, even if the test function has failed, i.e. you still get a chance to cleanup the environment after a failure.
    \row \o cleanupTestCase \o called once after the last test function has been executed.
    \endtable

    Let's re-write our testcase a bit, but this time we use the special functions.

    \code
        testcase = {
            initTestCase: function() {
                print( "Init complete test" );
            },
            cleanupTestCase: function() {
                print( "Cleanup complete test" );
            },
            init: function() {
                print( "Init test function" );
            },
            cleanup: function() {
                print( "Cleanup test function" );
            },
            testFunction1: function() {
                print( "Hello World" );
            },
            testFunction2: function() {
                print( "Testfunction 2" );
            }
        }
    \endcode

    When we run \c {qbuild test} again we get the following output:
    \code
********* Start testing of sys_demo *********
Config: Using QTest library 4.4.2, Qt 4.4.2
QDEBUG : sys_demo::initTestCase() Init complete test
PASS   : sys_demo::initTestCase()
QDEBUG : sys_demo::testFunction1() Init test function
QDEBUG : sys_demo::testFunction1() Hello World
QDEBUG : sys_demo::testFunction1() Cleanup test function
PASS   : sys_demo::testFunction1()
QDEBUG : sys_demo::testFunction2() Init test function
QDEBUG : sys_demo::testFunction2() Testfunction 2
QDEBUG : sys_demo::testFunction2() Cleanup test function
PASS   : sys_demo::testFunction2()
QDEBUG : sys_demo::cleanupTestCase() Cleanup complete test
PASS   : sys_demo::cleanupTestCase()
Totals: 4 passed, 0 failed, 0 skipped
********* Finished testing of sys_demo *********
    \endcode

    \section2 Interacting with the System Under Test

    System tests do not have direct access to the code under test. Instead, the system testrunner connects to the application, and sends and receives information via a communication protocol. This effectively controls access to the tested system, reducing the impact of tests on results and emulating standard user behaviour.

    Suppose we wish to create a simple test which launches an application; we first need to ensure that Qt Extended has finished loading successfully, since we can't be
    sure of the state of the system we have connected to. This is accomplished by using the \l{QtopiaSystemTest::}{waitForQtopiaStart()} function in
    \l{QTestLib Manual#creating-a-test}{initTestCase()}:

    \code
        initTestCase: function()
        {
            waitForQtopiaStart();
        }
    \endcode


    Since \c initTestCase() is called before any test functions are executed, this will pause test execution until it receives confirmation that Qt Extended has started. Now we're ready to start the application:

    \code
        testFunction1: function()
        {
            startApplication( "Contacts" );
        }
    \endcode

    The \l{QSystemTest::}{startApplication()} method will attempt to start the specified application (in this case, Contacts).  If the specified application cannot be started, \c{testFunction} will generate a test failure and abort the test.

    Once the application has started, we can begin testing. While it is possible to execute as many actions in a test function as desired, the recommendation is to dedicate each test function to a single use case. The test function should then attempt to realistically simulate each step of the use case, and verify that the expected result occurs. For example, assume we need to test creating a new contact; a simple test function could look like this:

    \code
        creating_a_contact: function()
        {
            // Start the application
            startApplication( "Contacts" );

            // Open the options menu and choose "New contact"
            select( "New contact", optionsMenu() );

            // Enter some details in the "Name" and "Emails" fields
            enter( "Frank Grimes", "Name" );
            enter( "frank@example.com", "Emails" );

            // Select 'Back' from the softkey menu to commit changes
            select( "Back", softMenu() );

            // We should now be at the contacts list.
            // Verify that we can select the contact we just created.
            select( "Frank Grimes" );

            // We should now be viewing the contact.
            // Move to "Details" tab.
            select( "Details", tabBar() );

            // Now verify that the details screen contains
            // the expected details.
            var text = getText();
            verify( text.contains("Frank Grimes") );
            verify( text.contains("frank@example.com") );
        }
    \endcode

    This test function will start Contacts, navigate through several screens, input data, and verify that the data was successfully saved as a new contact.
    It is important to note that this test has not explicitly simulated any key or mouse events (although this can be done).  This means that the same test can be used as-is on both a keyboard and touchscreen device.  This is possible because the implementation of select() and enter() know how to select items and enter data on both keyboard and touchscreen devices.

    The next chapter gives more detail on how navigation works.
*/

/*!
    \page qtuitest-tutorial2.html
    \previouspage {Chapter 1: Writing a System Test}{Chapter 1}
    \contentspage {QtUiTest Tutorial}{Contents}
    \nextpage {Chapter 3: Data Driven Testing}{Chapter 3}

    \title Chapter 2: Input, Output and Widget Navigation

    System tests generally consist of a series of actions and verification that the expected behaviour took place.  When testing a Qt Extended application, this usually takes the form of simulating the input of specific text and verifying that the application subsequently displays the correct text.  Testing necessarily includes navigating through forms and menus, activating buttons and comboboxes, and similar tasks.  QtUiTest makes this very easy by supplying simple yet powerful navigation and input/output methods.

    \section1 Basic Input

    Using QtUitest, it is possible to simulate individual keyclicks, perhaps the most simple form of user interaction that can be simulated.

    \code
    mytestcase = {
        mytestfunction: function() {
            keyClick(Qt.Key_A);
            keyClick(Qt.Key_B);
            keyClick(Qt.Key_C);
        }
    }
    \endcode

    In the above example, the calls to \l{QSystemTest::}{keyClick()} simulate a few individual key clicks, exactly as if the user had physically pressed keys on the device.

    However, there are several disadvantages to this approach.  Firstly, it is extremely verbose.  Secondly, it means that the test will only work on a keypad device.  To avoid these problems, the \l{QSystemTest::}{enter()} function is provided:

    \code
    mytestcase = {
        mytestfunction: function() {
            enter("abc");
        }
    }
    \endcode

    On a keypad device, the above example has the same affect as the previous example, but is more concise.  However, this test now has the additional advantage that it can work on a touchscreen device, as \l{QSystemTest::}{enter()} knows how to simulate the necessary touchscreen events to input the given text.

    \section1 Input Into Specific Widgets

    In both of the examples above, input would be delivered to whichever widget is currently focused.  In practice, what we often wish to do is enter text into a series of widgets displayed to the user.

    For example, consider the following screen for entering the details of a new contact.

    \image fields_example.png

    It would be possible to enter text in each field on this screen by using \c {enter()} and \c{keyClick(Qt.Key_Down)}.  However, the test would easily break if the order of fields were changed, or if fields were added or removed, and the test would not work on a touchscreen device.

    To solve this problem, \c {enter()} (and many other functions in QtUiTest) take an optional \i{Query Path} parameter, which specifies which widget to operate on.  The most common usage of query paths takes the form "LabelText", which refers to any widget which can receive input and is named or labeled "LabelText".

    Entering text into a few fields on the screen featured above is achieved by the following example:

    \code
    mytestfunction: function() {
        enter("Yoyo Corporation", "Company");
        enter("Yoyo Engineer", "Title");
        enter("x51 YOYO", "Phone");
    }
    \endcode

    In the above example, if any of the specified fields are moved, the test will continue to work as expected.  If any of the specified fields are removed, renamed, or no longer visible, the test will fail with a message indicating it cannot find the specified field.

    \section1 Selecting Items from Lists and Menus, and Activating Buttons

    Often, a user will be presented with a list of items or a context menu.  QtUiTest provides a simple way to navigate to and select items in any list- or menu-like widget.  The \l{QSystemTest::}{select()} function will select an item from a list, combo box, menu or tab widget by looking at the display text for each item.  It can also be used to activate a button with the given text.

    \code
    mytestfunction: function() {
        // Select "Show contacts" from the options (context) menu.
        select("Show contacts", optionsMenu());

        // Select "Bob" in the currently shown list.
        select("Bob");

        // Activate the "Edit" button to edit Bob's details.
        select("Edit");

        // Fill in the "Gender" field (a combo box) for Bob.
        select("Male", "Gender");
    }
    \endcode

    select() allows a test to be reused by keypad and touchscreen devices.  For instance, consider select("Edit") from the above example.  On a touchscreen device, this will simply result in a click being simulated at the co-ordinates of the Edit button.  On a keypad device, keyclicks will be simulated to navigate to and activate the Edit button.

    \section1 Output

    Built-in Qt and Qt Extended widgets can be queried for their currently displayed text.  The \l{QSystemTest::}{getText()} function is the most common way of doing this.

    \code
    mytestfunction: function() {
        enter("Yoyo Corporation", "Company");
        compare( getText("Company"), "Yoyo Corporation" );
    }
    \endcode

    The above example simply enters "Yoyo Corporation" into the "Company" field, then verifies that the field actually contains the text "Yoyo Corporation". Note that it is not necessary to explicitly call compare immediately after the enter command: it would have failed if the enter text had failed.

    The next chapter explains how to make use of data driven testing.
*/

/*!
    \page qtuitest-tutorial3.html
    \previouspage {Chapter 2: Input, Output and Widget Navigation}{Chapter 2}
    \contentspage {QtUiTest Tutorial}{Contents}
    \nextpage {Chapter 4: Putting It All Together}{Chapter 4}

    \title Chapter 3: Data driven testing

    The first page of this tutorial gave an example of testing creation of a contact:

    \code
        creating_a_contact: function()
        {
            // Start the application
            startApplication( "Contacts" );

            // Open the options menu and choose "New contact"
            select( "New contact", optionsMenu() );

            // Enter some details in the "Name" and "Emails" fields
            enter( "Frank Grimes", "Name" );
            enter( "frank@example.com", "Emails" );

            // Select 'Back' from the softkey menu to commit changes
            select( "Back", softMenu() );

            // We should now be at the contacts list.
            // Verify that we can select the contact we just created.
            select( "Frank Grimes" );

            // We should now be viewing the contact.
            // Move to "Details" tab.
            select( "Details", tabBar() );

            // Now verify that the details screen contains
            // the expected details.
            var text = getText();
            verify( text.contains("Frank Grimes") );
            verify( text.contains("frank@example.com") );
        }
    \endcode

    One problem with this test function is that only one set of values is tested.
    If we want to test additional values, the best way to do it is with QtUiTest's
    built-in support for data driven testing.

    We create a new object, named the same as our testfunction with '_data' appended,
    and we make the object consist of a series of arrays:

    \code
        creating_a_contact_data: {
            simple:          [ "Frank Grimes",        "frank@example.com", "+61 0321 FRANK" ],
            no_email:        [ "Bob Jones",           undefined,          "+61 0321 BOB"   ],
            with_middlename: [ "Jane Middlename Doe", "jmd@example.com",      undefined        ]
        }
    \endcode

    \c{testcase.creating_a_contact_data} is a table of test data with rows \c{simple}, \c{no_email} and \c{with_middlename}.  Each row has 3 columns, and these are passed to the \c{creating_a_contact} test function as the arguments \c{name}, \c{email}, \c{homephone} in the example below:

    \code
        creating_a_contact: function(name, email, homephone)
        {
            // Start the application
            startApplication( "Contacts" );

            // Open the options menu and choose "New contact"
            select( "New contact", optionsMenu() );

            // Enter details
            enter( name, "Name" );
            enter( email, "Emails" );
            enter( homephone, "Home Phone" );

            // Select 'Back' from the softkey menu to commit changes
            select( "Back", softMenu() );

            // We should now be at the contacts list.
            // Verify that we can select the contact we just created.
            select( name );

            // We should now be viewing the contact.
            // Move to "Details" tab.
            select( "Details", tabBar() );

            // Now verify that the details screen contains
            // the expected details.
            var text = getText();
            if (name != undefined)      verify( text.contains(name) );
            if (email != undefined)     verify( text.contains(email) );
            if (homephone != undefined) verify( text.contains(homephone) );
        }
    \endcode

    This test is now much more extensible. New test cases can simply be added as new rows to the testdata table and will automatically be tested without needing to add further logic to the test function.
*/

/*!
    \page qtuitest-tutorial4.html
    \previouspage {Chapter 3: Data Driven Testing}{Chapter 3}
    \contentspage {QtUiTest Tutorial}{Contents}

    \title Chapter 4: Putting It All Together

    Using the simple building blocks of textual input and output and navigation functions, with the data driven testing approach, a full testcase can be written for creating a new contact.  Note that the functions for creating and verifying a contact have been moved out into helper functions; this allows them to be reused for subsequent tests.  This is very useful for cases where, for instance, one test's precondition might be that a contact has been successfully created.

    \code
    testcase = {
        initTestCase: function() {
            // Start "Contacts" when the testcase starts.
            startApplication( "Contacts" );
        },

        creating_a_contact: function(name, emails, company, jobTitle, businessPhone) {
            create_contact(name, emails, company, jobTitle, businessPhone);
            verify_contact(name, emails, company, jobTitle, businessPhone);
        },

        creating_a_contact_data: {
            simple:           [ "Billy Jones",      "billy@example.com",   "Hotdog Inc.",     "Hotdog Engineer",    "12345"   ],
            letters_in_phone: [ "Joan Example",     "joan@example.com",    "Example Inc.",    "Exemplary Engineer", "555 EXA" ],
            three_names:      [ "Jon John Johnson", "jjj@example.com",     "Sillynames Inc.", "Dog Walker",         "12345"   ],
            no_job:           [ "William Doe",      "bill@example.net",     undefined,         undefined,            undefined ]
        }
    }

    // Create a contact with the given details.
    function create_contact(name, emails, company, jobTitle, businessPhone) {
        // Verify that we're on the home screen of the "Contacts" app.
        waitForTitle( "Contacts" );

        // Select "New contact" from context menu.
        select( "New contact", optionsMenu() );

        // Navigate to the "Contact" tab.
        // This is the default tab, but with this code we ensure the
        // test will work if it becomes no longer the default.
        select( "Contact", tabBar() );

        // Fill in fields on the "Contact" tab.
        // enter() returns immediately if we try to enter an undefined
        // value.
        enter( name,   "Name" );
        enter( emails, "Emails" );

        // Navigate to the "Business" tab.
        select( "Business", tabBar() );

        // Fill in fields on the "Business" tab.
        enter( company,       "Company" );
        enter( jobTitle,      "Title" );
        enter( businessPhone, "Phone" );

        // Press the Back key to commit the new contact
        select( "Back", softMenu() );
    }

    // Verify that a contact with the given details exists.
    function verify_contact(name, emails, company, jobTitle, businessPhone) {
        // Verify that we're on the contacts list.
        waitForTitle( "Contacts" );

        // Select the contact with the given name.
        select( name );

        // Navigate to the "Details" tab and get the displayed text.
        select( "Details", tabBar() );
        var details = getText();

        // Now verify that the details contains all of the non-null information
        // for the contact.
        if (name != undefined)          verify( details.contains(name) );
        if (emails != undefined)        verify( details.contains(emails) );
        if (company != undefined)       verify( details.contains(company) );
        if (jobTitle != undefined)      verify( details.contains(jobTitle) );
        if (businessPhone != undefined) verify( details.contains(businessPhone) );
    }
    \endcode

    The above test has been written to be reasonably permissive; it will succeed as long as the text shown by the contacts application contains all of the information for the created contact, and it does not test things such as the formatting of the given text, and does not test every single field.  However, this test is well insulated against minor changes to the tested application GUI.

    QtUiTest allows the tester to decide how strict a test should be.  If pixel-perfect accuracy is required for output, a test can be written to test every screen with \l{QSystemTest::}{verifyImage()}.  In contrast, a high-level text-based approach as shown above can result in effective tests which remain correct even when significant UI changes occur.

    There are many other methods available for use; for further information, refer to the \l{QSystemTest} documentation.
 */