path: root/doc/testcases.txt

/****************************************************************************
**
** Copyright (C) 2010 Nokia Corporation and/or its subsidiary(-ies).
** All rights reserved.
** Contact: Nokia Corporation (qt-info@nokia.com)
**
** This file is part of the documentation of the Qt Toolkit.
**
** $QT_BEGIN_LICENSE:FDL$
** No Commercial Usage
** This file contains pre-release code and may not be distributed.
** You may use this file in accordance with the terms and conditions
** contained in the Technology Preview License Agreement accompanying
** this package.
**
** GNU Free Documentation License
** Alternatively, this file may be used under the terms of the GNU Free
** Documentation License version 1.3 as published by the Free Software
** Foundation and appearing in the file included in the packaging of this
** file.
**
** If you have questions regarding the use of this file, please contact
** Nokia at qt-info@nokia.com.
** $QT_END_LICENSE$
**
****************************************************************************/

Organization of test cases
==========================

The test cases are launched by a C++ harness that consists of
the following code:

    #include <QtQuickTest/quicktest.h>
    QUICK_TEST_MAIN(qmlexample)

Where "qmlexample" is an identifier to use to uniquely identify
this set of tests.  You should add "CONFIG += qmltestcase" to your
*.pro file; for example:

    TEMPLATE = app
    TARGET = tst_qmlexample
    CONFIG += warn_on qmltestcase
    SOURCES += tst_qmlexample.cpp
    RESOURCES += qmlexample.qrc

The test harness scans recursively for "tst_*.qml" files in the qrc
resources that are bound into the test harness binary.  The following
is an example .qrc file:

    <!DOCTYPE RCC><RCC version="1.0">
    <qresource>
        <file>tst_basic.qml</file>
        <file>tst_item.qml</file>
    </qresource>
    </RCC>

The QUICK_TEST_SOURCE_DIR macro can be defined at compile time to
run tests from plain files without binding them into resources.
Modify your .pro file to include the following line:

    DEFINES += QUICK_TEST_SOURCE_DIR=\"\\\"$$PWD\\\"\"

The QUICK_TEST_SOURCE_DIR environment variable can also be set
at runtime to run test cases from a different directory.  This may
be needed to run tests on a target device where the compiled-in
directory name refers to a host.

Other *.qml files may appear for auxillary QML components that are
used by the test.

See "tests/qmlexample" for an example of creating a test harness
that uses resources and "tests/qmlauto" for an example that uses
the QUICK_TEST_SOURCE_DIR macro.

Basic test cases
================

Test cases are written as JavaScript functions within a "TestCase" element:

----------------------
import QtQuick 1.0
import QtQuickTest 1.0

TestCase {
    name: "MathTests"

    function test_math() {
        compare(2 + 2, 4, "2 + 2 = 4")
    }

    function test_fail() {
        compare(2 + 2, 5, "2 + 2 = 5")
    }
}
----------------------

Functions that start with "test_" are treated as test cases to be
executed.  The "name" is used to prefix the functions in the output:

********* Start testing of MathTests *********
FAIL!  : MathTests::test_fail() 2 + 2 = 5: actual: 4, expected: 5
PASS   : MathTests::test_math()
Totals: 1 passed, 1 failed, 0 skipped
********* Finished testing of MathTests *********

Because of the way JavaScript properties work, the order in which the
test functions are found is unpredictable.  To assist with predictability,
the test framework will sort the functions on ascending order of name.
This can help when there are two tests that must be run in order.

A number of helper functions are available to assist with writing tests:

    fail(msg)
        - Fail the current test and show "msg".
        - Similar to QFAIL in C++

    verify(cond, msg)
        - Verify that "cond" is true, fail with "msg" if not.
        - Similar to QVERIFY in C++

    compare(actual, expected, msg)
        - Compare "actual" with "expected", fail with "msg" if not.
        - Similar to QCOMPARE in C++

    tryCompare(obj, prop, value[, timeout])
        - Tries comparing the property called "prop" on "obj" with "value".
          If the compare fails, wait for up to "timeout" milliseconds,
          processing Qt events, until the property becomes "value".
        - The default timeout is 5000 milliseconds.
        - e.g. tryCompare(img, "status", BorderImage.Ready)

    skip(msg)                   ["Single" mode]
    skipAll(msg)
        - Skip the current test and show "msg".
        - Similar to QSKIP in C++

    expectFail(msg)             ["Abort" mode]
    expectFailContinue(msg)
        - Mark the current test as expected to fail, with "msg" on failure.

    warn(msg)
        - Similar to QWARN in C++

    ignoreWarning(msg)
        - Ignore warning message
        - Similar to QTest::ignoreMessage(QtWarningMsg, msg)

    wait(ms)
        - Wait for "ms" milliseconds, processing Qt events.

    sleep(ms)
        - Sleep for "ms" milliseconds, but do not process events.

The "msg" parameters can be omitted if there is no particular message
that should be displayed other than "FAIL" or "SKIP".

Special functions
=================

The following special function names have the same meaning as in C++:

    initTestCase()
    cleanupTestCase()
    init()
    cleanup()

Note: The initTestCase_data() function from C++ is not supported.

Data-driven tests
=================

Table data can be provided to a test using a function name that ends
with "_data":

----------------------
import QtQuick 1.0
import QtQuickTest 1.0

TestCase {
    name: "DataTests"

    function test_table_data() {
        return [
            {tag: "2 + 2 = 4", a: 2, b: 2, answer: 4 },
            {tag: "2 + 6 = 8", a: 2, b: 6, answer: 8 },
        ]
    }

    function test_table(data) {
        compare(data.a + data.b, data.answer)
    }
}
----------------------

The test framework will iterate over all of the rows in the table
and pass each row to the test function.  As shown, the columns can be
extracted for use in the test.  The "tag" column is special - it is printed
by the test framework when a row fails, to help the reader identify which
case failed amongst a set of otherwise passing tests.

Asynchronous testing
====================

The easiest method for asynchronous testing is to use tryCompare() to
wait for a property to transition to an expected value.  For example:

    tryCompare(img, "status", BorderImage.Ready)
    compare(img.width, 120)
    compare(img.height, 120)
    compare(img.horizontalTileMode, BorderImage.Stretch)
    compare(img.verticalTileMode, BorderImage.Stretch)

The "when" property can be used to cause a "TestCase" element to trigger
only when a certain condition is true.  For example, the following example
runs a test when the user presses the mouse button:

----------------------
import QtQuick 1.0
import QtQuickTest 1.0

Rectangle {
    id: foo
    width: 640; height: 480
    color: "cyan"

    MouseArea {
        id: area
        anchors.fill: parent
    }

    property bool bar: true

    TestCase {
        name: "ItemTests"
        when: area.pressed
        id: test1

        function test_bar() {
            verify(bar)
        }
    }
}
----------------------

Multiple "TestCase" elements can be supplied.  The test program will exit
once they have all completed.  If a test case doesn't need to run (because
a precondition has failed), then "optional" can be set to true:

----------------------
TestCase {
    when: false
    optional: true

    function test_not_run() {
        verify(false)
    }
}
----------------------

The special property "windowShown" can be used to trigger a test to run
just after the main window has been shown on-screen:

----------------------
Button {
    id: button
    onClicked: text = "Clicked"
    TestCase {
        name: "ClickTest"
        when: windowShown

        function test_click() {
            button.clicked();
            compare(button.text, "Clicked");
        }
    }
}
----------------------

Benchmarks
==========

If the test function name starts with "benchmark_", then it will be
run multiple times with the Qt benchmark framework, with a average
timing value reported for the runs.  This is equivalent to using the
QBENCHMARK macro in QTestLib.

----------------------
TestCase {
    id: top
    name: "CreateBenchmark"

    function benchmark_create_component() {
        var component = Qt.createComponent("item.qml")
        var obj = component.createObject(top)
        obj.destroy()
        component.destroy()
    }
}
----------------------

RESULT : CreateBenchmark::benchmark_create_component:
     0.23 msecs per iteration (total: 60, iterations: 256)
PASS   : CreateBenchmark::benchmark_create_component()

To get the effect of the QBENCHMARK_ONCE macro, prefix the test
function name with "benchmark_once_".