=========================================================================
OVERVIEW

This repository contains BM2 - a system for storing and analyzing
performance testing results.

The idea is to run performance tests on test machines and upload the
results to a database on a server machine. The database can then be
inspected either from a web application or using SQL queries directly.


=========================================================================
INSTALLATION

STEP 1: Install the BM2 directory (containing this README!) under an
    arbitrary directory <BM2 dir>.


STEP 2: Install Python 2.6 (with development headers) on both the
    server machine and the machine from which results will be uploaded
    (NOTE: the latter may or may not be identical to the test
    machine).

    http://www.python.org/download/releases/2.6.7/

    NOTE: Version 2.7 or later may not work due to potential source
    incompatibilies!


STEP 3: Install a recent version of PostgreSQL (with development
    headers) (8.4 or later should work fine).

    http://www.postgresql.org/

    NOTE: Ensure to install both the server (postgres) and interactive
    terminal (psql).


STEP 4: Start the database server and (if desirable) arrange for it be
    started automatically at boot time (on a Linux system this would
    typically mean to add some stuff under /etc/init.d/).


STEP 5: Initialize the database:

    $ <BM2 dir>/database/scripts/reset.py


STEP 6: Install a recent version the Apache web server (2.2 or later
    should do fine).

    http://httpd.apache.org/


STEP 7: Detect and install missing Python modules:

    $ cd <BM2 dir>/scripts
    $ ./getstats.py

    A missing module will reported as:

        ...
        ImportError: No module named <name>

    You are likely to be missing the following modules:

        * statlib (statistics)
              http://code.google.com/p/python-statlib

        * psycopg2 (PostgreSQL access)
              http://initd.org/psycopg

    It should be straightforward to install both of them by following
    the instructions on the respective web pages.

    NOTE: Installing psycopg2 fails if you don't have the appropriate
    PostgreSQL client library (libpq) installed.


STEP 8: Install a script named 'getstatswrapper' for executing
    getstats.py with the appropriate environment. The script
    typically looks like this on Linux (<BM2 dir> is
    /home/qt/repos/bm2 in this example):

    #!/bin/sh
    LD_LIBRARY_PATH=/usr/local/pgsql/lib \
    /home/qt/repos/bm2/scripts/getstats.py


STEP 9: Configure the Apache web server to 1) allow access to the
    <BM2 dir>/web/ directory, and 2) to allow launching
    'getstatswrapper' as a CGI script (see separate documentation
    for the web server - typically this requires changes to
    httpd.conf etc.).


STEP 10: Start the web server and (if desirable) arrange for it be
    started automatically at boot time (on a Linux system this would
    typically mean to add some stuff under /etc/init.d/).


=========================================================================
USAGE

--- Producing and uploading results -------------------------------

STEP 1: Run tests and record performance measurements in an XML file:

    <TestCase name="MyTestCase">
        ...
        <TestFunction name="myTestFunction">
            ...
            <BenchmarkResult metric="WalltimeMilliseconds" tag="my tag" value="98" iterations="4" />
            ...
        </TestFunction>
        ...
    </TestCase>

    Notes:
      - This is the same format as is generated by a QTestLib program started with the -xml
        command-line arguument.
      - There is a hierarchichal relationship between test cases, test functions, and data tags:
        A test case consists of one or more test functions, which in turn consists of one or more
        data tags. Each such combination is considered a "benchmark" and is associated with a
        single measurement ("benchmark result").
      - The 'tag' attribute may be empty.
      - The 'iterations' attribute is optional with a default value of 1.
      - The actual value is value / iterations (in the example above it would be 98 / 4 = 24.5).


STEP 2: Upload results to the server using the upload script:

    Typically something like this:

    $ ./uploadresults.py --dbhost ... --dbport ... --db bm --host oslo-bm-mac-mini-64-1 \
      --platform macx-g++ --branch qt/qt-script-v8#master-v8 \
      --sha1 ae15094c6818904b47a53590644992f24e7faa97 --file results.xml

    Run './uploadresults.py --help' for a description of the command-line options!

    Notes:
      - Results for different SHA-1s (revisions) must be uploaded in the order in
        which they appear the revision history.
      - Results may be uploaded multiple times for the same SHA-1 (this can be useful
        to remove outliers since BM2 uses the median value in such a case).



--- Viewing results ------------------------------------

--- Option 1: Using web application --------------------------

STEP 1: Open the URL

    Open the top-level index.html page, located under <BM2 dir>/web/, for example:

        http://bmc.test.qt.nokia.com:8080/bm2

    where bmc.test.qt.nokia.com is the host address, 8080 the Apache port number,
    and bm2 a symbolic link to <BM2 dir>/web/.

    Obviously, this may be configured differently on different installations.


STEP 2: Interact with the web app to show various reports.

    <Examples to be filled in>


--- Option 2: Query the database directly --------------------------

<Examples to be filled in>