diff options
Diffstat (limited to 'README')
| -rw-r--r-- | README | 173 |
1 files changed, 171 insertions, 2 deletions
@@ -1,3 +1,172 @@ -Fill in some contents here at some point. -For now, please refer to http://wikis.in.nokia.com/QtSoftware/BM2 +========================================================================= +OVERVIEW +This repository contains BM2 - a system for storing and analyzing +performance testing results. + +The idea is to run performance tests on test machines and upload the +results to a database on a server machine. The database can then be +inspected either from a web application or using SQL queries directly. + + +========================================================================= +INSTALLATION + +STEP 1: Install the BM2 directory (containing this README!) under an + arbitrary directory <BM2 dir>. + + +STEP 2: Install Python 2.6 (with development headers) on both the + server machine and the machine from which results will be uploaded + (NOTE: the latter may or may not be identical to the test + machine). + + http://www.python.org/download/releases/2.6.7/ + + NOTE: Version 2.7 or later may not work due to potential source + incompatibilies! + + +STEP 3: Install a recent version of PostgreSQL (with development + headers) (8.4 or later should work fine). + + http://www.postgresql.org/ + + NOTE: Ensure to install both the server (postgres) and interactive + terminal (psql). + + +STEP 4: Start the database server and (if desirable) arrange for it be + started automatically at boot time (on a Linux system this would + typically mean to add some stuff under /etc/init.d/). + + +STEP 5: Initialize the database: + + $ <BM2 dir>/database/scripts/reset.py + + +STEP 6: Install a recent version the Apache web server (2.2 or later + should do fine). + + http://httpd.apache.org/ + + +STEP 7: Detect and install missing Python modules: + + $ cd <BM2 dir>/scripts + $ ./getstats.py + + A missing module will reported as: + + ... + ImportError: No module named <name> + + You are likely to be missing the following modules: + + * statlib (statistics) + http://code.google.com/p/python-statlib + + * psycopg2 (PostgreSQL access) + http://initd.org/psycopg + + It should be straightforward to install both of them by following + the instructions on the respective web pages. + + NOTE: Installing psycopg2 fails if you don't have the appropriate + PostgreSQL client library (libpq) installed. + + +STEP 8: Install a script named 'getstatswrapper' for executing + getstats.py with the appropriate environment. The script + typically looks like this on Linux (<BM2 dir> is + /home/qt/repos/bm2 in this example): + + #!/bin/sh + LD_LIBRARY_PATH=/usr/local/pgsql/lib \ + /home/qt/repos/bm2/scripts/getstats.py + + +STEP 9: Configure the Apache web server to 1) allow access to the + <BM2 dir>/web/ directory, and 2) to allow launching + 'getstatswrapper' as a CGI script (see separate documentation + for the web server - typically this requires changes to + httpd.conf etc.). + + +STEP 10: Start the web server and (if desirable) arrange for it be + started automatically at boot time (on a Linux system this would + typically mean to add some stuff under /etc/init.d/). + + +========================================================================= +USAGE + +--- Producing and uploading results ------------------------------- + +STEP 1: Run tests and record performance measurements in an XML file: + + <TestCase name="MyTestCase"> + ... + <TestFunction name="myTestFunction"> + ... + <BenchmarkResult metric="WalltimeMilliseconds" tag="my tag" value="98" iterations="4" /> + ... + </TestFunction> + ... + </TestCase> + + Notes: + - This is the same format as is generated by a QTestLib program started with the -xml + command-line arguument. + - There is a hierarchichal relationship between test cases, test functions, and data tags: + A test case consists of one or more test functions, which in turn consists of one or more + data tags. Each such combination is considered a "benchmark" and is associated with a + single measurement ("benchmark result"). + - The 'tag' attribute may be empty. + - The 'iterations' attribute is optional with a default value of 1. + - The actual value is value / iterations (in the example above it would be 98 / 4 = 24.5). + + +STEP 2: Upload results to the server using the upload script: + + Typically something like this: + + $ ./uploadresults.py --dbhost ... --dbport ... --db bm --host oslo-bm-mac-mini-64-1 \ + --platform macx-g++ --branch qt/qt-script-v8#master-v8 \ + --sha1 ae15094c6818904b47a53590644992f24e7faa97 --file results.xml + + Run './uploadresults.py --help' for a description of the command-line options! + + Notes: + - Results for different SHA-1s (revisions) must be uploaded in the order in + which they appear the revision history. + - Results may be uploaded multiple times for the same SHA-1 (this can be useful + to remove outliers since BM2 uses the median value in such a case). + + + +--- Viewing results ------------------------------------ + +--- Option 1: Using web application -------------------------- + +STEP 1: Open the URL + + Open the top-level index.html page, located under <BM2 dir>/web/, for example: + + http://bmc.test.qt.nokia.com:8080/bm2 + + where bmc.test.qt.nokia.com is the host address, 8080 the Apache port number, + and bm2 a symbolic link to <BM2 dir>/web/. + + Obviously, this may be configured differently on different installations. + + +STEP 2: Interact with the web app to show various reports. + + <Examples to be filled in> + + +--- Option 2: Query the database directly -------------------------- + +<Examples to be filled in> |