blob: 2264587a1d5c223fc7aa91c3eafb0694716f5ab5 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
|
=========================================================================
OVERVIEW
This repository contains BM2 - a system for storing and analyzing
performance testing results.
The idea is to run performance tests on test machines and upload the
results to a database on a server machine. The database can then be
inspected either from a web application or using SQL queries directly.
=========================================================================
INSTALLATION
STEP 1: Install the BM2 directory (containing this README!) under an
arbitrary directory <BM2 dir>.
STEP 2: Install Python 2.6 (with development headers) on both the
server machine and the machine from which results will be uploaded
(NOTE: the latter may or may not be identical to the test
machine).
http://www.python.org/download/releases/2.6.7/
NOTE: Version 2.7 or later may not work due to potential source
incompatibilies!
STEP 3: Install a recent version of PostgreSQL (with development
headers) (8.4 or later should work fine).
http://www.postgresql.org/
NOTE: Ensure to install both the server (postgres) and interactive
terminal (psql).
STEP 4: Start the database server and (if desirable) arrange for it be
started automatically at boot time (on a Linux system this would
typically mean to add some stuff under /etc/init.d/).
STEP 5: Initialize the database:
$ <BM2 dir>/database/scripts/reset.py
STEP 6: Install a recent version the Apache web server (2.2 or later
should do fine).
http://httpd.apache.org/
STEP 7: Detect and install missing Python modules:
$ cd <BM2 dir>/scripts
$ ./getstats.py
A missing module will reported as:
...
ImportError: No module named <name>
You are likely to be missing the following modules:
* statlib (statistics)
http://code.google.com/p/python-statlib
* psycopg2 (PostgreSQL access)
http://initd.org/psycopg
It should be straightforward to install both of them by following
the instructions on the respective web pages.
NOTE: Installing psycopg2 fails if you don't have the appropriate
PostgreSQL client library (libpq) installed.
STEP 8: Install a script named 'getstatswrapper' for executing
getstats.py with the appropriate environment. The script
typically looks like this on Linux (<BM2 dir> is
/home/qt/repos/bm2 in this example):
#!/bin/sh
LD_LIBRARY_PATH=/usr/local/pgsql/lib \
/home/qt/repos/bm2/scripts/getstats.py
STEP 9: Configure the Apache web server to 1) allow access to the
<BM2 dir>/web/ directory, and 2) to allow launching
'getstatswrapper' as a CGI script (see separate documentation
for the web server - typically this requires changes to
httpd.conf etc.).
STEP 10: Start the web server and (if desirable) arrange for it be
started automatically at boot time (on a Linux system this would
typically mean to add some stuff under /etc/init.d/).
=========================================================================
USAGE
--- Producing and uploading results -------------------------------
STEP 1: Run tests and record performance measurements in an XML file:
<TestCase name="MyTestCase">
...
<TestFunction name="myTestFunction">
...
<BenchmarkResult metric="WalltimeMilliseconds" tag="my tag" value="98" iterations="4" />
...
</TestFunction>
...
</TestCase>
Notes:
- This is the same format as is generated by a QTestLib program started with the -xml
command-line arguument.
- There is a hierarchichal relationship between test cases, test functions, and data tags:
A test case consists of one or more test functions, which in turn consists of one or more
data tags. Each such combination is considered a "benchmark" and is associated with a
single measurement ("benchmark result").
- The 'tag' attribute may be empty.
- The 'iterations' attribute is optional with a default value of 1.
- The actual value is value / iterations (in the example above it would be 98 / 4 = 24.5).
STEP 2: Upload results to the server using the upload script:
Typically something like this:
$ ./uploadresults.py --dbhost ... --dbport ... --db bm --host oslo-bm-mac-mini-64-1 \
--platform macx-g++ --branch qt/qt-script-v8#master-v8 \
--sha1 ae15094c6818904b47a53590644992f24e7faa97 --file results.xml
Run './uploadresults.py --help' for a description of the command-line options!
Notes:
- Results for different SHA-1s (revisions) must be uploaded in the order in
which they appear the revision history.
- Results may be uploaded multiple times for the same SHA-1 (this can be useful
to remove outliers since BM2 uses the median value in such a case).
--- Viewing results ------------------------------------
--- Option 1: Using web application --------------------------
STEP 1: Open the URL
Open the top-level index.html page, located under <BM2 dir>/web/, for example:
http://bmc.test.qt.nokia.com:8080/bm2
where bmc.test.qt.nokia.com is the host address, 8080 the Apache port number,
and bm2 a symbolic link to <BM2 dir>/web/.
Obviously, this may be configured differently on different installations.
STEP 2: Interact with the web app to show various reports.
<Examples to be filled in>
--- Option 2: Query the database directly --------------------------
<Examples to be filled in>
|
