1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
|
--------------------------------------------------------------------------------------------
OVERVIEW
This repository contains BM - a set of tools for storing and analyzing
benchmark results produced by QTestLib (or strictly anything that outputs the
same XML format as QTestLib).
A server (bmserver) maintains a database of benchmark results. A simple
stream-based protocol is defined for storing results and retrieving results
and/or context (meta) data associated with the results.
A client (bmclient) takes input from command-line, standard input, and/or
files, and generates output to standard output. The bmclient may either be
used directly by a user from a console, or launched from another program such
as bmrunner (see below) or a web server. When launched from a web server, the
bmclient acts as a CGI program and outputs JSON data to be forwarded to the
JavsScript program in the web client.
A benchmark runner (bmrunner) executes a given set of QTestLib-based benchmark
programs and passes the results of each one to a bmsrever via a bmclient.
--------------------------------------------------------------------------------------------
CONTENTS OF THIS DIRECTORY
README - This file.
src/ - Source code:
src/bm/ - Common code (C++/Qt).
src/bmclient/ - The bmclient (C++/Qt).
src/bmserver/ - The bmserver (C++/Qt).
src/bmrunner/ - A program for running QTestLib-based benchmark
programs and passing the results to a bmserver (C++/Qt).
src/addtestdata/ - A program for passing artificial benchmark results
to a bmserver (C++/Qt).
src/bmweb/ - Web client code (HTML and JavaScript).
doc/ - Various documentation, including:
doc/bmclient.html - The bmclient (including command-line args and JSON output).
doc/bmserver.html - The bmserver.
doc/bmproto.html - The XML-based client-server protocol.
doc/bmrunner.html - The bmrunner.
--------------------------------------------------------------------------------------------
DEPLOYMENT AND INSTALLATION
The computers that form the BM system play different roles (some may play more
than one). Below is a brief description of the typical installation- and
deployment steps required for each role.
Role 1: Running the bmserver.
Step 1: Build and install the bmserver (a Qt program).
Step 2: Start the bmserver, passing a SQLite database file to be either
created (with no data) or reopened.
Step 3: Arrange for the bmserver to be automatically started at boot time
if desired (on Linux, this typically involves updating files under
/etc/init.d/).
Role 2: Generating new benchmark results and passing them to the bmserver.
Step 1: Use git to fetch the latest version of the product to be
benchmarked (typically a particular version of Qt).
Step 2: Build and install the product.
Step 3: Build and install the benchmark programs (typically QTestLib
applications linked with the latest product version).
Step 4: Build and install the bmclient (a Qt program).
Step 5: Build and install the bmrunner (a Qt program).
Step 6: Execute the bmrunner, passing the location of the bmserver, the
directory of the benchmark programs etc. (see separate
documentation for bmrunner). Note: The bmrunner assumes that the
bmclient can be launched directly, so for example on Linux, the
PATH and LD_LIBRARY_PATH may need to be modified first. The
bmclient in turn assumes that the git command is available
(ditto).
Note: It is strictly not necessary to repeat steps 4 and 5 every time new
results are to be generated, but in a fully automated setup it could
be convenient to rebuild/reinstall even these two programs.
Role 3: Running the web server.
Step 1: Build and install the bmclient (a Qt program).
Step 2: Install the bmweb directory.
Step 3: Build and install the web server (see separate documentation for
the web server).
Step 4: Install a script named 'bmcclientwrapper' that typically looks
like this on Linux:
#!/bin/sh
LD_LIBRARY_PATH=<location of any shared libraries required by the bmclient> \
BMSERVER=<location of the bmserver (host:port)> \
<bmclient executable>
Step 5: Configure the web server to 1) allow access to the bmweb directory,
and 2) to allow launching 'bmclientwrapper' as a CGI script (see
separate documentation for the web server).
Step 6: Start the web server.
Step 7: Arrange for the web server to be automatically started at boot time
if desired.
Note: The BM system has been tested with Apache 2.2.11 on Linux.
Role 4: Viewing results in a web client.
Step 1: Build, install, and run the web client.
Step 2: Open the URL to the BM site (i.e. referring to the bmweb directory on
the web server host.)
Step 3: Enable cookies for the BM site to store certain settings between
sessions.
Note: The BM system has been tested with Firefox 3.5.3 on Linux. Currently,
the use of the <canvas> tag for client-side graphics (in particular
the history plot) limits the usability of certain browsers, like
konqueror.
|
