1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
|
# qmlbench
qmlbench is a tool for benchmarking Qt, QML and QtQuick as a whole stack rather
than in isolation. The benchmarks it provides exercise a very large part of
Quick, QML, Gui, Core, and as a result can be considered a decent metric for
overall Qt performance.
As with any benchmarking, if you want reliable results, the first thing you must
attend to is ensuring that the system under test is doing the absolute minimum
possible. "Typical" operating systems (OS X, Ubuntu, ...) all have a lot of
"background noise" in the form of file indexing, automatic updates. All of this
should be disabled if possible. You may want to consider disabling networking
for the duration of the benchmarking run to help keep things "quiet".
In terms of running applications, you should have the bare minimum if at all
possible. No email clients, no IRC, no media players. Only your terminal, and
qmlbench should be running. If possible (e.g. on Linux), you may want to even
look into setting up a custom desktop environment session that doesn't launch
anything else (so no gnome-session etc).
Remember: Any background process that might pop up will cause instability in
the results!
# shells
qmlbench provides a number of "shells", which are the containers for the
benchmarking logic. Different shells work in different ways, and might measure
different things.
## frame-count shell
The frame-count shell constantly performs a set number of operations each frame,
for a given amount of (real world) time, and counts the number of frames the
window is able to generate. If the GL drivers & OS support it, vsync will be
disabled.
The frame-count shell is used for automated benchmark runs, as it is less
susceptible to variance than the sustained-fps shell.
## sustained-fps shell
The sustained-fps shell provides a measure of how many times a certain operation
can be performed per frame while sustaining a velvet smooth framerate matching the
host's screen refresh rate.
Summarized, it can be looked at as a test of:
*How much stuff can I do before things start to fall apart.*.
Note that the sustained-fps shell _requires_ a stable frame rate (see the
prerequisites section), which is often hard for systems to provide, so this
shell may not be a good choice for you, and may not be appropriate for automated
regression testing.
### prerequisites
As already mentioned, the goal of the sustained-fps shell is to find how much is
possible while hitting a perfect framerate. So the first goal is to verify
that your system is capable of hitting a perfect framerate at all. This is
called the swap test. Test it by running the following, and observing the screen:
> ./tools/decidefps/decidefps
If you see a pulsating purple rectangle you are in good shape. If you see
flashes of red and/or blue or if the purple rectangle has horizontal regions
of red and blue, your system is not able to hit a perfect framerate.
Fix that first, or you will not get stable/sensible results from the
sustained-fps shell.
Lastly, check if your system is running with the 'basic' render loop
(QSG_INFO=1 in the environment will tell you). If you are, you will most likely
not be able to get velvet FPS because the animations are timer driven and will
skip once or twice per frame by default. If threaded is not an option for
whatever reason, try with QSG_RENDER_LOOP=windows (which uses vsync driven
animations).
## static-count shell
The static-count shell constantly performs a set number of operations (the
staticCount property in benchmarks) each frame. It is useful for profiling.
You can override the staticCount value by passing a `--count N` command line
argument, where N is the staticCount you want to use instead.
The static-count shell is most useful for profiling, as it will repeat the same
operation continuously.
## the qmlbench tool
It comes with a number of default settings which aim to help give stable numbers:
* ``--fps-override``: This one may be very important when using the
sustained-fps shell. When you ran decidefps,
it told you a fairly accurate FPS. If QScreen reports something different
through refreshRate() the calibration won't work and the resulting numbers
won't mean anything. On OSX, we should be mostly good. On Linux, it is not
uncommon that the QScreen value is wrong so this option is needed as an
override to make the tests stable. The tool has not been tested on Windows at
the time when this is written.
* ``--delay [ms]``: defaults to 2 seconds. An idle time after showing the window
before starting the benchmark. Especially useful on OSX where the system
specific to-fullscreen animation takes a while. And that to-fullscreen will
severly damage the benchmarks.
* ``--help``: Tells you the other options.
## the actual benchmarks
Benchmarks are divided into two types, automated, and manual. The automated
tests are useful for regression testing, and the manual tests are useful for
determining the capabilities of a new piece of hardware.
To run all the automated tests, simply run this, and go have lunch (it is not a
fast process):
> ./src/qmlbench benchmarks/auto/
You can also run a subset of them by providing that directory instead:
> ./src/qmlbench benchmarks/auto/creation/quick.item/
Or even individual tests, by passing the filenames directly.
## the results
Here's an example of running `src/qmlbench --shell frame-count /Users/mitch/dev/qmlbench/benchmarks/auto/creation/quick.controls2/delegates_tumbler.qml` on macOS:
ID:
OS: macOS High Sierra (10.13)
QPA: cocoa
GL_VENDOR: ATI Technologies Inc.
GL_RENDERER: AMD Radeon Pro 460 OpenGL Engine
GL_VERSION: 2.1 ATI-1.60.24
running: /Users/mitch/dev/qmlbench/benchmarks/auto/creation/quick.controls2/delegates_tumbler.qml
153 frames
152 frames
154 frames
154 frames
153 frames
Average: 153.2 frames; MedianAll=153; StdDev=0.83666, CoV=0.00546123
All done...
The test was repeated 5 times, and for each run, the frames are printed. For the frame-count shell, the higher the number of frames is, the better.
- "Average: 153.2 frames" - this is the average of each run.
- "MedianAll=153" - the [median](https://en.wikipedia.org/wiki/Median) of all of the runs:
(152, 153, 153, 154, 154)
^
- "StdDev=0.83666" - the [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation). The lower the better.
- "CoV=0.00546123" - the [coefficient of variation](https://en.wikipedia.org/wiki/Coefficient_of_variation). The lower the better; a general guide is to aim for less than 5% CoV.
## Running tests on Android
To run qmlbench on Android, you need to use the subprocess mode. To
pass arguments to the Android app, they need to be base64-encoded.
Here's an example:
~/Android/Sdk/platform-tools/adb shell am start -n org.qtproject.example.qmlbench/org.qtproject.qt.android.bindings.QtActivity -e extraappparams `echo -n "--subprocess-mode --json-file :benchmarks/auto/creation/quick.text/delegates_longtext_arabic.qml" | base64 -w 0`
So, what we're doing here is
- launch the right app with the right activity:
-n org.qtproject.example.qmlbench/org.qtproject.qt.android.bindings.QtActivity
- pass it the parameters, \-\-subprocess-mode \-\-json-file
- the command line is created with echo \-n so that we don't insert a superfluous newline into the data
- and then we encode with base64 \-w 0 to disable base64 line wrapping, so that that doesn't insert superfluous newlines either.
Since we're passing \-\-json-file, that tells qmlbench to output the benchmark
result into a file in JSON format. The file is qmlbench.txt in the application's
data directory, so the usual incantation to get it out from a phone is
~/Android/Sdk/platform-tools/adb pull sdcard/Android/data/org.qtproject.example.qmlbench/files/Documents/qmlbench.txt
The benchmark app will not recreate the file, it always appends to it.
If you need to start from a clean slate, just adb rm that same file.
## Creating benchmarks
See [Benchmarks/benchmarks.md](./benchmarks/benchmarks.md)
|
