Update README a bit

* Improve formatting
* Fix/remove outdated bits and pieces, and move some information like
  test names out of the README (since they grow stale easily).

Change-Id: I70460b785f3112b47e6a4cb6cbe096e88781c2af
Reviewed-by: Gunnar Sletta <gunnar@crimson.no>

1 files changed, 121 insertions, 67 deletions

diff --git a/README.md b/README.md
index 923a91a..0f0a283 100644
--- a/README.md
+++ b/README.md
@@ -27,6 +27,16 @@ qmlbench provides a number of "shells", which are the contaners for the
 benchmarking logic. Different shells work in different ways, and might measure
 different things.
 
+## frame-count shell
+
+The frame-count shell constantly performs a set number of operations each frame,
+for a given amount of (real world) time, and counts the number of frames the
+window is able to generate. If the GL drivers & OS support it, vsync will be
+disabled.
+
+The frame-count shell is used for automated benchmark runs, as it is less
+susceptible to variance than the sustained-fps shell.
+
 ## sustained-fps shell
 
 The sustained-fps shell provides a measure of how many times a certain operation
@@ -72,78 +82,122 @@ staticCount property in benchmarks) each frame. It is useful for profiling.
 You can override the staticCount value by passing a `--count N` command line
 argument, where N is the staticCount you want to use instead.
 
-## frame-count shell
-
-The frame-count shell constantly performs a set number of operations each frame,
-for a given amount of (real world) time, and counts the number of frames the
-window is able to generate. If the GL drivers & OS support it, vsync will be
-disabled.
+The static-count shell is most useful for profiling, as it will repeat the same
+operation continuously.
 
 ##the qmlbench tool
 
 It comes with a number of default settings which aim to help give stable numbers:
 
-- --fps-override: This one is potentially very important. When you ran decidefps,
-it told you a fairly accurate FPS. If QScreen reports something different through refreshRate() the calibration won't work and the resulting numbers won't mean anything. On OSX, we should be mostly good. On Linux, it is not uncommon that the QScreen value is wrong so this option is needed as an override to make the tests stable. The tool has not been tested on Windows at the time when this is written.
-- --delay [ms]: defaults to 2 seconds. An idle time after showing the window before starting the benchmark. Especially useful on OSX where the system specific to-fullscreen animation takes a while. And that to-fullscreen will severly damage the benchmarks
-- --help: Tells you the other options..
-
-TODO describe --hardware-multiplier, --count...
-
-##the actual benchmarks
-
-### benchmarks/creation
-
-This is a small suite of different creation tests. The idea is to find out how good we are with creation and destruction of objects. Both how good we are overall, and which items need more work than others. As we would like items to be light, these numbers should be high. Creating a dialog page easily involves creating 2-300 Item objects, 100 Rectangle objects, 200 Text elements, etc. And any bindings, js logic, model interaction, sql database fetching (threaded of course) comes in addition. For the items, we should strive for 1000 ops / frame on desktop machines and 100 ops / frame on medium to low end embedded. 
-
-- delegates_qobject.qml: Tests raw QObject creation, this is without any of the QML / QQuickItem overhead so these should be a lot higher than anything else.
-- delegates_item.qml: Tests raw QQuickItem creation. 
-
-These two tests are non-visual, so the addition from the graphics stack is mostly just a the swap (which can take quite a bit of time on its own, but that is no fault of Qt).
-
-- delegates_rect.qml
-- delegates_text.qml
-- delegates_image.qml
-
-And others.. Test the basic built-in items
-
-Just run:
-
-> ./qmlbench benchmarks/creation
-
-to run and test them all. As all tests are run 5 times and the results averaged, it will take a while :)
-
-
-### benchmarks/gputhroughput
-
-These benchmarks test your GPU for the most part. The QML part is mostly there to set things up and the benchmark will fill the graphics pipeline with enough stuff to make it suffer. The purpose of these benchmarks is to give an indication how how much one can expect to put on screen, and to decide how big the screen can actually be. Especially in the embedded space where a low end chip may have to drive an HD display, these tests can help to illustrate if the GPU is up to the task or not.
-
-- blendedrect.qml: Stacks x number of alphablended rectangles on top of each other. Rough test of fillrate. 
-- blendedtexture.qml: Stacks x number of alphablended textures on top of each other. Rouch test of fill and texel rate.  
-- opaquerect.qml: Stacks x number of opaque rectangles on top of each other. Will go a lot higher than blendedrect if the target hardware supprts early-z (https://en.wikipedia.org/?title=Z-buffering)
-- opaquetexture.qml: Stacks x number of opaque textures on top of each other. Will go a lot higher than blendedtexture if target hardware supports early-z.
-
-The numbers in these benchmarks mean how many times you can fill the screen. Run them in --fullscreen to get the right numbers. 1 means your in trouble. 2 means you can manage with a lot of work. 4 and above and you should be good shape, GPU-wise.
-
-- drawcalls.qml: Take this one with a grain of salt. Graphics drivers have a lot of overhead in how drawing is set up, and seeing an individual GL call take up to a millisecond (yes, a millisecond) is not uncommon. This test is a highly constructed case to try to pinpoint the rough ballpart of how many discrete draw calls the GL stack is capable of. This is mostly important if you end up with an application that fails to do batching in the scene graph renderer, but then you will have loads of other performance problems as well, so perhaps just ignore this benchmark all together. 
-- gaussblur.qml: This benchmark is added to test how feasible live blurring is. Live blurring is an extreme fillrate test and is only something that should be considered on gaming and industrial hardware. And then you still probably want to cheat :p
-
-### benchmarks/images
-
-These benchmarks are there to validate the casual gaming idea which should be very viable with QML. 
-
-The following three benchmarks give an indication of how many animated items can run simultaneously in the UI. It should be in the thousands. 
-
-- moving-images-animation.qml
-- moving-images-animators.qml
-- moving-images-script.qml
-
-One quirk if you run these is that on a threaded renderloop, the animation one runs faster than animators. This is because the work is broken into two major chunks. One is doing the animation while the other is doing the batching in the renderer and scheduling the rendering. If those happen on separate threads, we get better parallelization so 'animation' comes out better. However, if you try again with QSG_RENDER_LOOP=windows in the environment, you'll see that if it all happens on the same thread, then animators are a bit cheaper (because they are simpler)
-
-### benchmarks/changes
+* ``--fps-override``: This one may be very important when using the
+  sustained-fps shell. When you ran decidefps,
+  it told you a fairly accurate FPS. If QScreen reports something different
+  through refreshRate() the calibration won't work and the resulting numbers
+  won't mean anything. On OSX, we should be mostly good. On Linux, it is not
+  uncommon that the QScreen value is wrong so this option is needed as an
+  override to make the tests stable. The tool has not been tested on Windows at
+  the time when this is written.
+* ``--delay [ms]``: defaults to 2 seconds. An idle time after showing the window
+  before starting the benchmark. Especially useful on OSX where the system
+  specific to-fullscreen animation takes a while. And that to-fullscreen will
+  severly damage the benchmarks.
+* ``--help``: Tells you the other options.
+
+## the actual benchmarks
+
+Benchmarks are divided into two types, automated, and manual. The automated
+tests are useful for regression testing, and the manual tests are useful for
+determining the capabilities of a new piece of hardware.
+
+To run all the automated tests, simply run this, and go have lunch (it is not a
+fast process):
+
+> ./src/qmlbench benchmarks/auto/
+
+You can also run a subset of them by providing that directory instead:
+
+> ./src/qmlbench benchmarks/auto/creation/quick.item/
+
+Or even individual tests, by passing the filenames directly.
+
+### benchmarks/auto/creation/
+
+This is a suite of tests to find out how good we are with creation, rendering
+and destruction of objects. Some of the tests are also written in such a way
+that they can be compared between each other -- these are usually noted in the
+descriptive comment at the top of the test.
+
+Creation is an important factor: our items should be light, as creating a dialog
+or page of UI can often creating a few hundred different items (especially Text,
+Item, Rectangle, etc). In addition to this, there's code outside of our control
+on the application end: JS logic, model interaction, database or file I/O all
+comes in addition - so we need to leave plenty of performance left over for the
+end user.
+
+### benchmarks/manual/gputhroughput
+
+These benchmarks test your GPU for the most part. The QML part is mostly there
+to set things up and the benchmark will fill the graphics pipeline with enough
+stuff to make it suffer.
+
+**Note**! These tests are only designed to be used with the sustained-fps shell.
+
+The purpose of these benchmarks is to give an indication how how much one can
+expect to put on screen, and to decide how big the screen can actually be.
+Especially in the embedded space where a low end chip may have to drive an HD
+display, these tests can help to illustrate if the GPU is up to the task or not.
+
+* ``blendedrect.qml``: Stacks x number of alphablended rectangles on top of
+  each other. Rough test of fillrate.
+* ``blendedtexture.qml``: Stacks x number of alphablended textures on top of
+  each other. Rouch test of fill and texel rate.
+* ``opaquerect.qml``: Stacks x number of opaque rectangles on top of each other.
+  Will go a lot higher than blendedrect if the target hardware supprts
+  [early-z](https://en.wikipedia.org/?title=Z-buffering)
+* ``opaquetexture.qml``: Stacks x number of opaque textures on top of each other.
+  Will go a lot higher than blendedtexture if target hardware supports
+  [early-z](https://en.wikipedia.org/?title=Z-buffering)
+
+The numbers in these benchmarks mean how many times you can fill the screen. Run
+them in --fullscreen to get the right numbers. 1 means your in trouble. 2 means
+you can manage with a lot of work. 4 and above and you should be good shape, GPU-wise.
+
+* ``drawcalls.qml``: Take this one with a grain of salt. Graphics drivers have a
+  lot of overhead in how drawing is set up, and seeing an individual GL call take
+  up to a millisecond (yes, a millisecond) is not uncommon. This test is a highly
+  constructed case to try to get a rough ballpart of how many discrete draw calls
+  the GL stack is capable of.
+
+  This is mostly important if you end up with an application that fails to do
+  batching in the scene graph renderer, but as this situation will typically
+  have many other performance problems, this may not be a useful benchmark for
+  the most part.
+* ``gaussblur.qml``: This benchmark is added to test how feasible live blurring
+  is. Live blurring is an extreme fillrate test and is only something that
+  should be considered on gaming and industrial hardware. And then you still
+  probably want to cheat :p
+
+### benchmarks/auto/animations/comparison/
+
+These benchmarks compare a number of different ways of moving images around, and
+are help validate the casual gaming idea which should be very viable with QML.
+
+The benchmarks give an indication of how many animated items can run
+simultaneously in the UI. It should be in the thousands.
+
+One quirk if you run these is that on a threaded renderloop, the animation one
+runs faster than animators. This is because the work is broken into two major
+chunks. One is doing the animation while the other is doing the batching in the
+renderer and scheduling the rendering. If those happen on separate threads, we
+get better parallelization so 'animation' comes out better.
+
+However, if you try again with QSG_RENDER_LOOP=windows in the environment,
+you'll see that if it all happens on the same thread, then animators are a bit
+cheaper (because they are simpler).
 
-These benchmarks were created as a result of some painful bugreports we got about text performance. So we're testing how different types of changes in text affect the rest of the scene rendering. It is a very specific benchmark, and could be ignored by anyone which is not directly investigating scene graph internals of text drawing.
+### benchmarks/auto/changes
 
-## what's missing?
+These benchmarks help measure the impact of various types of changes in a scene
+(for instance, changing one text item out of many, changing all text items at
+once).
 
-I'd love to get creating benchmarks and some page-stack transition benchmarks from Qt Quick Controls 1.0 and the upcoming 2.0 in here.