Add readme for making new benchmarks

Provide documentation on how benchmark files are constructed, along
with some usage examples.

Change-Id: Ica3d12ece98e8c1f37d0290bbdf6671c91e539c1
Reviewed-by: Mitch Curtis <mitch.curtis@qt.io>

2 files changed, 128 insertions, 0 deletions

diff --git a/README.md b/README.md
index 6aa3240..547c02b 100644
--- a/README.md
+++ b/README.md
@@ -147,3 +147,7 @@ The test was repeated 5 times, and for each run, the frames are printed. For the
                ^
 - "StdDev=0.83666" - the [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation). The lower the better.
 - "CoV=0.00546123" - the [coefficient of variation](https://en.wikipedia.org/wiki/Coefficient_of_variation). The lower the better; a general guide is to aim for less than 5% CoV.
+
+## Creating benchmarks
+
+See [Benchmarks/benchmarks.md](./benchmarks/benchmarks.md)
diff --git a/benchmarks/benchmarks.md b/benchmarks/benchmarks.md
new file mode 100644
index 0000000..dace5dd
--- /dev/null
+++ b/benchmarks/benchmarks.md
@@ -0,0 +1,124 @@
+[<-- Back to README.md](../README.md)
+
+# Creating custom benchmarks
+Sometimes an existing test doesn't cover your hacks to qt, or maybe you're adding new functionality,
+either way, creating new benchmarks is easy. ~Here's how!~
+
+## The fundamentals of QMLBench
+When running qmlbench in `frame-count shell` (default mode), a static count of operations is
+performed for each frame rendered. QMLbench renders as quickly as possible over a 20 second
+(default) period counting the total frames rendered during that period. This is the 'score'.
+In this way, a benchmark should be defined to perform as many operations as possible without
+hanging qmlbench, typically managing to run at a few frames-per-second on mid-grade hardware.
+If this rule is observed, a benchmark should be able to run on low-grade embedded hardware without
+timing out and hanging, as well as on high-end hardware without acheiving 60fps
+(the typical soft cap if vsync cannot be disabled).
+
+
+## A sample file
+
+    import QtQuick 2.0
+    import QmlBench 1.0
+
+    // Tests the creation of rectangles with *NO* alpha channel.
+    // Compare with delegates_rect_blended & delegates_rect_radius
+    CreationBenchmark {
+        id: root;
+        count: 50;
+        staticCount: 2500;
+        delegate: Rectangle {
+            x: QmlBench.getRandom() * (root.width - width)
+            y: QmlBench.getRandom() * (root.height - height)
+            width: 30
+            height: 15
+            color: "steelblue"
+        }
+    }
+
+
+## The root level declarations
+In the example above, we're testing the creation speed of the `Rectangle` delegate, so we're using `CreationBenchmark`.
+
+There are three types of rool level declarations allowed:
+- Benchmark
+    - The basic QMLBench type.
+    - Good for testing performance of operations with fixed durations, such as animations.
+    - Provides the `count` and `staticCount` properties.
+    - Provides a tick property `t` from 0 to 1, lasting 1000ms that benchmarks can access.
+- CreationBenchmark
+    - Inherits Benchmark
+    - Creates `count` instances of an object before displaying the rendered frame.
+    - Destroys all items and recreates them each frame.
+- Item (Inherited from QtBase)
+    - must declare the following manually
+    ```
+    id: root;
+    property int count: your_count_here;
+    property int staticCount: your_static_count_here;
+
+    property real t;
+    NumberAnimation on t { from: 0; to: 1; duration: your_ticker_duration_in_ms; loops: Animation.Infinite }
+    ```
+
+
+## Count and StaticCount
+In general, it is good practice to set both values so the benchmark can be used with any shell type.
+
+### count
+Used in `susained-fps shell` mode. The benchmark will alter this value trying to achieve 60fps
+(or rather, the screen's refresh rate). Set this value to as a starting point for the benchmark to
+try first.
+
+### staticCount
+Used in all `static` shell types. This is the count of objects to create on startup.
+- If using CreationBenchmark root type, this is the count of objects to create each frame.
+- If using Benchmark, this is the count of objects to create at the start of the benchmark run.
+
+
+## Item delegates (Creation Benchmarks)
+In the code sample above, the `delegate` (a property of CreationBenchmark) is defined as a
+Rectangle with a fixed size and random position, shaded a single color with no transparency.
+- This `delegate` will be repeated in the render window in accordance with the properties `count`
+or `staticCount`, for a given shell type.
+- The delegates will be destroyed and regenerated for each frame rendered.
+
+
+## Repeaters (Non-creation benchmarks)
+Let's take a look at a new code sample. For this benchmark, we're repeating an operation `count`
+times using `Repeater` at the benchmark outset, and continuing to render for the duration of the test.
+
+    import QtQuick 2.2
+    import QmlBench 1.0
+
+    // Move images around using Animation types, to be compared with a number of
+    // other similar ways to move them around.
+    Benchmark {
+        id: root;
+
+        count: 500
+        staticCount: 20000
+
+        Repeater {
+            model: root.count
+            Image {
+                source: "../../../../shared/butterfly-wide.png"
+                x: QmlBench.getRandom() * (root.width - width)
+                y: QmlBench.getRandom() * (root.height - height)
+                width: 40
+                height: 40
+
+                SequentialAnimation on rotation {
+                    NumberAnimation { from: -10; to: 10; duration: 500; easing.type: Easing.InOutCubic }
+                    NumberAnimation { from: 10; to: -10; duration: 500; easing.type: Easing.InOutCubic }
+                    loops: Animation.Infinite
+                }
+            }
+        }
+    }
+
+In this case, the image `butterfly-wide.png` is being randomly placed in the render window, and has
+an animation applied on `rotation`, looping infinitely. The image and animation will be repeated 500
+times, resulting in 500 randomly placed, rotating images for the duration of the benchmark run.
+
+- Multiple `Repeaters` can be added if necessary, though it's recommended to test the smallest
+possible operation specific to your change.