summaryrefslogtreecommitdiffstats
path: root/doc
diff options
context:
space:
mode:
authorjasplin <qt-info@nokia.com>2009-10-05 13:20:17 +0200
committerjasplin <qt-info@nokia.com>2009-10-05 13:20:17 +0200
commit2d1fae93437c3bdba2fb879def6885ca6d5214ab (patch)
tree815b394fc2086586b6333f0524b69f5bb89a683b /doc
parent64eabf478d1e17e1119007480edf131d8db1c6bf (diff)
Lot's of changes.
Diffstat (limited to 'doc')
-rw-r--r--doc/bmc.html167
-rw-r--r--doc/bmcp.html12
2 files changed, 143 insertions, 36 deletions
diff --git a/doc/bmc.html b/doc/bmc.html
index aeced44..c13da75 100644
--- a/doc/bmc.html
+++ b/doc/bmc.html
@@ -106,7 +106,7 @@ The new results are assumed to have been generated using a product built from
a clean checkout of the current head commit of the current git branch in the
git directory.
<br /><br />
-The sha1 to be stored in the database is assumed to be the current head commit
+The SHA-1 to be stored in the database is assumed to be the current head commit
in the git directory (extracted running <span class="command"> git log -n 1 HEAD
--pretty=format:%H</span> or equivalent).
<br /><br />
@@ -383,62 +383,146 @@ Content-type: text/json
&lt;test function&gt; &lt;data tag&gt; \
&lt;metric&gt; &lt;platform&gt; &lt;host&gt; \
&lt;git repo&gt; &lt;git branch&gt; \
- {-timestamp &lt;first Unix timestamp&gt; | \
+ {-timestamp \
+ &lt;first Unix timestamp&gt; | \
-sha1 &lt;first sha1&gt;} \
- {-timestamp &lt;last Unix timestamp&gt; | \
+ {-timestamp \
+ &lt;last Unix timestamp&gt; | \
-sha1 &lt;last sha1&gt;} \
- [&lt;size&gt; [&lt;tolerance&gt;]]
+ &lt;diff tolerance&gt; \
+ &lt;stab tolerance&gt; \
+ &lt;max size&gt; [-statonly]
</pre>
</td>
<td class="commandDescr">
-Lists the latest results within a particular historical range for a particular context.
+Lists a subset of the time series of results within a particular historical
+range for a particular context. Also lists some basic trend statistics of the
+time series.
+
<br /><br />
-The results are filtered in three ways:
+The time series is listed chronological order (earliest result first).
-<ol>
+<br /><br />
+The range of the time series is defined by the <span
+class="command">-timestamp</span>/<span class="command">-sha1</span> options.
+If the <span class="command">-timestamp</span> option is used, the values
+<b><code>first</code></b> and <b><code>last</code></b> specify the first and
+last available snapshot respectively. Otherwise, the first timestamp refers to
+the earliest snapshot that is not earlier than the specified value, while the
+second timestamp refers to the latest snapshot that is not later than the
+specified value.
+
+<b>Note:</b> Using the <span class="command">-sha1</span> option requires a snapshot with
+this exact value to exist for the command to succeed.
+
+<br /><br />
+
+If <span class="command">&lt;max size&gt;</span> is non-negative, the oldest
+data points of the resulting time series are dropped as necessary to ensure
+that the length does not exceed the maximum size.
-<li>
-Only results within a certain snapshot range are considered. This range is
-defined by the <span class="command">-timestamp</span>/<span
-class="command">-sha1</span> options. If the <span
-class="command">-timestamp</span> option is used, the values
-<i>first</i> and <i>last</i> specify the first and last
-available snapshot respectively. Otherwise, the first timestamp refers to the earliest
-snapshot that is not earlier than the specified value, while the second
-timestamp refers to the latest snapshot that is not later than the specified value.
<br /><br />
-If the <span class="command">-sha1</span> option is used, a snapshot with this
-exact value must exist.
+For the sake of the following description, denote the final time series as:
<br /><br />
-</li>
+&nbsp;&nbsp;&nbsp;&nbsp;
+(t<sub>1</sub>,&nbsp;v<sub>1</sub>),&nbsp;(t<sub>2</sub>,&nbsp;v<sub>2</sub>),&nbsp;
+&hellip;,&nbsp;(t<sub><i>n</i></sub>,&nbsp;v<sub><i>n</i></sub>),
+<br /><br />
+where t<sub><i>i</i></sub> is the timestamp component of snapshot <i>i</i> and
+v<sub><i>i</i></sub> is the accumulated value of the corresponding result
+divided by the number of iterations required to produce it. (<b>Note:</b> The
+accumulated value is an integer, and is sometimes imprecisely referred to as
+the "value"!)
-<li>
-Chronologically consecutive results that differ by less than a certain
-<i>tolerance</i> are collapsed into a single result. Results are processed
-in reverse chronological order, and the the representative result of each
-"equivalence span" is always the latest result. Differences are are measured
-in percentages of the representative result.
<br /><br />
-Observe that a tolerance value of zero effectively disables this filter (since
-no (absolute) differences can be less than zero).
+<b>Current Change</b>
<br /><br />
-</li>
-<li>
-The number of equivalence spans is limited by a certain <i>size</i>. Older
-results are not included.
+This is the position of the most recent value (if any) that differs from the
+last value by more than the difference tolerance (measured in percentage of
+the last value). In other words, this is the largest <i>c</i> such that <br
+/><br /> &nbsp;&nbsp;&nbsp;&nbsp;
+|((v<sub><i>n</i></sub>&nbsp;-&nbsp;v<sub><i>c</i></sub>)&nbsp;/&nbsp;v<sub><i>n</i></sub>)
+&nbsp;*&nbsp;100|&nbsp;>&nbsp;<span
+class="command">&lt;diff&nbsp;tolerance&gt;</span>.
+
+<br /><br />
+
+<span class="command">&lt;diff&nbsp;tolerance&gt;</span> must be a
+non-negative real number.
+
<br /><br />
-Use a negative size value to disable this filter and get all available spans.
+<b>Current Change Stability</b>
+<br /><br />
+
+The current change <i>c</i> is said to be <i>stable</i> if both
+v<sub><i>n</i></sub> and v<sub><i>c</i></sub> are preceded consecutively by at
+least <span class="command">&lt;stab&nbsp;tolerance&gt;</span> values that
+don't differ significantly from v<sub><i>n</i></sub> and v<sub><i>c</i></sub>
+respectively.
+
+<br /><br />
+
+<span class="command">&lt;stab&nbsp;tolerance&gt;</span> must be a
+non-negative integer.
+
<br /><br />
-Observe that a size of one simply retrieves the latest result within the
-snapshot range.
-</li>
+<b>Trend</b>
+<br /><br />
+
+The trend is estimated as the <i>&beta;</i> value of the <a
+href="http://en.wikipedia.org/wiki/Simple_linear_regression">simple linear
+regression</a> of the normalized time series:
+<br /><br />
+&nbsp;&nbsp;&nbsp;&nbsp;
+<i>y</i>&nbsp;=&nbsp;<i>&alpha;</i>&nbsp;+&nbsp;<i>&beta;x</i>
+
+<br /><br />
+
+If the value at time <i>t</i> is <i>v</i>(<i>t</i>), and the trend continues,
+the value at time <i>t</i>&nbsp;+&nbsp;<i>dt</i> is expected to be
+<i>v</i>(<i>t</i>)&nbsp;+&nbsp;<i>&beta;</i>*<i>dt</i>.
+
+<br /><br />
+
+(Observe that positive and negative <i>&beta;</i> values indicate decreased
+and increased performance respectively.)
+
+<br /><br />
+
+The normalized time series is defined like this:
+<ol>
+<li>
+Scale the timestamps linearly into the [0,&nbsp;1] range.
+</li>
+<li>
+Divide each value by the last value. The normalized value 1 thus
+indicates no change, while 0.5 and 1.5 indicate 50% better and worse than the
+last value respectively.
+</li>
</ol>
+<br />
+
+If the <span class="command">-statonly</span> option is passed, only the
+latest result (if any) of the time series and the result representing the
+current change (if any) will be listed after the statistics.
+
</td>
<td>
<pre>
+currChange: \
+ &lt;index of \
+ the current change, \
+ or -1 if no such \
+ change exists&gt;
+currChangeStable: \
+ &lt;true or false&gt; \
+regression: \
+ &lt;a b or -1 if \
+ regression cannot \
+ be computed&gt;
&lt;timestamp 1&gt;
&lt;sha1 1&gt;
&lt;value 1&gt;
@@ -454,6 +538,19 @@ snapshot range.
Content-type: text/json
{
+ "currChange":
+ &lt;index of the current change&gt;,
+ // Property omitted if no
+ // such change exists
+
+ "currChangeStable":
+ &lt;true or false&gt;,
+
+ "regression": [&lt;a&gt;, &lt;b&gt;],
+ // Property omitted if the
+ // regression cannot be
+ // computed
+
"items": [
["&lt;timestamp 1&gt;",
"&lt;sha1 1&gt;",
diff --git a/doc/bmcp.html b/doc/bmcp.html
index ce4ee6f..c6bc332 100644
--- a/doc/bmcp.html
+++ b/doc/bmcp.html
@@ -266,6 +266,7 @@ Behaves similarly to GetBenchmarks (see this).
<td class="requestDescr">
Lists the raw results within a particular historical range for a particular
context.
+
<br /><br />
<b>Note:</b> For each snapshot (defining the beginning and end of the range
respectively) only one of <i>timestamp</i> or <i>sha1</i> is required. If
@@ -274,6 +275,11 @@ first and last available snapshot respectively. Otherwise, the first timestamp
refers to the earliest snapshot that is not earlier than the specified value,
while the second timestamp refers to the latest snapshot that is not later than
the specified value.
+
+<br /><br />
+<b>Note:</b> The currChange and/or a,b attributes of the &lt;statistics&gt;
+tag will be empty if the corresponding values could not be computed.
+
</td>
<td>
@@ -285,7 +291,8 @@ the specified value.
gitRepo="..." gitBranch="..."
timestamp1="..." sha11="..."
timestamp2="..." sha12="..."
- size="..." tolerance="..." /&gt;
+ diffTolerance="..." stabTolerance="..."
+ maxSize="..." /&gt;
&lt;/request&gt;
</pre>
</td>
@@ -293,6 +300,9 @@ the specified value.
<td>
<pre>
&lt;reply type="GetHistory"&gt;
+ &lt;statistics currChange="..."
+ currChangeStable="..."
+ a="..." b="..." /&gt;
&lt;result timestamp="..." sha1="..."
value="..." iterations="..." /&gt;
&lt;result timestamp="..." sha1="..."