diff options
author | jasplin <qt-info@nokia.com> | 2009-10-29 10:01:06 +0100 |
---|---|---|
committer | jasplin <qt-info@nokia.com> | 2009-10-29 10:01:06 +0100 |
commit | 5efb9c7370792303adfb312f5027cf1756bfac1d (patch) | |
tree | 5322137e2b5389bb76d665afdaff66a65462790e /doc | |
parent | 32886d4960dbcec5e1dbbae2229ed6b4cca812ab (diff) |
Added new request types.
This commit essentially adds the following request types:
- get rankedbenchmarks
- get rankedbenchmarks2
- get stats
Plus a lot of other (minor) modifications.
Diffstat (limited to 'doc')
-rw-r--r-- | doc/bmclient.html | 456 | ||||
-rw-r--r-- | doc/bmproto.html | 159 |
2 files changed, 542 insertions, 73 deletions
diff --git a/doc/bmclient.html b/doc/bmclient.html index 85fbfc1..89b1494 100644 --- a/doc/bmclient.html +++ b/doc/bmclient.html @@ -88,7 +88,7 @@ to the raw mode command line. <tr> <td> <pre class="command"> -./bmclient put results <file> \ +./bmclient <b>put results</b> <file> \ <platform> <host> \ <git repo> [<git dir>] </pre> @@ -131,7 +131,7 @@ repository). <tr> <td> <pre class="command"> -./bmclient get snapshots \ +./bmclient <b>get snapshots</b> \ <test case> <test function> \ <data tag> <metric> \ <platform> <host> \ @@ -171,7 +171,7 @@ Content-type: text/json <tr> <td> <pre class="command"> -./bmclient get benchmarks \ +./bmclient <b>get benchmarks</b> \ [-metric <metric>] \ [-platform <platform>] \ [-host <host>] \ @@ -225,7 +225,7 @@ Content-type: text/json <tr> <td> <pre class="command"> -./bmclient get metrics \ +./bmclient <b>get metrics</b> \ [-benchmark <test case> \ <test function> <data tag>] \ [-platform <platform>] \ @@ -235,7 +235,8 @@ Content-type: text/json </td> <td class="commandDescr"> Lists available metrics. -Behaves similarly to <span class="command">./bmclient get benchmarks ...</span> (see this). +Behaves similarly to the <span class="command"><b>get benchmarks</b></span> command +(see this). </td> <td> <pre> @@ -263,7 +264,7 @@ Content-type: text/json <tr> <td> <pre class="command"> -./bmclient get platforms \ +./bmclient <b>get platforms</b> \ [-benchmark <test case> \ <test function> <data tag>] \ [-metric <metric>] \ @@ -273,7 +274,8 @@ Content-type: text/json </td> <td class="commandDescr"> Lists available platforms. -Behaves similarly to <span class="command">./bmclient get benchmarks ...</span> (see this). +Behaves similarly to the <span class="command"><b>get benchmarks</b></span> command +(see this). </td> <td> <pre> @@ -301,7 +303,7 @@ Content-type: text/json <tr> <td> <pre class="command"> -./bmclient get hosts \ +./bmclient <b>get hosts</b> \ [-benchmark <test case> \ <test function> <data tag>] \ [-metric <metric>] \ @@ -311,7 +313,8 @@ Content-type: text/json </td> <td class="commandDescr"> Lists available hosts. -Behaves similarly to <span class="command">./bmclient get benchmarks ...</span> (see this). +Behaves similarly to the <span class="command"><b>get benchmarks</b></span> command +(see this). </td> <td> <pre> @@ -339,7 +342,7 @@ Content-type: text/json <tr> <td> <pre class="command"> -./bmclient get branches \ +./bmclient <b>get branches</b> \ [-benchmark <test case> \ <test function> <data tag>] \ [-metric <metric>] \ @@ -349,7 +352,8 @@ Content-type: text/json </td> <td class="commandDescr"> Lists available branches. -Behaves similarly to <span class="command">./bmclient get benchmarks ...</span> (see this). +Behaves similarly to the <span class="command"><b>get benchmarks</b></span> command +(see this). </td> <td> <pre> @@ -381,7 +385,7 @@ Content-type: text/json <tr> <td> <pre class="command"> -./bmclient get history <test case> \ +./bmclient <b>get history</b> <test case> \ <test function> <data tag> \ <metric> <platform> <host> \ <git repo> <git branch> \ @@ -393,19 +397,21 @@ Content-type: text/json -sha1 <last sha1>} \ <diff tolerance> \ <stab tolerance> \ - <max size> [-statonly] + <max size> </pre> </td> <td class="commandDescr"> -Lists a subset of the time series of results within a particular historical -range for a particular context. Also lists some basic trend statistics of the -time series. -<br /><br /> -The time series is listed chronological order (earliest result first). +Computes statistics for a subrange of the time series of results +for a specific benchmark within a specific context. This subrange is called +the <i>input</i> time series. The command also lists the most recent fraction +(possibly all) of the input time series as the <i>output</i> time series. <br /><br /> -The range of the time series is defined by the <span +<b>Snapshot range</b> +<br /><br /> + +The range of the input time series is defined by the <span class="command">-timestamp</span>/<span class="command">-sha1</span> options. If the <span class="command">-timestamp</span> option is used, the values <b><code>first</code></b> and <b><code>last</code></b> specify the first and @@ -414,17 +420,11 @@ the earliest snapshot that is not earlier than the specified value, while the second timestamp refers to the latest snapshot that is not later than the specified value. -<b>Note:</b> Using the <span class="command">-sha1</span> option requires a snapshot with -this exact value to exist for the command to succeed. - -<br /><br /> - -If <span class="command"><max size></span> is non-negative, the oldest -data points of the resulting time series are dropped as necessary to ensure -that the length does not exceed the maximum size. +<b>Note:</b> Using the <span class="command">-sha1</span> option requires a +snapshot with this exact value to exist for the command to succeed. <br /><br /> -For the sake of the following description, denote the final time series as: +For the sake of the following description, denote the input time series as: <br /><br /> (t<sub>1</sub>, v<sub>1</sub>), (t<sub>2</sub>, v<sub>2</sub>), @@ -437,16 +437,16 @@ accumulated value is an integer, and is sometimes imprecisely referred to as the "value"!) <br /><br /> -<b>Current Change</b> +<b>Difference</b> <br /><br /> -This is the position of the most recent value (if any) that differs from the -last value by more than the difference tolerance (measured in percentage of -the last value). In other words, this is the largest <i>c</i> such that <br -/><br /> -|((v<sub><i>n</i></sub> - v<sub><i>c</i></sub>) / v<sub><i>n</i></sub>) - * 100| > <span -class="command"><diff tolerance></span>. +For a given value v<sub><i>i</i></sub> in the time series, the <i>difference +tolerance</i> (specified as <span +class="command"><diff tolerance></span>) is the maximum percentage +by which another value v<sub><i>j</i></sub> is allowed to differ from +v<sub><i>i</i></sub> to be considered equal to v<sub><i>i</i></sub>. Values +that are equal/different by this definition are said to be +<i>significantly</i> equal/different. <br /><br /> @@ -454,14 +454,13 @@ class="command"><diff tolerance></span>. non-negative real number. <br /><br /> -<b>Current Change Stability</b> +<b>Value Stability</b> <br /><br /> -The current change <i>c</i> is said to be <i>stable</i> if both -v<sub><i>n</i></sub> and v<sub><i>c</i></sub> are preceded consecutively by at -least <span class="command"><stab tolerance></span> values that -don't differ significantly from v<sub><i>n</i></sub> and v<sub><i>c</i></sub> -respectively. +The value at index <i>i</i> is said to be <i>stable</i> if it is preceded +consecutively by at least <span +class="command"><stab tolerance></span> values that +don't differ significantly from v<sub><i>i</i></sub>. <br /><br /> @@ -469,6 +468,30 @@ respectively. non-negative integer. <br /><br /> +<b>Current Change</b> +<br /><br /> + +This is the position of the most recent value (if any) that differs +significantly from the last value. More precisely, this is the largest +<i>c</i> such that <br /><br /> +|((v<sub><i>n</i></sub> - v<sub><i>c</i></sub>) / v<sub><i>n</i></sub>) + * 100| > <span +class="command"><diff tolerance></span>. + +<br /><br /> +Observe how the above formula implies that a <i>positive</i> current change +difference suggests a (possible) performance <i>regression</i> since the +latest execution of the benchmark spent more "resources" than the execution +that produced the current change. + +<br /><br /> +<b>Current Change Stability</b> +<br /><br /> + +The current change <i>c</i> is said to be <i>stable</i> if and only if both +v<sub><i>n</i></sub> and v<sub><i>c</i></sub> are stable. + +<br /><br /> <b>Trend</b> <br /><br /> @@ -488,8 +511,8 @@ the value at time <i>t</i> + <i>dt</i> is expected to be <br /><br /> -(Observe that positive and negative <i>β</i> values indicate decreased -and increased performance respectively.) +Observe that positive and negative <i>β</i> values indicate a decrease +and increase in performance respectively. <br /><br /> @@ -506,25 +529,38 @@ last value respectively. </ol> <br /> +<b>Limiting the output size</b> +<br /><br /> -If the <span class="command">-statonly</span> option is passed, only the -latest result (if any) of the time series and the result representing the -current change (if any) will be listed after the statistics. +If <span class="command"><max size></span> is +non-negative, it limits the part of the input time series to pass to the +output. The oldest part will be dropped if necessary. </td> <td> <pre> -currChange: \ - <index of \ - the current change, \ - or -1 if no such \ - change exists> -currChangeStable: \ - <true or false> \ -regression: \ - <a b or -1 if \ - regression cannot \ - be computed> +<index (0 = first) \ + of the current \ + change in the \ + results list \ + below, or -1> +<current change \ + stability (true, \ + false, or -1)> +<timestamp of the \ + current change, \ + or -1> +<sha1 (ditto)> +<value (ditto)> +<iterations (ditto)> +<last value \ + stability (true, \ + false, or -1)> +<a, or -1 if \ + a regression \ + cannot be \ + computed> +<b (ditto)> <timestamp 1> <sha1 1> <value 1> @@ -540,20 +576,30 @@ regression: \ Content-type: text/json { - "currChange": - <index of the current change>, + "currChange": { + "index": <index in results + list, or -1>, + "stable": <true or false>, + "result": ["<timestamp>", + "<sha1>", + "<value>", + "<iterations>"] + }, // Property omitted if no - // such change exists + // current change exists - "currChangeStable": - <true or false>, + "lastValue": { + "stable": <true or false> + }, + // Property omitted if no + // last value exists "regression": [<a>, <b>], // Property omitted if the // regression cannot be // computed - "items": [ + "results": [ ["<timestamp 1>", "<sha1 1>", "<value 1>", @@ -569,11 +615,289 @@ Content-type: text/json </td> </tr> + +<!-- ---------------------------------------------------------------------------------- --> +<tr> +<td> +<pre class="command"> +./bmclient <b>get rankedbenchmarks</b> \ + <metric> <platform> <host> \ + <git repo> <git branch> \ + {-timestamp \ + <first Unix timestamp> | \ + -sha1 <first sha1>} \ + {-timestamp \ + <last Unix timestamp> | \ + -sha1 <last sha1>} \ + <diff tolerance> \ + <stab tolerance> \ + <ranking> <scope> <max size> +</pre> +</td> +<td class="commandDescr"> + +For each benchmark within a specific context, this command computes the +<i>current change</i>. The output includes the subset of benchmarks whose +current changes are considered most important by some user-defined criteria. + +<br /><br /> + +For each benchmark within the specified context and snapshot range, the +statistics are computed in exactly the same way as in the <span +class="command"><b>get history</b></span> command. See this command for a +description of the types of statistics to be computed as well as how to +specify the snapshot range itself. + +<br /><br /> +<b>Controlling the output list</b> +<br /><br /> + +The output consists of a list of benchmarks along with statistics for each +one. The size and contents of this list is controlled by +<span class="command"><ranking></span>, +<span class="command"><scope></span>, and +<span class="command"><max size></span>. + +<br /><br /> +The <span class="command"><ranking></span> defines which of two +benchmarks is considered the most important one. The value +<code><b>worst</b></code> indicates that a positive <i>current change</i> +value is more important than a negative one (in other words: a performance +<i>decrease</i> is more important than a performance <i>increase</i>). The +value <code><b>best</b></code> indicates the exact opposite: performance +increases rank over decreases. Finally, the value +<code><b>absolute</b></code> indicates that a large current change (regardless +of the sign) is more important than a small one (in other words: it is the +<i>magnitude</i> of the current change that counts). + +<br /><br /> + +A stable current change always ranks over a non-stable one (regardless of the +value of <span class="command"><ranking></span>), while an existing +current change always ranks over a non-existing one. Two benchmarks without +any current change are ranked in an arbitrary order. + +<br /><br /> + +The <span class="command"><scope></span> determines if the benchmarks +should be sorted as one big list (value is <code><b>global</b></code>) or as +one list per test function (value is <code><b>testFunction</b></code>). The +final output consists of the topmost <span +class="command"><max size></span> benchmarks of the list(s). + +<br /><br /> + +If the <span class="command"><max size></span> is a non-negative +integer, it serves to limit the size of the output list(s) as described above +(note: a value of zero is allowed, but would obviously make little sense). + +A negative value for <span class="command"><max size></span> +specifies "no size limit", but the output is still grouped and sorted +according to <span class="command"><scope></span> and <span +class="command"><ranking></span>. + +<br /><br /> + +Observe that unless <span class="command"><max size></span> is 0, +the <code><b>testFunction</b></code> scope guarantees that all test functions +appear in the output. Conversely, the <code><b>global</b></code> scope may +result in one or more test functions being dropped from the output (i.e. a +small number of test functions may contain current change values that are more +important than other test functions). + +</td> +<td> +<pre> +<span style="color:red"><i>not supported yet</i></span> +</pre> +<td> +<pre> +Content-type: text/json + +{ + "benchmarks": [ + { + "name": [ + "<test case>", + "<test function>", + "<data tag>" + ], + + "lastValue": [ + <timestamp>, + "<sha1>", + <value>, + <iterations>, + <stable> + ], + // Property omitted if no + // last value exists + + "currChange": [ + <timestamp>, + "<sha1>", + <value>, + <iterations>, + <stable> + ], + // Property omitted if no + // current change exists + + "regression": [<a>, <b>] + // Property omitted if the + // regression cannot be + // computed + }, + ... + ] +} +</pre> +</td> +</tr> + + +<!-- ---------------------------------------------------------------------------------- --> +<tr> +<td> +<pre class="command"> +./bmclient <b>get rankedbenchmarks2</b> \ + <metric> <platform> <host> \ + <git repo 1> <git branch 1> \ + <git repo 2> <git branch 2> \ + <diff tolerance> \ + <stab tolerance> \ + <ranking> <scope> <max size> +</pre> +</td> +<td class="commandDescr"> + +For each benchmark in two specific branches within a specific context, this +command computes the difference between the last result in each branch. The +output includes the subset of benchmarks whose differences are considered most +important by some user-defined criteria. + +<br /><br /> +<b>Controlling the output list</b> +<br /><br /> + +The output consists of a list of benchmarks along with basic statistics for +the last value in each branch. The size and contents of this list is +controlled in exactly the same way as in the <span class="command"><b>get +rankedbenchmarks</b></span> command except that the subject of ranking is now +the difference between the last result of each branch (rather than the current +change). The difference is defined like this: + +<br /><br /> +(v<sub><i>2</i></sub> - v<sub><i>1</i></sub>) / v<sub><i>1</i></sub> +<br /><br /> + +where v<sub><i>1</i></sub> and v<sub><i>2</i></sub> denote the last value of +branch 1 and branch 2 respectively. A positive difference thus indicates that +branch 2 currently performs worse than branch 1 (since branch 2 spends more +"resources" running the benchmark). + +</td> +<td> +<pre> +<span style="color:red"><i>not supported yet</i></span> +</pre> +<td> +<pre> +Content-type: text/json + +{ + "benchmarks": [ + { + "name": [ + "<test case>", + "<test function>", + "<data tag>" + ], + + "lastValue1": [ + <timestamp>, + "<sha1>", + <value>, + <iterations>, + <stable> + ], + + "lastValue2": [ + <timestamp>, + "<sha1>", + <value>, + <iterations>, + <stable> + ] + }, + ... + ] +} +</pre> +</td> +</tr> + + +<!-- ---------------------------------------------------------------------------------- --> +<tr> +<td> +<pre class="command"> +./bmclient <b>get stats</b> \ + <metric> <platform> <host> \ + <git repo> <git branch> \ + {-timestamp \ + <first Unix timestamp> | \ + -sha1 <first sha1>} \ + {-timestamp \ + <last Unix timestamp> | \ + -sha1 <last sha1>} \ + <diff tolerance> \ + <stab tolerance> +</pre> +</td> +<td class="commandDescr"> + +Computes basic statistics for the benchmarks within a certain context. + +<br /><br /> + +See the <span class="command"><b>get history</b></span> command for a +definition of <i>snapshot range</i>, <i>current change</i>, <i>current change +stability</i>, <i>difference tolerance</i>, and <i>stability tolerance</i>. + +<td> +<pre> +<span style="color:red"><i>not supported yet</i></span> +</pre> +<td> +<pre> +Content-type: text/json + +{ + "nBenchmarks": + <number of benchmarks found>, + "nCurrChanges": + <number of current changes found>, + "nStableCurrChanges": + <number of stable current + changes found>, + "minStableCurrChange": + <minimum stable current change>, + "avgStableCurrChange": + <average stable current change>, + "maxStableCurrChange": + <maximum stable current change> +} +</pre> +</td> +</tr> + + <!-- ---------------------------------------------------------------------------------- --> <tr> <td> <pre class="command"> -./bmclient get value <test case> \ +./bmclient <b>get value</b> <test case> \ <test function> <data tag> \ <metric> <file> </pre> @@ -597,7 +921,7 @@ test function, data tag, and metric from the given file. <tr> <td> <pre class="command"> -./bmclient get server +./bmclient <b>get server</b> </pre> </td> <td class="commandDescr"> diff --git a/doc/bmproto.html b/doc/bmproto.html index e6a96e3..7fca78c 100644 --- a/doc/bmproto.html +++ b/doc/bmproto.html @@ -264,8 +264,7 @@ Behaves similarly to GetBenchmarks (see this). <tr> <td class="requestName">GetHistory</td> <td class="requestDescr"> -Lists the raw results within a particular historical range for a particular -context. +Lists statistics and history for a specific benchmark. <br /><br /> <b>Note:</b> For each snapshot (defining the beginning and end of the range @@ -277,8 +276,19 @@ while the second timestamp refers to the latest snapshot that is not later than the specified value. <br /><br /> -<b>Note:</b> The currChange and/or a,b attributes of the <statistics> -tag will be empty if the corresponding values could not be computed. +<b>Notes:</b> +<ul> +<li>The <currChange> tag is omitted altogether if no current change + exists (within the specified range).</li> +<li>A non-negative value for the <i>index</i> attribute in the + <currChange> tag refers to the index (0 = first) of the current change + in the list of <result> tags. (</b>A negative value means + that the list of <result> tags doesn't include the current change.) +<li>The <lastValue> tag is omitted altogether if no last value + exists (i.e. the specified range is empty).</li> +<li>The <regression> tag is omitted altogether if the regression + cannot be computed.</li> +</ul> </td> @@ -300,9 +310,11 @@ tag will be empty if the corresponding values could not be computed. <td> <pre> <reply type="GetHistory"> - <statistics currChange="..." - currChangeStable="..." - a="..." b="..." /> + <currChange index="..." stable="..." + timestamp="..." sha1="..." + value="..." iterations="..." /> + <lastValue stable="..." /> + <regression a="..." b="..." /> <result timestamp="..." sha1="..." value="..." iterations="..." /> <result timestamp="..." sha1="..." @@ -315,6 +327,139 @@ tag will be empty if the corresponding values could not be computed. </tr> +<!-- ---------------------------------------------------------------------------------- --> +<tr> +<td class="requestName">GetRankedBenchmarks</td> +<td class="requestDescr"> +Lists the benchmarks with the most important <i>current change</i> values. + +<br /><br /> +The snapshot range (timestamp{1..2} / sha1{1..2}) is defined as for the <span +class="requestName">GetHistory</span> request (see this). + +<br /><br /> +For each benchmark in the reply message, each of the <lastValue>, +<currChange>, and <regression> is included only if the value +exists (or can be computed in the case of the regression). + +</td> + +<td> +<pre> +<request type="GetRankedBenchmarks"> + <context metric="..." + platform="..." host="..." + gitRepo="..." gitBranch="..." + timestamp1="..." sha11="..." + timestamp2="..." sha12="..." + diffTolerance="..." stabTolerance="..." + ranking="..." scope="..." + maxSize="..." /> +</request> +</pre> +</td> + +<td> +<pre> +<reply type="GetRankedBenchmarks"> + <benchmark testCase="..." + testFunction="..." dataTag="..."> + <lastValue timestamp="..." + sha1="..." value="..." + iterations="..." stable="..." /> + <currChange timestamp="..." + sha1="..." value="..." + iterations="..." stable="..." /> + <regression a="..." b="..." /> + </benchmark> + ... +</reply> +</pre> +</td> + +</tr> + + +<!-- ---------------------------------------------------------------------------------- --> +<tr> +<td class="requestName">GetRankedBenchmarks2</td> +<td class="requestDescr"> +Lists the benchmarks with the most important <i>last value differences</i> for +two branches. + +</td> + +<td> +<pre> +<request type="GetRankedBenchmarks2"> + <context metric="..." + platform="..." host="..." + gitRepo1="..." gitBranch1="..." + gitRepo2="..." gitBranch2="..." + diffTolerance="..." stabTolerance="..." + ranking="..." scope="..." + maxSize="..." /> +</request> +</pre> +</td> + +<td> +<pre> +<reply type="GetRankedBenchmarks2"> + <benchmark testCase="..." + testFunction="..." dataTag="..."> + <lastValue1 timestamp="..." + sha1="..." value="..." + iterations="..." stable="..." /> + <lastValue2 timestamp="..." + sha1="..." value="..." + iterations="..." stable="..." /> + </benchmark> + ... +</reply> +</pre> +</td> + +</tr> + + +<!-- ---------------------------------------------------------------------------------- --> +<tr> +<td class="requestName">GetStats</td> +<td class="requestDescr"> +Computes basic statistics for the benchmarks within a certain context. + +</td> + +<td> +<pre> +<request type="GetStats"> + <context metric="..." + platform="..." host="..." + gitRepo="..." gitBranch="..." + timestamp1="..." sha11="..." + timestamp2="..." sha12="..." + diffTolerance="..." stabTolerance="..." /> +</request> +</pre> +</td> + +<td> +<pre> +<reply type="GetStats"> + <stats nBenchmarks="..." + nCurrChanges="..." + nStableCurrChanges="..." + minStableCurrChange="..." + avgStableCurrChange="..." + maxStableCurrChange="..." /> +</reply> +</pre> +</td> + +</tr> + + </table> </body> |