diff options
author | jasplin <qt-info@nokia.com> | 2009-08-28 11:01:23 +0200 |
---|---|---|
committer | jasplin <qt-info@nokia.com> | 2009-08-28 11:01:23 +0200 |
commit | c6b4e8f370fc2fbef13680c7d932db0640cef09b (patch) | |
tree | 114901475d14b33dec51bd85fab7355eff742efc /doc | |
parent | 92cb237907cae16f53b5e441a12950c5239883b6 (diff) |
Renamed API so that all commands/requests are either 'put ...' or 'get ...'.
Diffstat (limited to 'doc')
-rw-r--r-- | doc/bmc.html | 113 | ||||
-rw-r--r-- | doc/bmcp.html | 130 | ||||
-rw-r--r-- | doc/notes.txt | 73 |
3 files changed, 121 insertions, 195 deletions
diff --git a/doc/bmc.html b/doc/bmc.html index 744e140..364ccfa 100644 --- a/doc/bmc.html +++ b/doc/bmc.html @@ -58,10 +58,12 @@ or via the <code>BMCSERVER</code> environment variable, e.g.: <h3>Commands</h3> -<b>Note:</b> The 'Command' column in the table below shows the raw mode -command line syntax. In CGI mode the initial arguments (passed in environment -variables) are converted internally to a format that is equivalent to the raw -mode command line. +<b>Note:</b> +The 'Command' column in the table below shows the raw mode command line +syntax. In CGI mode the initial arguments (passed in environment variables) +are converted internally to a format that is equivalent to the raw mode +command line. + <br /><br /> <table border=1> @@ -76,7 +78,7 @@ mode command line. <tr> <td> <pre class="command"> -./bmc add results <file> \ +./bmc put results <file> \ <platform> <host> \ <git repo> [<git dir>] </pre> @@ -119,7 +121,7 @@ repository). <tr> <td> <pre class="command"> -./bmc list snapshots \ +./bmc get snapshots \ <test case> <test function> \ <data tag> <metric> \ <platform> <host> \ @@ -145,7 +147,7 @@ Lists snapshots available for a particular context. <tr> <td> <pre class="command"> -./bmc list benchmarks \ +./bmc get benchmarks \ [-metric <metric>] \ [-platform <platform>] \ [-host <host>] \ @@ -183,7 +185,7 @@ exists in combination with those values. <tr> <td> <pre class="command"> -./bmc list metrics \ +./bmc get metrics \ [-benchmark <test case> \ <test function> <data tag>] \ [-platform <platform>] \ @@ -193,7 +195,7 @@ exists in combination with those values. </td> <td class="commandDescr"> Lists available metrics. -Behaves similarly to <span class="command">./bmc list benchmarks ...</span> (see this). +Behaves similarly to <span class="command">./bmc get benchmarks ...</span> (see this). </td> <td> <pre> @@ -209,7 +211,7 @@ Behaves similarly to <span class="command">./bmc list benchmarks ...</span> (see <tr> <td> <pre class="command"> -./bmc list platforms \ +./bmc get platforms \ [-benchmark <test case> \ <test function> <data tag>] \ [-metric <metric>] \ @@ -219,7 +221,7 @@ Behaves similarly to <span class="command">./bmc list benchmarks ...</span> (see </td> <td class="commandDescr"> Lists available platforms. -Behaves similarly to <span class="command">./bmc list benchmarks ...</span> (see this). +Behaves similarly to <span class="command">./bmc get benchmarks ...</span> (see this). </td> <td> <pre> @@ -235,7 +237,7 @@ Behaves similarly to <span class="command">./bmc list benchmarks ...</span> (see <tr> <td> <pre class="command"> -./bmc list hosts \ +./bmc get hosts \ [-benchmark <test case> \ <test function> <data tag>] \ [-metric <metric>] \ @@ -245,7 +247,7 @@ Behaves similarly to <span class="command">./bmc list benchmarks ...</span> (see </td> <td class="commandDescr"> Lists available hosts. -Behaves similarly to <span class="command">./bmc list benchmarks ...</span> (see this). +Behaves similarly to <span class="command">./bmc get benchmarks ...</span> (see this). </td> <td> <pre> @@ -261,7 +263,7 @@ Behaves similarly to <span class="command">./bmc list benchmarks ...</span> (see <tr> <td> <pre class="command"> -./bmc list branches \ +./bmc get branches \ [-benchmark <test case> \ <test function> <data tag>] \ [-metric <metric>] \ @@ -271,7 +273,7 @@ Behaves similarly to <span class="command">./bmc list benchmarks ...</span> (see </td> <td class="commandDescr"> Lists available branches. -Behaves similarly to <span class="command">./bmc list benchmarks ...</span> (see this). +Behaves similarly to <span class="command">./bmc get benchmarks ...</span> (see this). </td> <td> <pre> @@ -289,22 +291,24 @@ Behaves similarly to <span class="command">./bmc list benchmarks ...</span> (see <tr> <td> <pre class="command"> -./bmc compare history <test case> \ +./bmc get history <test case> \ <test function> <data tag> \ <metric> <platform> <host> \ <git repo> <git branch> \ - {-timestamp <Unix timestamp> | \ - -sha1 <sha1>} + {-timestamp <first Unix timestamp> | \ + -sha1 <first sha1>} \ + {-timestamp <last Unix timestamp> | \ + -sha1 <last sha1>} </pre> </td> <td class="commandDescr"> -Compares results as a history for a particular context since a particular -snapshot. +Lists the raw results within a particular historical range for a particular context. <br /><br /> -If the <span class="command">-timestamp</span> option is used, the values -<i>first</i> and <i>last</i> may be used to refer to the first and last -available result respectively. Otherwise, the timestamp refers to the earliest -result that is not earlier than the specified value. +If a <span class="command">-timestamp</span> option is used, the values +<i>first</i> and <i>last</i> specify the first and last +available snapshot respectively. Otherwise, the first timestamp refers to the earliest +snapshot that is not earlier than the specified value, while the second +timestamp refers to the latest snapshot that is not later than the specified value. <br /><br /> If the <span class="command">-sha1</span> option is used, a snapshot with this exact value must exist. @@ -329,36 +333,26 @@ exact value must exist. <tr> <td> <pre class="command"> -./bmc compare branches <test case> \ +./bmc get trend <test case> \ <test function> <data tag> \ <metric> <platform> <host> \ - <git repo 1> <git branch 1> \ - {-timestamp <Unix timestamp 1> | \ - -sha1 <sha1 1>} \ - <git repo 2> <git branch 2> \ - {-timestamp <Unix timestamp 2> | \ - -sha1 <sha1 2>} + <git repo> <git branch> \ + {-timestamp <first Unix timestamp> | \ + -sha1 <first sha1>} \ + {-timestamp <last Unix timestamp> | \ + -sha1 <last sha1>} </pre> </td> <td class="commandDescr"> -Compares the results of two branches. +Computes a trend estimation of the results within a particular historical +range for a particular context. This may simplify the task of detecting +performance regressions. <br /><br /> The <span class="command">-timestamp</span> and <span -class="command">-sha1</span> options behaves as for <span -class="command">./bmc compare history ...</span> (see this). -</td> -<td> -<pre> -<timestamp 1> -<sha1 1> -<value 1> -<iterations 1> -<timestamp 2> -<sha1 2> -<value 2> -<iterations 2> -</pre> +class="command">-sha1</span> options behave as for <span +class="command">./bmc analyze dump ...</span> (see this). </td> +<td><span style="color:red">2 B DEFINED</span></td> <td><span style="color:red">2 B DONE</span></td> </tr> @@ -366,39 +360,24 @@ class="command">./bmc compare history ...</span> (see this). <tr> <td> <pre class="command"> -./bmc compare direct <test case> \ +./bmc get value <test case> \ <test function> <data tag> \ - <metric> <platform> <host> \ - <git repo> <git branch> \ - {-timestamp <Unix timestamp> | \ - -sha1 <sha1>} \ - {-value <value> <iterations> | \ - -file <file>} + <metric> <file> </pre> </td> <td class="commandDescr"> -Compares a result in the database to an explicitly given result. -<br /><br /> -If the <span class="command">-file</span> option is used, the file is assumed -to have the same XML structure as the one produced by QTestLib and contain a -value that matches the given (test case, test function, data tag, metric) -combination. +Extracts the value (if any) that matches the given combination of test case, +test function, data tag, and metric from the given file. <br /><br /> -The <span class="command">-timestamp</span> and <span -class="command">-sha1</span> options behaves as for <span -class="command">./bmc compare history ...</span> (see this). +<b>Note:</b> This command is supported in raw mode only. </td> <td> <pre> -<timestamp> -<sha1> <value> <iterations> -<value (explicit)> -<iterations (explicit)> </pre> </td> -<td><span style="color:red">2 B DONE</span></td> +<td><i>not supported</i></td> </tr> </table> diff --git a/doc/bmcp.html b/doc/bmcp.html index de7f21b..63b4146 100644 --- a/doc/bmcp.html +++ b/doc/bmcp.html @@ -48,14 +48,14 @@ successfully processed. <!-- ---------------------------------------------------------------------------------- --> <tr> -<td class="requestName">AddResults</td> +<td class="requestName">PutResults</td> <td class="requestDescr"> Adds new results to the database. </td> <td> <pre> -<request type="AddResults"> +<request type="PutResults"> <context platform="..." host="..." gitRepo="..." gitBranch="..." timestamp="..." sha1="..." @@ -71,21 +71,21 @@ Adds new results to the database. </td> <td> -<reply type="AddResults" /> +<reply type="PutResults" /> </td> </tr> <!-- ---------------------------------------------------------------------------------- --> <tr> -<td class="requestName">ListSnapshots</td> +<td class="requestName">GetSnapshots</td> <td class="requestDescr"> Lists snapshots available for a particular context. </td> <td> <pre> -<request type="ListSnapshots"> +<request type="GetSnapshots"> <context testCase="..." testFunction="..." dataTag="..." metric="..." platform="..." host="..." @@ -96,7 +96,7 @@ Lists snapshots available for a particular context. <td> <pre> -<reply type="ListSnapshots"> +<reply type="GetSnapshots"> <snapshot timestamp="..." sha1="..." /> <snapshot timestamp="..." sha1="..." /> ... @@ -108,7 +108,7 @@ Lists snapshots available for a particular context. <!-- ---------------------------------------------------------------------------------- --> <tr> -<td class="requestName">ListBenchmarks</td> +<td class="requestName">GetBenchmarks</td> <td class="requestDescr"> Lists available benchmarks. Additional entity values (metric, platform, host, or git repo/branch) may be specified as non-empty values to restrict the @@ -120,7 +120,7 @@ any result yet. <td> <pre> -<request type="ListBenchmarks"> +<request type="GetBenchmarks"> <context metric="..." platform="..." host="..." gitRepo="..." gitBranch="..." /> @@ -130,7 +130,7 @@ any result yet. <td> <pre> -<reply type="ListBenchmarks"> +<reply type="GetBenchmarks"> <benchmark testCase="..." testFunction="..." dataTag="..." /> <benchmark testCase="..." testFunction="..." @@ -144,14 +144,14 @@ any result yet. <!-- ---------------------------------------------------------------------------------- --> <tr> -<td class="requestName">ListMetrics</td> +<td class="requestName">GetMetrics</td> <td class="requestDescr"> -Behaves similarly to ListBenchmarks (see this). +Behaves similarly to GetBenchmarks (see this). </td> <td> <pre> -<request type="ListMetrics"> +<request type="GetMetrics"> <context testCase="..." testFunction="..." dataTag="..." platform="..." host="..." gitRepo="..." @@ -162,7 +162,7 @@ Behaves similarly to ListBenchmarks (see this). <td> <pre> -<reply type="ListMetrics"> +<reply type="GetMetrics"> <metric name="..." /> <metric name="..." /> ... @@ -174,14 +174,14 @@ Behaves similarly to ListBenchmarks (see this). <!-- ---------------------------------------------------------------------------------- --> <tr> -<td class="requestName">ListPlatforms</td> +<td class="requestName">GetPlatforms</td> <td class="requestDescr"> -Behaves similarly to ListBenchmarks (see this). +Behaves similarly to GetBenchmarks (see this). </td> <td> <pre> -<request type="ListPlatforms"> +<request type="GetPlatforms"> <context testCase="..." testFunction="..." dataTag="..." metric="..." host="..." gitRepo="..." @@ -192,7 +192,7 @@ Behaves similarly to ListBenchmarks (see this). <td> <pre> -<reply type="ListPlatforms"> +<reply type="GetPlatforms"> <platform name="..." /> <platform name="..." /> ... @@ -204,14 +204,14 @@ Behaves similarly to ListBenchmarks (see this). <!-- ---------------------------------------------------------------------------------- --> <tr> -<td class="requestName">ListHosts</td> +<td class="requestName">GetHosts</td> <td class="requestDescr"> -Behaves similarly to ListBenchmarks (see this). +Behaves similarly to GetBenchmarks (see this). </td> <td> <pre> -<request type="ListHosts"> +<request type="GetHosts"> <context testCase="..." testFunction="..." dataTag="..." metric="..." platform="..." gitRepo="..." @@ -222,7 +222,7 @@ Behaves similarly to ListBenchmarks (see this). <td> <pre> -<reply type="ListHosts"> +<reply type="GetHosts"> <host name="..." /> <host name="..." /> ... @@ -234,14 +234,14 @@ Behaves similarly to ListBenchmarks (see this). <!-- ---------------------------------------------------------------------------------- --> <tr> -<td class="requestName">ListBranches</td> +<td class="requestName">GetBranches</td> <td class="requestDescr"> -Behaves similarly to ListBenchmarks (see this). +Behaves similarly to GetBenchmarks (see this). </td> <td> <pre> -<request type="ListBranches"> +<request type="GetBranches"> <context testCase="..." testFunction="..." dataTag="..." metric="..." platform="..." host="..." /> @@ -251,7 +251,7 @@ Behaves similarly to ListBenchmarks (see this). <td> <pre> -<reply type="ListBranches"> +<reply type="GetBranches"> <branch gitRepo="..." gitBranch="..." /> <branch gitRepo="..." gitBranch="..." /> ... @@ -262,59 +262,28 @@ Behaves similarly to ListBenchmarks (see this). <!-- ---------------------------------------------------------------------------------- --> <tr> -<td class="requestName">CompareHistory</td> +<td class="requestName">GetHistory</td> <td class="requestDescr"> -Compares results as a history for a particular context since a particular snapshot. -<b>Note:</b> Only one of <i>timestamp</i> or <i>sha1</i> is required. If -<i>timestamp</i> is used, the values <i>first</i> and <i>last</i> may be used -to refer to the first and last available result respectively. Otherwise, -<i>timestamp</i> refers to the earliest result that is not earlier than the -specified value. +Lists the raw results within a particular historical range for a particular +context. +<br /><br /> +<b>Note:</b> For each snapshot (defining the beginning and end of the range +respectively) only one of <i>timestamp</i> or <i>sha1</i> is required. If +<i>timestamp</i> is used, the values <i>first</i> and <i>last</i> specify the +first and last available snapshot respectively. Otherwise, the first timestamp +refers to the earliest snapshot that is not earlier than the specified value, +while the second timestamp refers to the latest snapshot that is not later than +the specified value. </td> <td> <pre> -<request type="CompareHistory"> +<request type="GetHistory"> <context testCase="..." testFunction="..." dataTag="..." metric="..." platform="..." host="..." gitRepo="..." gitBranch="..." - timestamp="..." sha1="..." /> -</request> -</pre> -</td> - -<td> -<pre> -<reply type="CompareHistory"> - <result timestamp="..." sha1="..." - value="..." iterations="..." /> - <result timestamp="..." sha1="..." - value="..." iterations="..." /> - ... -</reply> -</pre> -</td> - -</tr> - -<!-- ---------------------------------------------------------------------------------- --> -<tr> -<td class="requestName">CompareBranches</td> -<td class="requestDescr"> -Compares the results of two branches. See note above regarding semantics of -the timestamp and sha1 attributes. -</td> - -<td> -<pre> -<request type="CompareBranches"> - <context testCase="..." testFunction="..." - dataTag="..." metric="..." - platform="..." host="..." - gitRepo1="..." gitBranch1="..." timestamp1="..." sha11="..." - gitRepo2="..." gitBranch2="..." timestamp2="..." sha12="..." /> </request> </pre> @@ -322,11 +291,12 @@ the timestamp and sha1 attributes. <td> <pre> -<reply type="CompareBranches"> +<reply type="GetHistory"> <result timestamp="..." sha1="..." value="..." iterations="..." /> <result timestamp="..." sha1="..." value="..." iterations="..." /> + ... </reply> </pre> </td> @@ -335,33 +305,33 @@ the timestamp and sha1 attributes. <!-- ---------------------------------------------------------------------------------- --> <tr> -<td class="requestName">CompareDirect</td> +<td class="requestName">GetTrend</td> <td class="requestDescr"> -Compares a result in the database to an explicitly given result. See note -above regarding semantics of the timestamp and sha1 attributes. +Computes a trend estimation of the results within a particular historical +range for a particular context. This may simplify the task of detecting +performance regressions. +<br /><br /> +See note above regarding semantics of the timestamp and sha1 attributes. </td> <td> <pre> -<request type="CompareDirect"> +<request type="GetTrend"> <context testCase="..." testFunction="..." dataTag="..." metric="..." platform="..." host="..." gitRepo="..." gitBranch="..." - timestamp="..." sha1="..." - value="..." iterations="..." /> + timestamp1="..." sha11="..." + timestamp2="..." sha12="..." /> </request> </pre> </td> <td> <pre> -<reply type="CompareDirect"> - <result timestamp="..." sha1="..." - value="..." iterations="..." /> - <result value="..." iterations="..." /> -</reply> +<reply type="GetTrend" /> </pre> +<span style="color:red">2 B DEFINED</span> </td> </tr> diff --git a/doc/notes.txt b/doc/notes.txt index e5ed814..fdac218 100644 --- a/doc/notes.txt +++ b/doc/notes.txt @@ -31,7 +31,7 @@ Note: The -sha1 option requires exact matching and is thus less powerful than -t ---------------------------------- Use case 1: Add new results: - ./bmc add -results <xml file> <platform> <host> <git repo> [<git dir>] + ./bmc put results <xml file> <platform> <host> <git repo> [<git dir>] Notes: @@ -59,12 +59,12 @@ Use case 1: Add new results: Example: - ./bmc add -results results.xml linux-g++ barbarella qt master $QTDIR + ./bmc put results results.xml linux-g++ barbarella qt master $QTDIR ---------------------------------- Use case 2: List available snapshots for a given branch: - ./bmc list -snapshots BMCONTEXT <branch> + ./bmc get snapshots BMCONTEXT <branch> Example: @@ -74,101 +74,78 @@ Use case 2: List available snapshots for a given branch: ---------------------------------- Use case 3: List available benchmarks (subject to optional restrictions): - ./bmc list -benchmarks [-metric <metric>] [-platform <platform>] [-host <host>] \ + ./bmc get benchmarks [-metric <metric>] [-platform <platform>] [-host <host>] \ [-branch <branch>] Example 1: - ./bmc list -benchmarks + ./bmc get benchmarks Example 2: - ./bmc list -benchmarks -platform linux-g++ + ./bmc get benchmarks -platform linux-g++ Example 3: - ./bmc list -benchmarks -platform linux-g++ -host barbarella -metric walltime \ + ./bmc get benchmarks -platform linux-g++ -host barbarella -metric walltime \ -branch qt master ---------------------------------- Use case 4: List available metrics (subject to optional restrictions): - ./bmc list -metrics [-benchmark <test case> <test function> <data tag>] \ + ./bmc get metrics [-benchmark <test case> <test function> <data tag>] \ [-platform <platform>] [-host <host>] [-branch <branch>] Example 1: - ./bmc list -metrics + ./bmc get metrics Example 2: - ./bmc list -metrics -benchmark tst_QWidget updateOpaque 10x10x1 \ + ./bmc get metrics -benchmark tst_QWidget updateOpaque 10x10x1 \ -platform linux-g++ -host barbarella -branch qt master ---------------------------------- Use case 5: List available platforms (subject to optional restrictions): - ./bmc list -platforms [-benchmark <test case> <test function> <data tag>] \ + ./bmc get platforms [-benchmark <test case> <test function> <data tag>] \ [-metric <metric>] [-host <host>] [-branch <branch>] ---------------------------------- Use case 6: List available hosts (subject to optional restrictions): - ./bmc list -hosts [-benchmark <test case> <test function> <data tag>] \ + ./bmc get hosts [-benchmark <test case> <test function> <data tag>] \ [-metric <metric>] [-platform <platform>] [-branch <branch>] ---------------------------------- Use case 7: List available branches (subject to optional restrictions): - ./bmc list -platforms [-benchmark <test case> <test function> <data tag>] \ + ./bmc get platforms [-benchmark <test case> <test function> <data tag>] \ [-metric <metric>] [-platform <platform>] [-host <host>] ---------------------------------- -Use case 8: Compare results as a history of a given branch since a given snapshot: +Use case 8: List the raw results within a particular historical range. - ./bmc compare -history BMCONTEXT <branch> SNAPSHOT + ./bmc get history BMCONTEXT <branch> SNAPSHOT1 SNAPSHOT2 Example 1: - ./bmc compare -history tst_QWidget updateOpaque 10x10x1 walltime \ - linux-g++ barbarella qt master -timestamp first + ./bmc get history tst_QWidget updateOpaque 10x10x1 walltime \ + linux-g++ barbarella qt master -timestamp first -timestamp last Example 2 (1250146917 corresponds to the date Thu Aug 13 09:01:57 2009 +0200): - ./bmc compare -history tst_QWidget updateOpaque 10x10x1 walltime \ - linux-g++ barbarella qt master -timestamp 1250146917 + ./bmc get history tst_QWidget updateOpaque 10x10x1 walltime \ + linux-g++ barbarella qt master -timestamp 1250146917 -timestamp last Example 3: - ./bmc compare -history tst_QWidget updateOpaque 10x10x1 walltime \ - linux-g++ barbarella qt master -sha1 3bad2f608813968d85c2ea51c1bd75dca7f7e22a + ./bmc get history tst_QWidget updateOpaque 10x10x1 walltime \ + linux-g++ barbarella qt master -timestamp first \ + -sha1 3bad2f608813968d85c2ea51c1bd75dca7f7e22a ---------------------------------- -Use case 9: Compare the results of two branches: +Use case 9: Compute trend values within a particular historical range + (to detect perfomance regressions etc.). - ./bmc compare -branches BMCONTEXT <branch1> SNAPSHOT1 <branch2> SNAPSHOT2 - - Example: - - ./bmc compare -branches tst_QWidget updateOpaque 10x10x1 walltime \ - linux-g++ barbarella qt 45 -timestamp last qt 46 -timestamp last - ----------------------------------- -Use case 10: Compare a result of a given branch directly to an unregistered - result (i.e. one not yet in the database): - - ./bmc compare -direct BMCONTEXT <branch> SNAPSHOT (<value> <iterations>) | <xml file> - - Note: If an xml file is specified (i.e. there's just one instead of two arguments after - the SNAPSHOT argument), it is assumed to contain a value/iterations pair that - matches the benchmark and metric in the BMCONTEXT. - - Example 1: - - ./bmc compare -direct tst_QWidget updateOpaque 10x10x1 walltime \ - linux-g++ barbarella qt master -timestamp last 40 1024 - - Example 2: - - ./bmc compare -direct tst_QWidget updateOpaque 10x10x1 walltime \ - linux-g++ barbarella qt master -timestamp last results.xml + ./bmc get trend BMCONTEXT <branch> SNAPSHOT1 SNAPSHOT2 |