| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
|
|
| |
It is now possible to save, load, and delete
a performance index configuration (i.e. all the
settings in the 'Index' section in the BM web app).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch constitutes a slightly different and also simpler
approach to computing the performance index.
The hierarchical index structure is gone for now,
and the index is computed essentially like this:
1: Specify a base timestamp BT.
2: Specify a set of evaluation timestamps ET.
3: Specify a window size for median smoothing.
4: Specify a set of filters to effectively narrow
down the set of result histories that may contribute
to the index (there are currently five separate filters:
metric, platform, host, branch, and test case.
Zero or more values may be specified for each one).
5: Evaluate the index at a given evaluation timestamp ET(i) by
taking the arithmetic mean of the differences between the
values at ET(i) and BT for the individual result histories.
Both values are median-smoothed using a right-aligned window.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch moves the timestamp attribute from
the snapshot table to the result table. This
is done for two purposes:
1) Retrieving a pure result history (not including the
SHA-1 values) becomes quicker, since a join is avoided
(simply fetch all (timestamp, value) pairs matching a
given BM context).
2) It constitutes the first of two steps towards supporting
multiple result recordings per (SHA-1, BM context)
combination (the other step will be to remove the current
uniqueness constraint on that combination!).
The (future) use case is to improve statistical confidence
by e.g. taking the median value of the set of values for
a certain (SHA-1, BM context) combination as the representative
value for the combination.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
more commands.
It turned out that we need to keep a time series
of index trees for the performance index after all:
There will be one base index tree (at timestamp -1, i.e.
older than any result), and one index tree for the
timestamps where the current index tree was changed
(i.e. essentially one or more result histories and/or
sub-indexes added or removed).
The index is evaluated on the basis of
the closest index tree earlier than the requested
evaluation time. This ensures that the correct
history constants are used. (The history constants
in turn ensure continuity of the index values
upon adding or removing result histories and/or
sub-indexes).
The following bmclient commands are implemented so far:
- index set base - sets the index base from an XML file
- index get tree - lists a certain index tree as XML
- index get names - lists the available index names
- index get values - evaluates an index at a set of timestamps;
dumps the result to stdout
- index get plot - same, except the result is plotted in an image
- get result - verifies a result ID in the XML file
|
| |
|
|
|
| |
The performance index support does not work yet.
It remains to implement the functions in indextree.cpp.
|
| |
|
|
|
|
|
|
| |
The output from the 'get history' command now
includes the result IDs (i.e. the primary keys
in the database). These IDs will be used to
identify baseline results in the XML structure
that represents the performance index hierarchy.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The most important change of this patch
is a restructring of the database schema
in order to better support current
and future use cases.
In particular, the 'result' table has been reduced
in size (which is important since this table is
the one that will grow on a regular basis), and
a 'bmcontext' table has been introduced as an extra
structural level that is expected to make certain
queries faster than they would have otherwise been
(due to reduced need for complex join expressions
etc.).
|
| |
|
|
|
|
|
|
|
| |
Time ranges are now specified using either
Unix timestamps or 'first' or 'last'. The
option of specifying a SHA-1 value was
considered not to be particularly useful
in practice. This option has therefore
been obsoleted to simplify the code.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch adds support for negative result values
to BM (thus allowing all real values).
A typical use case for negative values is a benchmark
that measures the difference between two implementations
of the same problem/task.
In order to support negative values, the formula for
computing the relative difference between v1 and v2
has been changed from
(a - b) / (b + 1)
(or (a - b) / (a + 1) )
to
(a == b) ? 0 : ((a - b) / max(|a|, |b|))
Note that adding 1 in the first formula was necessarry to
allow for zero values. This is no longer needed.
Note also that the new formula doesn't allow direct
selection of the normalization base (i.e. the denominator),
but rather automatically choosing the one with the largest
magnitude. This is assumed not to be a problem, and also
has the interesting property that the absolute value of
the expression is bounded by 2.
This patch also renames the term "current change" to "last difference".
|
| |
|
|
|
|
|
|
|
| |
The 'get rankedbenchmarks' request now considers only
benchmarks matching a certain filter. E.g. specifying
'Graphics*' as the test case filter and '*' for the
test function and data tag filters, ensures that only
benchmarks with test cases beginning with "Graphics"
will be processed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The context part of the configuration of
a benchmark summary page may be interactively
manipulated using HTML <select> menus for
the individual components (metric, platform,
host, and branch). This patch both speeds up
the interactive selection and fixes a bug
that caused the selection from going haywire
(due to improper handling of asynchronous
communication).
This patch also involves a small visibility
change: Rather than just hiding the
non-matching options in each menu, all
available values are now shown, but only
the matching ones are selectable
(i.e. enabled.). WARNING: Manipulating
the disabled attribute of an <option>
element doesn't seem to be supported by
all web browsers.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The most important change of this commit is that the
'iterations' component of the result value is gone.
Now the result value represented by just the
final real-valued result, rather than two integer
components: (accumulated) result and number of
iterations.
NOTE: The old file format is still accepted,
and when an iterations component is present, the result
component will be divided by this one.
The other significant change of this commit is that
indexes are added to the database scheme to speed up
queries. This only affects the 'initdb'
command-line option, though.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Branch 1 is now compared against branch 2 instead of
vice versa. This seems to be more intuitive: Branch 1
is your main focus, and you want to check how it compares
against another branch (100% corresponds to the latest
result in branch 2, so -5% means that branch 1 is 5% better
than branch 2 and so on).
This patch also includes a few cosmetic changes, like swapping
the sides for vertical labels in the plot (percentages are now
shown on the right hand side).
|
| |
|
|
|
|
|
|
|
| |
Passing the current timestamp as an argument to the
'get history*' requests makes it possible to use the
current time of the web browser as the observation
time instead of the current time of the bmclient host.
In general this makes more sense. (Pass -1 to use the
local time of the bmclient host anyway).
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change improves the support for 'details' pages in several ways:
- The page is generated on the server side
(i.e. by bmclient) instead of in the browser.
- The page is refered to by a stand-alone URL that
can be opened to regenerate the page.
- The plot (showing the history graph(s)) and table
is shown in the same page.
- The plot is generated directly as a PNG image by
Qt graphics view instead of by the HTML <canvas> tag.
- When comparing two branches, the history graphs
for both branches appear in the same plot for easy
comparison. (The time scale may be shared or not!)
Moreover, the history values for both branches
are shown next to each other in two tables.
|
| |
|
|
|
|
|
|
| |
This patch removes the assumption that benchmark results are always
positive. Some metrics, like the 'events' metric, may produce
zero valued results. To support zero values, the ratio between
two results r1 and r2 is expressed as (r1 + 1) / (r2 + 1) instead
of just r1 / r2.
|
| |
|
|
|
|
|
|
|
| |
This commit essentially adds the following request types:
- get rankedbenchmarks
- get rankedbenchmarks2
- get stats
Plus a lot of other (minor) modifications.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
#Please enter the commit message for your changes. Lines starting
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
history' command.
Adding size and tolerance to the 'get history' command effectively eliminates the
need for a 'get trend' command (for now).
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
When adding new results to the database, any missing values
(such as test case, host etc.) should be automatically inserted.
This also eliminates the need for the commands/requests
for explicitly adding benchmarks, metrics, platforms, hosts,
and branches.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
| |
The following commands are now implemented:
add <all>
list -snapshots
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
| |
In particular, a tentative protocol specification (bmcp.txt) has been added.
|
| | |
|
