diff options
Diffstat (limited to 'tests/arthur/README')
| -rw-r--r-- | tests/arthur/README | 84 |
1 files changed, 0 insertions, 84 deletions
diff --git a/tests/arthur/README b/tests/arthur/README deleted file mode 100644 index 0178351b95..0000000000 --- a/tests/arthur/README +++ /dev/null @@ -1,84 +0,0 @@ -This is a simple regression testing framework for Arthur. - -There are three parts to it. - -1) datagenerator . which is the main part. it's used to - run all the tests, create output png files, - measure rendering performance and output it in - data.xml file. datagenerator reads in - framework.ini to figure out where to look for tests. - if the framework.ini file isn't in the current directory - you have to specify its location with the -framework - parameter. by default datagenerator generates output in the - current directory - to specify output directory pass in - -output parameter followed by a directory name. if the given - directory doesn't exist it will be created. - - it's recommended to always specify -iterations argument (followed - by the number). if the argument has been specified datagenerator - will try to render all testcases the given amount of times yielding - a lot more precise results. - - if one wants to test just a specified engine -engine argument - should be specified (followed by the desired engine name). - - individual testsuites can be run with -suite paramenter. - individual testcases from testcases can also be reran with - -testcase paramenter. - - given files can also be tested individually. if the option -file - is specified with a SVG file engines will be tested against just - that file and output will be written directly to stdout (not files). - it's useful when optimizing one engine with particular case in - mind. - - Example command line: - ./bin/datagenerator -framework data/framework.ini -output sampleout -iterations 5 -engine [ENGINE] - -2) htmlgenerator. which is used to generate html pages out of - generated output (generated by datagenerator). htmlgenerator also - reads in the framework.ini file. so the option -framework also - applies. but htmlgenerator takes a mandatory directory name as the - second argument. the mandatory directory name is the directory to - which you output the files generated by datagenerator (either - whatever followed the -output or the current directory) - - htmlgenerator will generate html pages that can be viewed to - manually inspect the results. - - Example command line: - ./bin/htmlgenerator -framework data/framework.ini sampleout - -3) performancediff. diffs the performance data between two output - directories. so assuming you already have data for all the engines, - you change something in an engine. to see the performance - difference run - ./bin/datagenerator -framework data/framework.ini -output newoutput -iterations 5 - and then - ./bin/performancediff sampleout newoutput - - The performance results will be printed out. - - The two first columns are in ms. The last one is a - percentage of difference. - - After the percentage a '+' or a '-' sign followed by a - number will be sometimes printed. - - '+' signifies the new results were mathematically better. '-' is of - course just the opposite. - the number following '+' is the minimum rendering time the engine - spent rendering the given file up till this moment. - the number following '-' is the maximum rendering time the engine - spent rendering the given file up till this moment. - - (if the current results are out of the scope from within maximum and - minimum number then the respective sign is printed out.) -------------------------------------------------------------------- - -Note that the generated output directories can be copied from one -machine to the other. htmlgenerator will generate webpages for all -engines with valid data.xml files. So if you want add NativeWin32 and -NativeMac engines you'll have to have someone with a windows and mac -run datagenerator with "-engine NativeWin32" or "-engine NativeMac" -options and then sending you the respective output directories. |