From 2c9ac4fc3fe0b8f39e7f7a93894b56e49fd3d887 Mon Sep 17 00:00:00 2001 From: Thiago Macieira Date: Wed, 31 Oct 2018 17:06:21 -0700 Subject: RCC: Add support for Zstandard compression [ChangeLog][RCC] RCC now supports compressing content using the Zstandard (https://zstd.net) algorithm. Compared to zlib, it compresses better for the same CPU time, so this algorithm is the default. To go back to the previous algorithm, pass command-line option --compress-algo=zlib. Compression levels range from 1 (fastest, least compression) to 19 (slowest, best compression). Level 0 tells the library to choose an implementation-defined default. \ The default compression level is "heuristic" (level -1): under this mode, RCC will attempt a very fast compression (level 1) and check if the file was sufficiently compressed. If it was, then RCC will compress again using an implementation-defined level. The following are the 4 biggest files we store as resources in qtbase: Orig Size Name 2197605 src/corelib/mimetypes/mime/packages/freedesktop.org.xml 2462423 tests/auto/corelib/tools/qchar/data/NormalizationTest.txt 6878748 tests/auto/other/qcomplextext/data/BidiCharacterTest.txt 7959972 tests/auto/other/qcomplextext/data/BidiTest.txt The current RCC (zlib, level -1 "default" and level 9), produces for those files: L(-1) Compr. L9 Compr. Decomp. Name Ratio CPU time Ratio CPU time CPU time BidiCharacterTest.txt 16.9:1 106.1ms 17.2:1 789.3ms 5.1ms BidiTest.txt 6.3:1 228.0ms 6.1:1 1646.3ms 10.9ms freedesktop.org.xml 7.0:1 17.5ms 7.1:1 53.6ms 2.6ms NormalizationTest.txt 5.8:1 41.2ms 5.9:1 256.4ms 3.4ms Zstandard produces the following for levels 1 ("check"), 14 ("store") and 19 ("best"): L1 Compr. L14 Compr. L19 Compr. Decomp Name Ratio time Ratio time Ratio CPU time time BidiCharacterTest.txt 15.8:1 8.0ms 26.1:1 168.9ms 49.2:1 2504.7ms 3.8ms BidiTest.txt 8.2:1 17.0ms 8.7:1 323.9ms 14.9:1 1700.9ms 12.1ms freedesktop.org.xml 6.7:1 4.0ms 8.7:1 63.3ms 9.5:1 642.5ms 1.7ms NormalizationTest.txt 5.7:1 5.0ms 7.5:1 54.0ms 8.4:1 447.3ms 3.0ms This shows use of zstd at the default RCC level settings always produce smaller outputs compared to the current zlib-based defaults, with roughly 50% CPU increase. It also produces better results at less CPU time than the best compression zlib has to offer. More importantly, the decompression time reduces in all cases (the numbers listed are for max compression, with slightly better results for the defaults). For the sake of comparison, the same files compressed with libxz at levels 3 and 6: Level 3 Level 6 Decompr. Name Ratio CPU Ratio CPU time BidiCharacterTest.txt 28.5:1 109.1ms 42.9:1 1390.5ms 16.7ms BidiTest.txt 10.7:1 281.0ms 18.4:1 2333.1ms 43.6ms freedesktop.org.xml 9.1:1 62.0ms 10.2:1 499.1ms 12.0ms NormalizationTest.txt 10.2:1 75.5ms 13.2:1 417.6ms 14.7ms LZMA at level 3 consumes roughly the same CPU time as Zstd at level 14 and produces incrementally smaller results, but the decompression time increases considerably. It's not a good trade-off for the Qt resource system. Change-Id: I343f2beed55440a7ac0bfffd1562d754bd71d964 Reviewed-by: Lars Knoll Reviewed-by: Oswald Buddenhagen --- src/corelib/doc/src/resource-system.qdoc | 67 +++++++++++++++++++++++++++----- 1 file changed, 58 insertions(+), 9 deletions(-) (limited to 'src/corelib/doc') diff --git a/src/corelib/doc/src/resource-system.qdoc b/src/corelib/doc/src/resource-system.qdoc index 9b6b613f79..69ec5e556b 100644 --- a/src/corelib/doc/src/resource-system.qdoc +++ b/src/corelib/doc/src/resource-system.qdoc @@ -100,6 +100,9 @@ See the QLocale documentation for a description of the format to use for locale strings. + See QFileSelector for an additional mechanism to select locale-specific + resources, in addition to the ability to select OS-specific and other + features. \section2 External Binary Resources @@ -143,24 +146,70 @@ \section1 Compression - Resources are compressed by default (in the \c ZIP format). It is - possible to turn off compression. This can be useful if your - resources already contain a compressed format, such as \c .png - files. You do this by giving the \c {-no-compress} command line - argument. + \c rcc attempts to compress the content to optimize disk space usage in the + final binaries. By default, it will perform a heuristic check to determine + whether compressing is worth it and will store the content uncompressed if + it fails to sufficiently compress. To control the threshold, you can use + the \c {-threshold} option, which tells \c rcc the percentage of the + original file size that must be gained for it to store the file in + compressed form. + + \code + rcc -threshold 25 myresources.qrc + \endcode + + The default value is "70", indicating that the compressed file must be 70% + smaller than the original (no more than 30% of the original file size). + + It is possible to turn off compression, if desired. This can be useful if + your resources already contain a compressed format, such as \c .png files, + and you do not want to incur the CPU cost at build time to confirm that it + can't be compressed. Another reason is if disk usage is not a problem and + the application would prefer to keep the content as clean memory pages at + runtime. You do this by giving the \c {-no-compress} command line argument. \code rcc -no-compress myresources.qrc \endcode - \c rcc also gives you some control over the compression. You can - specify the compression level and the threshold level to consider - while compressing files, for example: + \c rcc also gives you some control over the compression level and + compression algorithm, for example: \code - rcc -compress 2 -threshold 3 myresources.qrc + rcc -compress 2 -compress-algo zlib myresources.qrc \endcode + \c rcc supports the following compression algorithms and compression + levels: + + \list + \li \c{best}: use the best algorithm among the ones below, at its highest + compression level, to achieve the most compression at the expense of + using a lot of CPU time during compilation. This value is useful in the + XML file to indicate a file should be most compressed, regardless of + which algorithms \c rcc supports. + + \li \c{zstd}: use the \l{Zstandard}{https://zstd.net} library to compress + contents. Valid compression levels range from 1 to 19, 1 is least + compression (least CPU time) and 19 is the most compression (most CPU + time). The default level is 14. A special value of 0 tells the \c{zstd} + library to choose an implementation-defined default. + + \li \c{zlib}: use the \l{zlib}{https://zlib.net} library to compress + contents. Valid compression levels range from 1 to 9, with 1the least + compression (least CPU time) and 9 the most compression (most CPU time). + The special value 0 means "no compression" and should not be used. The + default is implementation-defined, but usually is level 6. + + \li \c{none}: no compression. This is the same as the \c{-no-compress} + option. + \endlist + + Support for both Zstandard and zlib are optional. If a given library was + not detected at compile time, attempting to pass \c {-compress-algo} for + that library will result in an error. The default compression algorithm is + \c zstd if it is enabled, \c zlib if not. + \section1 Using Resources in the Application In the application, resource paths can be used in most places -- cgit v1.2.3