Port qUncompress() to zstream/inflate()

The zlib convenience API we've been using so far has two problems:

- On Windows-64, where sizeof(long) == 4, the use of ulong for sizes
  meant that we could not uncompress data compressed on other 64-bit
  platforms (Unix). While zstream also uses ulong, being a stream API,
  it allows feeding data in chunks. The total_in and total_out members
  are only required for gzip compression and are otherwise just
  informational. They're unsigned, so their overflow does not cause
  UB. In summary, using zstream + inflate() allows us to decompress
  more than 4GiB of data even on Windows-64.

- On all platforms, if the size hint in the header was too short, we'd
  double the output buffer size and try again, from scratch. Using
  zstream + inflate(), we still need to reallocate, but we can then
  let zlib pick up where it left off when it ran out of output buffer
  space. In all but the most pathological cases, copying the
  already-decoded data instead of re-decoding it again should be
  faster, esp. if QArrayData uses realloc() instead of malloc() +
  free() to grow the buffer.

We also now directly allocate at least as much output buffer as we
have input, to cut the first few rounds of reallocations when the
expectedSize was created, as qCompress still does, using modulo
arithmetic mod 4GiB instead of saturation arithmethic.

Factor the growing of the output buffer into a wrapper function,
flate(), which can be reused when porting qCompress().

This completely fixes the uncompression side of QTBUG-106542 and
QTBUG-104972.

Pick-to: 6.4 6.3 6.2
Task-number: QTBUG-104972
Task-number: QTBUG-106542
Change-Id: I97f55ea322c24db1ac48b31c16855bc91708e7e2
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>

1 files changed, 63 insertions, 0 deletions

diff --git a/tests/auto/corelib/text/qbytearray/tst_qbytearray.cpp b/tests/auto/corelib/text/qbytearray/tst_qbytearray.cpp
index b138609448..72f66640cd 100644
--- a/tests/auto/corelib/text/qbytearray/tst_qbytearray.cpp
+++ b/tests/auto/corelib/text/qbytearray/tst_qbytearray.cpp
@@ -12,6 +12,8 @@
 
 #include "../shared/test_number_shared.h"
 
+#include <q20iterator.h>
+
 #include <stdexcept>
 #include <string_view>
 
@@ -31,6 +33,7 @@ private slots:
     void qCompress();
     void qUncompressCorruptedData_data();
     void qUncompressCorruptedData();
+    void qUncompress4GiBPlus();
     void qCompressionZeroTermination();
 #endif
     void constByteArray();
@@ -300,6 +303,66 @@ void tst_QByteArray::qUncompressCorruptedData()
     QCOMPARE(res, QByteArray());
 }
 
+void tst_QByteArray::qUncompress4GiBPlus()
+{
+    // after three rounds, this decompresses to 4GiB + 1 'X' bytes:
+    constexpr uchar compressed_3x[] = {
+       0x00, 0x00, 0x1a, 0x76, 0x78, 0x9c, 0x63, 0xb0, 0xdf, 0xb4, 0xad, 0x62,
+       0xce, 0xdb, 0x3b, 0x0b, 0xf3, 0x26, 0x27, 0x4a, 0xb4, 0x3d, 0x34, 0x5b,
+       0xed, 0xb4, 0x41, 0xf1, 0xc0, 0x99, 0x2f, 0x02, 0x05, 0x67, 0x26, 0x88,
+       0x6c, 0x66, 0x71, 0x34, 0x62, 0x9c, 0x75, 0x26, 0xb1, 0xa0, 0xe5, 0xcc,
+       0xda, 0x94, 0x83, 0xc9, 0x05, 0x73, 0x0e, 0x3c, 0x39, 0xc2, 0xc7, 0xd0,
+       0xae, 0x38, 0x53, 0x7b, 0x87, 0xdc, 0x01, 0x91, 0x45, 0x59, 0x4f, 0xda,
+       0xbf, 0xca, 0xcc, 0x52, 0xdb, 0xbb, 0xde, 0xbb, 0xf6, 0xd3, 0x55, 0xff,
+       0x7d, 0x77, 0x0e, 0x1b, 0xf0, 0xa4, 0xdf, 0xcf, 0xdb, 0x5f, 0x2f, 0xf5,
+       0xd7, 0x7c, 0xfe, 0xbf, 0x3f, 0xbf, 0x3f, 0x9d, 0x7c, 0xda, 0x2c, 0xc8,
+       0xc0, 0xc0, 0xb0, 0xe1, 0xf1, 0xb3, 0xfd, 0xfa, 0xdf, 0x8e, 0x7d, 0xef,
+       0x7f, 0xb9, 0xc1, 0xc2, 0xae, 0x92, 0x19, 0x28, 0xf2, 0x66, 0xd7, 0xe5,
+       0xbf, 0xed, 0x93, 0xbf, 0x6a, 0x14, 0x7c, 0xff, 0xf6, 0xe1, 0xe8, 0xb6,
+       0x7e, 0x46, 0xa0, 0x90, 0xd9, 0xbb, 0xcf, 0x9f, 0x17, 0x37, 0x7f, 0xe5,
+       0x6f, 0xb4, 0x7f, 0xfe, 0x5e, 0xfd, 0xb6, 0x1d, 0x1b, 0x50, 0xe8, 0xc6,
+       0x8e, 0xe3, 0xab, 0x9f, 0xe6, 0xec, 0x65, 0xfd, 0x23, 0xb1, 0x4e, 0x7e,
+       0xef, 0xbd, 0x6f, 0xa6, 0x40, 0xa1, 0x03, 0xc7, 0xfe, 0x0a, 0xf1, 0x00,
+       0xe9, 0x06, 0x91, 0x83, 0x40, 0x92, 0x21, 0x43, 0x10, 0xcc, 0x11, 0x03,
+       0x73, 0x3a, 0x90, 0x39, 0xa3, 0x32, 0xa3, 0x32, 0xa3, 0x32, 0xa3, 0x32,
+       0xa3, 0x32, 0xa3, 0x32, 0xa3, 0x32, 0xa3, 0x32, 0xa3, 0x32, 0xa3, 0x32,
+       0xa3, 0x32, 0xa3, 0x32, 0xa3, 0x32, 0xa3, 0x32, 0xa3, 0x32, 0xa3, 0x32,
+       0xa3, 0x32, 0xa3, 0x32, 0xa3, 0x32, 0xa3, 0x32, 0xa3, 0x32, 0xa3, 0x32,
+       0xa3, 0x32, 0xa3, 0x32, 0xa3, 0x32, 0x34, 0x90, 0x99, 0xb6, 0x7e, 0xf5,
+       0xd3, 0xe9, 0xbf, 0x35, 0x13, 0xca, 0x8c, 0x75, 0xec, 0xec, 0xa4, 0x2f,
+       0x7e, 0x2d, 0xf9, 0xf3, 0xf0, 0xee, 0xea, 0xd5, 0xf5, 0xd3, 0x14, 0x57,
+       0x06, 0x00, 0x00, 0xb9, 0x1e, 0x35, 0xce
+    };
+
+    constexpr qint64 GiB = 1024LL * 1024 * 1024;
+
+    if constexpr (sizeof(qsizetype) == sizeof(int)) {
+        QSKIP("This is a 64-bit-only test.");
+    } else {
+
+        // 1st
+        auto c = ::qUncompress(std::data(compressed_3x), q20::ssize(compressed_3x));
+        QVERIFY(!c.isNull()); // check for decompression error
+
+        // 2nd
+        c = ::qUncompress(c);
+        QVERIFY(!c.isNull());
+
+        // 3rd
+        try {
+            c = ::qUncompress(c);
+            if (c.isNull())  // this step (~18MiB -> 4GiB) might have run out of memory
+                QSKIP("Failed to allocate enough memory.");
+        } catch (const std::bad_alloc &) {
+            QSKIP("Failed to allocate enough memory.");
+        }
+
+        QCOMPARE(c.size(), 4 * GiB + 1);
+        QCOMPARE(std::string_view{c}.find_first_not_of('X'),
+                 std::string_view::npos);
+    }
+}
+
 void tst_QByteArray::qCompressionZeroTermination()
 {
     QByteArray s = "Hello, I'm a string.";