Fix crashes and non-portable functionality in QDBusDemarshaller QByteArray extraction

QDBusArgument QByteArray extraction operator and QDBusDemarshaller that
implements the extraction do not check the type of the extracted value.
When extracting a QByteArray when the value actually is e.g. a struct of
mixed types the byte array extraction will crash as it attempts to extract
the struct data as a fixed array.

The fix adds DBus type checks to QDBusArgument byte array extraction
operator implementations.
The checks invalidate extracting arrays of other types than bytes to a
QByteArray that worked with the unchecked implementation. The rationale
for this restriction is
1) extracting a QByteArray to a variant checks already that the array
   element type is byte
2) Results of extracting arrays of types wider than a byte to a QByteArray
   are architecture-dependent making such code inherently non-portable.

Task-number: QTBUG-22840
Change-Id: Ie20f2adc06c697a68055c803215fb408568fdd90
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>

2 files changed, 12 insertions, 2 deletions

diff --git a/src/dbus/qdbusargument_p.h b/src/dbus/qdbusargument_p.h
index df2b2a011b..4327e74267 100644
--- a/src/dbus/qdbusargument_p.h
+++ b/src/dbus/qdbusargument_p.h
@@ -213,6 +213,7 @@ private:
     QDBusObjectPath toObjectPathUnchecked();
     QDBusSignature toSignatureUnchecked();
     QStringList toStringListUnchecked();
+    QByteArray toByteArrayUnchecked();
 };
 
 inline QDBusMarshaller *QDBusArgumentPrivate::marshaller()
diff --git a/src/dbus/qdbusdemarshaller.cpp b/src/dbus/qdbusdemarshaller.cpp
index b7e363a8d2..96729fd603 100644
--- a/src/dbus/qdbusdemarshaller.cpp
+++ b/src/dbus/qdbusdemarshaller.cpp
@@ -272,7 +272,7 @@ QVariant QDBusDemarshaller::toVariantInternal()
         switch (q_dbus_message_iter_get_element_type(&iterator)) {
         case DBUS_TYPE_BYTE:
             // QByteArray
-            return toByteArray();
+            return toByteArrayUnchecked();
         case DBUS_TYPE_STRING:
             return toStringListUnchecked();
         case DBUS_TYPE_DICT_ENTRY:
@@ -339,7 +339,7 @@ QStringList QDBusDemarshaller::toStringList()
         return QStringList();
 }
 
-QByteArray QDBusDemarshaller::toByteArray()
+QByteArray QDBusDemarshaller::toByteArrayUnchecked()
 {
     DBusMessageIter sub;
     q_dbus_message_iter_recurse(&iterator, &sub);
@@ -350,6 +350,15 @@ QByteArray QDBusDemarshaller::toByteArray()
     return QByteArray(data,len);
 }
 
+QByteArray QDBusDemarshaller::toByteArray()
+{
+    if (q_dbus_message_iter_get_arg_type(&iterator) == DBUS_TYPE_ARRAY
+            && q_dbus_message_iter_get_element_type(&iterator) == DBUS_TYPE_BYTE) {
+        return toByteArrayUnchecked();
+    }
+    return QByteArray();
+}
+
 bool QDBusDemarshaller::atEnd()
 {
     // dbus_message_iter_has_next is broken if the list has one single element