Q(Color|File)Dialog: Fix several UBs (invalid cast/member call) in Private::canBeNativeDialog()

Found by UBSan:

  qcolordialog.cpp:86:5: runtime error: downcast of address 0x7ffdf50c1ec0 which does not point to an object of type 'QColorDialog'
  0x7ffdf50c1ec0: note: object is of type 'QDialog'
   fd 7f 00 00  d8 6e c7 23 b7 2a 00 00  50 c1 af 01 00 00 00 00  b0 70 c7 23 b7 2a 00 00  00 00 1a 1e
                ^~~~~~~~~~~~~~~~~~~~~~~
                vptr for 'QDialog'
    #0 0x2ab720e4ec97 in QColorDialogPrivate::q_func() const qcolordialog.cpp:86
    #1 0x2ab720e4ec97 in QColorDialogPrivate::canBeNativeDialog() const qcolordialog.cpp:1865
    #2 0x2ab720e84ed6 in QDialog::setVisible(bool) qdialog.cpp:696
    #3 0x2ab720e6c1fa in QDialog::~QDialog() qdialog.cpp:357
    #4 0x2ab720e2b276 in QColorDialog::~QColorDialog() qcolordialog.cpp:2187
    #5 0x2ab720e5e2c6 in QColorDialog::getColor(QColor const&, QWidget*, QString const&, QFlags<QColorDialog::ColorDialogOption>) qcolordialog.cpp:2148
    #6 0x2ab720e5e473 in QColorDialog::getRgba(unsigned int, bool*, QWidget*) qcolordialog.cpp:2176
    #7 0x407180 in tst_QColorDialog::testGetRgba() tst_qcolordialog.cpp:118

  qfiledialog_p.h:112:5: runtime error: downcast of address 0x7ffd6858cc60 which does not point to an object of type 'QFileDialog'
  0x7ffd6858cc60: note: object is of type 'QDialog'
   a1 2b 00 00  d8 1e 5e 0c a1 2b 00 00  b0 af 01 20 a1 2b 00 00  b0 20 5e 0c a1 2b 00 00  00 00 46 00
                ^~~~~~~~~~~~~~~~~~~~~~~
                vptr for 'QDialog'
    #0 0x2ba10980a9e7 in QFileDialogPrivate::q_func() const qfiledialog_p.h:112
    #1 0x2ba10980a9e7 in QFileDialogPrivate::canBeNativeDialog() const qfiledialog.cpp:695
    #2 0x2ba1097efe36 in QDialog::setVisible(bool) qdialog.cpp:696
    #3 0x2ba1097d715a in QDialog::~QDialog() qdialog.cpp:357
    #4 0x2ba109854c4b in QFileDialog::~QFileDialog() qfiledialog.cpp:380
    #5 0x4179dc in tst_QFiledialog::init() tst_qfiledialog.cpp:175

Fix by replacing Q_Q with the the equivalent expression for QDialog.

We can't re-use QDialogPrivate::q_func() here, since that is private,
and probably should stay like that.

Also fix an invalid member call in
QColorDialogPrivate::canBeNativeDialog():

  qcolordialog.cpp:2050:5: runtime error: member call on address 0x7ffdf50c1ec0 which does not point to an object of type 'QColorDialog'
  0x7ffdf50c1ec0: note: object is of type 'QDialog'
   fd 7f 00 00  d8 6e c7 23 b7 2a 00 00  50 c1 af 01 00 00 00 00  b0 70 c7 23 b7 2a 00 00  00 00 1a 1e
                ^~~~~~~~~~~~~~~~~~~~~~~
                vptr for 'QDialog'
    #0 0x2ab720e4e5ea in QColorDialog::options() const qcolordialog.cpp:2050
    #1 0x2ab720e4e8c8 in QColorDialogPrivate::canBeNativeDialog() const qcolordialog.cpp:1870
    #2 0x2ab720e84ed6 in QDialog::setVisible(bool) qdialog.cpp:696
    #3 0x2ab720e6c1fa in QDialog::~QDialog() qdialog.cpp:357
    #4 0x2ab720e2b276 in QColorDialog::~QColorDialog() qcolordialog.cpp:2187
    #5 0x2ab720e5e2c6 in QColorDialog::getColor(QColor const&, QWidget*, QString const&, QFlags<QColorDialog::ColorDialogOption>) qcolordialog.cpp:2148
    #6 0x2ab720e5e473 in QColorDialog::getRgba(unsigned int, bool*, QWidget*) qcolordialog.cpp:2176
    #7 0x407180 in tst_QColorDialog::testGetRgba() tst_qcolordialog.cpp:118

by accessing the data member directly instead of through the Public API.

Fix the same code in QFileDialog, even though the autotest coverage is
too limited for UBSan to point that one out explicitly.

This commit amends abe8b4ab9b5243b477c72f3e900d4f6cca79b5c5, in which
it should have been included in the first place...

Change-Id: Iff0538eba61d2381359f0b61f35918d643f7aa0c
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>

2 files changed, 8 insertions, 4 deletions

diff --git a/src/widgets/dialogs/qcolordialog.cpp b/src/widgets/dialogs/qcolordialog.cpp
index c4cc21cbe5..8e699e87c5 100644
--- a/src/widgets/dialogs/qcolordialog.cpp
+++ b/src/widgets/dialogs/qcolordialog.cpp
@@ -1862,12 +1862,14 @@ void QColorDialogPrivate::retranslateStrings()
 
 bool QColorDialogPrivate::canBeNativeDialog() const
 {
-    Q_Q(const QColorDialog);
+    // Don't use Q_Q here! This function is called from ~QDialog,
+    // so Q_Q calling q_func() invokes undefined behavior (invalid cast in q_func()).
+    const QDialog * const q = static_cast<const QDialog*>(q_ptr);
     if (nativeDialogInUse)
         return true;
     if (q->testAttribute(Qt::WA_DontShowOnScreen))
         return false;
-    if (q->options() & QColorDialog::DontUseNativeDialog)
+    if (options->options() & QColorDialog::DontUseNativeDialog)
         return false;
 
     QLatin1String staticName(QColorDialog::staticMetaObject.className());
diff --git a/src/widgets/dialogs/qfiledialog.cpp b/src/widgets/dialogs/qfiledialog.cpp
index bc2de899f5..3aa9052917 100644
--- a/src/widgets/dialogs/qfiledialog.cpp
+++ b/src/widgets/dialogs/qfiledialog.cpp
@@ -695,12 +695,14 @@ void QFileDialogPrivate::emitFilesSelected(const QStringList &files)
 
 bool QFileDialogPrivate::canBeNativeDialog() const
 {
-    Q_Q(const QFileDialog);
+    // Don't use Q_Q here! This function is called from ~QDialog,
+    // so Q_Q calling q_func() invokes undefined behavior (invalid cast in q_func()).
+    const QDialog * const q = static_cast<const QDialog*>(q_ptr);
     if (nativeDialogInUse)
         return true;
     if (q->testAttribute(Qt::WA_DontShowOnScreen))
         return false;
-    if (q->options() & QFileDialog::DontUseNativeDialog)
+    if (options->options() & QFileDialog::DontUseNativeDialog)
         return false;
 
     QLatin1String staticName(QFileDialog::staticMetaObject.className());