Q(Font|Color|File)Dialog: Fix several UBs (invalid cast/member call) in Private::canBeNativeDialog()

Found by UBSan:

  qfontdialog_p.h:77:5: runtime error: downcast of address 0x7ffc3ceadc90 which does not point to an object of type 'QFontDialog'
  0x7ffc3ceadc90: note: object is of type 'QDialog'
   fc 7f 00 00  38 5f a8 27 fc 2a 00 00  60 e2 14 02 00 00 00 00  10 61 a8 27 fc 2a 00 00  00 00 00 00
                ^~~~~~~~~~~~~~~~~~~~~~~
                vptr for 'QDialog'
    #0 0x2afc24d29490 in QFontDialogPrivate::q_func() const qfontdialog_p.h:77
    #1 0x2afc24d29490 in QFontDialogPrivate::canBeNativeDialog() const qfontdialog.cpp:1033
    #2 0x2afc24c93f56 in QDialog::setVisible(bool) qdialog.cpp:696
    #3 0x2afc24c7b27a in QDialog::~QDialog() qdialog.cpp:357
    #4 0x2afc24d286a6 in QFontDialog::~QFontDialog() qfontdialog.cpp:339
    #5 0x2afc24d481a2 in QFontDialogPrivate::getFont(bool*, QFont const&, QWidget*, QString const&, QFlags<QFontDialog::FontDialogOption>) qfontdialog.cpp:402
    #6 0x2afc24d483f1 in QFontDialog::getFont(bool*, QWidget*) qfontdialog.cpp:396
    #7 0x407652 in tst_QFontDialog::testGetFont() tst_qfontdialog.cpp:120

  qcolordialog.cpp:86:5: runtime error: downcast of address 0x7ffdf50c1ec0 which does not point to an object of type 'QColorDialog'
  0x7ffdf50c1ec0: note: object is of type 'QDialog'
   fd 7f 00 00  d8 6e c7 23 b7 2a 00 00  50 c1 af 01 00 00 00 00  b0 70 c7 23 b7 2a 00 00  00 00 1a 1e
                ^~~~~~~~~~~~~~~~~~~~~~~
                vptr for 'QDialog'
    #0 0x2ab720e4ec97 in QColorDialogPrivate::q_func() const qcolordialog.cpp:86
    #1 0x2ab720e4ec97 in QColorDialogPrivate::canBeNativeDialog() const qcolordialog.cpp:1865
    #2 0x2ab720e84ed6 in QDialog::setVisible(bool) qdialog.cpp:696
    #3 0x2ab720e6c1fa in QDialog::~QDialog() qdialog.cpp:357
    #4 0x2ab720e2b276 in QColorDialog::~QColorDialog() qcolordialog.cpp:2187
    #5 0x2ab720e5e2c6 in QColorDialog::getColor(QColor const&, QWidget*, QString const&, QFlags<QColorDialog::ColorDialogOption>) qcolordialog.cpp:2148
    #6 0x2ab720e5e473 in QColorDialog::getRgba(unsigned int, bool*, QWidget*) qcolordialog.cpp:2176
    #7 0x407180 in tst_QColorDialog::testGetRgba() tst_qcolordialog.cpp:118

  qfiledialog_p.h:112:5: runtime error: downcast of address 0x7ffd6858cc60 which does not point to an object of type 'QFileDialog'
  0x7ffd6858cc60: note: object is of type 'QDialog'
   a1 2b 00 00  d8 1e 5e 0c a1 2b 00 00  b0 af 01 20 a1 2b 00 00  b0 20 5e 0c a1 2b 00 00  00 00 46 00
                ^~~~~~~~~~~~~~~~~~~~~~~
                vptr for 'QDialog'
    #0 0x2ba10980a9e7 in QFileDialogPrivate::q_func() const qfiledialog_p.h:112
    #1 0x2ba10980a9e7 in QFileDialogPrivate::canBeNativeDialog() const qfiledialog.cpp:695
    #2 0x2ba1097efe36 in QDialog::setVisible(bool) qdialog.cpp:696
    #3 0x2ba1097d715a in QDialog::~QDialog() qdialog.cpp:357
    #4 0x2ba109854c4b in QFileDialog::~QFileDialog() qfiledialog.cpp:380
    #5 0x4179dc in tst_QFiledialog::init() tst_qfiledialog.cpp:175

Fix by replacing Q_Q with the the equivalent expression for QDialog.

We can't re-use QDialogPrivate::q_func() here, since that is private,
and probably should stay like that.

Also fix an invalid member call in
QColorDialogPrivate::canBeNativeDialog():

  qcolordialog.cpp:2050:5: runtime error: member call on address 0x7ffdf50c1ec0 which does not point to an object of type 'QColorDialog'
  0x7ffdf50c1ec0: note: object is of type 'QDialog'
   fd 7f 00 00  d8 6e c7 23 b7 2a 00 00  50 c1 af 01 00 00 00 00  b0 70 c7 23 b7 2a 00 00  00 00 1a 1e
                ^~~~~~~~~~~~~~~~~~~~~~~
                vptr for 'QDialog'
    #0 0x2ab720e4e5ea in QColorDialog::options() const qcolordialog.cpp:2050
    #1 0x2ab720e4e8c8 in QColorDialogPrivate::canBeNativeDialog() const qcolordialog.cpp:1870
    #2 0x2ab720e84ed6 in QDialog::setVisible(bool) qdialog.cpp:696
    #3 0x2ab720e6c1fa in QDialog::~QDialog() qdialog.cpp:357
    #4 0x2ab720e2b276 in QColorDialog::~QColorDialog() qcolordialog.cpp:2187
    #5 0x2ab720e5e2c6 in QColorDialog::getColor(QColor const&, QWidget*, QString const&, QFlags<QColorDialog::ColorDialogOption>) qcolordialog.cpp:2148
    #6 0x2ab720e5e473 in QColorDialog::getRgba(unsigned int, bool*, QWidget*) qcolordialog.cpp:2176
    #7 0x407180 in tst_QColorDialog::testGetRgba() tst_qcolordialog.cpp:118

by accessing the data member directly instead of through the Public API.

Fix the same code in QFileDialog, even though the autotest coverage is
too limited for UBSan to point that one out explicitly.

Change-Id: Idd278744961435e417d91fb2f89b6d91a94e0c71
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>

1 files changed, 3 insertions, 1 deletions

diff --git a/src/widgets/dialogs/qfontdialog.cpp b/src/widgets/dialogs/qfontdialog.cpp
index 6644f26ad0..3ef844346b 100644
--- a/src/widgets/dialogs/qfontdialog.cpp
+++ b/src/widgets/dialogs/qfontdialog.cpp
@@ -1033,7 +1033,9 @@ void QFontDialog::done(int result)
 
 bool QFontDialogPrivate::canBeNativeDialog() const
 {
-    Q_Q(const QFontDialog);
+    // Don't use Q_Q here! This function is called from ~QDialog,
+    // so Q_Q calling q_func() invokes undefined behavior (invalid cast in q_func()).
+    const QDialog * const q = static_cast<const QDialog*>(q_ptr);
     if (nativeDialogInUse)
         return true;
     if (q->testAttribute(Qt::WA_DontShowOnScreen))