macOS: Rework worksWhenModal and update on modal session change

Instead of basing the worksWhenModal property on the window type, which
will include both windows inside the current modal session (as intend),
but also windows below the current modal session (wrongly), we check
to see if the window is a transient child of the current top level
modal window.

Going via NSApp.modalWindow means we also catch cases where the top
level modal session is run by a native window, not part of the modal
session stack in the Cocoa event dispatcher.

The new logic relies on windows such as popups, dialogs, etc to set
the correct transient parent, but this seems to be the case already.

To ensure the window tag is also updated, we call setWorksWhenModal
on modal session changes. We could change worksWhenModal into e.g.
shouldWorkWhenModal and always use the setter, but that would mean
the initial window tag update in [NSWindow _commonAwake] would pick
up the incorrect value. And if the window tag is not updated after
that due to the workaround in [QNSPanel setWorksWhenModal:] being
compiled out, the window would not be possible to order front,
which is worse than being able to order front a window with
worksWhenModal=NO.

Fixes: QTBUG-76654
Task-number: QTBUG-71480
Change-Id: I38b14422d274dcc03b4c7d5ef87066e282ed9111
Reviewed-by: Timur Pocheptsov <timur.pocheptsov@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>

1 files changed, 12 insertions, 0 deletions

diff --git a/src/plugins/platforms/cocoa/qcocoawindowmanager.mm b/src/plugins/platforms/cocoa/qcocoawindowmanager.mm
index 9c45d8c7fc..5e218157c2 100644
--- a/src/plugins/platforms/cocoa/qcocoawindowmanager.mm
+++ b/src/plugins/platforms/cocoa/qcocoawindowmanager.mm
@@ -100,6 +100,18 @@ void QCocoaWindowManager::modalSessionChanged()
             }
         }
     }
+
+    // Our worksWhenModal implementation is declarative and will normally be picked
+    // up by AppKit when needed, but to make sure AppKit also reflects the state
+    // in the window tag, so that the window can be ordered front by clicking it,
+    // we need to explicitly call setWorksWhenModal.
+    for (id window in NSApp.windows) {
+        if ([window isKindOfClass:[QNSPanel class]]) {
+            auto *panel = static_cast<QNSPanel *>(window);
+            // Call setter to tell AppKit that our state has changed
+            [panel setWorksWhenModal:panel.worksWhenModal];
+        }
+    }
 }
 
 static void initializeWindowManager() { Q_UNUSED(QCocoaWindowManager::instance()); }