macOS: Work around key view loop crossing NSWindow boundaries

AppKit will in some cases set up the key view loop for child views, even
if we don't set autorecalculatesKeyViewLoop, nor call recalculateKeyViewLoop
ourselves.

When a child window is promoted to a top level, AppKit will maintain the key
view loop between the views, even if these views now cross NSWindows, even
after we explicitly call recalculateKeyViewLoop. When the top level is then
hidden, AppKit will complain when -[NSView _setHidden:setNeedsDisplay:] tries
to transfer first responder by reading the nextValidKeyView, and it turns out
to live in a different window.

  ERROR: Setting <View: 0x145e0cfd0> as the first responder for window
  <Window: 0x10f904de0>, but it is in a different window (<Window: 0x147104f00>)!
  This would eventually crash when the view is freed. The first responder will
  be set to nil.

We mitigate this by a last second reset of the first responder, which is
what AppKit also falls back to.

It's unclear if the original situation of views having their nextKeyView
pointing to views in other windows is kosher or not, but that's left for
further investigations.

Pick-to: 6.5 6.6
Change-Id: I63636afbba85abf73a38db9701f32656c42c59cc
Reviewed-by: Timur Pocheptsov <timur.pocheptsov@qt.io>

1 files changed, 18 insertions, 0 deletions

diff --git a/src/plugins/platforms/cocoa/qcocoawindow.mm b/src/plugins/platforms/cocoa/qcocoawindow.mm
index dcdbaedc20..46f3564257 100644
--- a/src/plugins/platforms/cocoa/qcocoawindow.mm
+++ b/src/plugins/platforms/cocoa/qcocoawindow.mm
@@ -442,6 +442,24 @@ void QCocoaWindow::setVisible(bool visible)
             }
         }
 
+        // AppKit will in some cases set up the key view loop for child views, even if we
+        // don't set autorecalculatesKeyViewLoop, nor call recalculateKeyViewLoop ourselves.
+        // When a child window is promoted to a top level, AppKit will maintain the key view
+        // loop between the views, even if these views now cross NSWindows, even after we
+        // explicitly call recalculateKeyViewLoop. When the top level is then hidden, AppKit
+        // will complain when -[NSView _setHidden:setNeedsDisplay:] tries to transfer first
+        // responder by reading the nextValidKeyView, and it turns out to live in a different
+        // window. We mitigate this by a last second reset of the first responder, which is
+        // what AppKit also falls back to. It's unclear if the original situation of views
+        // having their nextKeyView pointing to views in other windows is kosher or not.
+        if (m_view.window.firstResponder == m_view && m_view.nextValidKeyView
+            && m_view.nextValidKeyView.window != m_view.window) {
+            qCDebug(lcQpaWindow) << "Detected nextValidKeyView" << m_view.nextValidKeyView
+                << "in different window" << m_view.nextValidKeyView.window
+                << "Resetting" << m_view.window << "first responder to nil.";
+            [m_view.window makeFirstResponder:nil];
+        }
+
         m_view.hidden = YES;
 
         if (parentCocoaWindow && window()->type() == Qt::Popup) {