macOS: Remove declarative override of -[NSView wantsLayer]

The -[NSView setWantsLayer:] method may have side effects that
extend beyond just setting an internal boolean property, so
we need to ensure it gets called.

This was observed on e.g. macOS 10.12.5, where the method
ends up creating the internal backing layer. On later macOS
versions the method just emits KVO notifications for wantsLayer,
but these may in turn result in similar logic being triggered.

The issue was masked somewhat by AppKit itself calling the
method from e.g. -[NSWindow setContentView:], so we still got
the backing layer created. The problem appeared when running
binaries built against an older SDK (10.6 in this case), which
triggered AppKit to not call -[NSView setWantsLayer:], due to
__NSViewLayerBackWindowFrame() in that case returning false.

This change removes the overridden -[NSView wantsLayer], and replaces
it with an explicit call to -[NSView setWantsLayer] when creating
a new QNSView, essentially revering c8c8cc790a315710b0dae2282dc32.

Task-number: PYSIDE-724
Task-number: PYSIDE-734
Change-Id: Idaff4ed38838311b37da4925b1eec241e077dbcc
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>

1 files changed, 5 insertions, 8 deletions

diff --git a/src/plugins/platforms/cocoa/qnsview_drawing.mm b/src/plugins/platforms/cocoa/qnsview_drawing.mm
index 4e6f71973f..0ad7844d56 100644
--- a/src/plugins/platforms/cocoa/qnsview_drawing.mm
+++ b/src/plugins/platforms/cocoa/qnsview_drawing.mm
@@ -133,31 +133,28 @@
     }
 }
 
-- (BOOL)shouldUseMetalLayer:(QSurface::SurfaceType)surfaceType
+- (BOOL)shouldUseMetalLayer
 {
     // MetalSurface needs a layer, and so does VulkanSurface (via MoltenVK)
+    QSurface::SurfaceType surfaceType = m_platformWindow->window()->surfaceType();
     return surfaceType == QWindow::MetalSurface || surfaceType == QWindow::VulkanSurface;
 }
 
-- (BOOL)wantsLayer
+- (BOOL)wantsLayerHelper
 {
     Q_ASSERT(m_platformWindow);
 
-    // Toggling the private QWindow property or the environment variable
-    // on and off is not a supported use-case, so this code is effectively
-    // returning a constant for the lifetime of our QSNSView, which means
-    // we don't care about emitting KVO signals for @"wantsLayer".
     bool wantsLayer = qt_mac_resolveOption(true, m_platformWindow->window(),
         "_q_mac_wantsLayer", "QT_MAC_WANTS_LAYER");
 
-    bool layerForSurfaceType = [self shouldUseMetalLayer:m_platformWindow->window()->surfaceType()];
+    bool layerForSurfaceType = [self shouldUseMetalLayer];
 
     return wantsLayer || layerForSurfaceType;
 }
 
 - (CALayer *)makeBackingLayer
 {
-    if ([self shouldUseMetalLayer:m_platformWindow->window()->surfaceType()]) {
+    if ([self shouldUseMetalLayer]) {
         // Check if Metal is supported. If it isn't then it's most likely
         // too late at this point and the QWindow will be non-functional,
         // but we can at least print a warning.