macOS: Compute NSWindow background color without checking styleMask

The check for styleMask == NSWindowStyleMaskBorderless to decide whether
to clear the NSWindow background was broken, as NSWindowStyleMaskBorderless
has the value 0, but is only supposed to be compared to its companion
NSWindowStyleMaskTitled (with value 1). A window can perfectly well be
NSWindowStyleMaskBorderless and NSWindowStyleMaskMiniaturizable e.g.,
so by comparing directly to NSWindowStyleMaskBorderless instead of
masking to the first bit first we ended up making miniaturizable
windows non-translucent.

We now check the Qt::FramelessWindowHint directly, and also whether
the window is opaque. Ideally we'd have QWindow flags that could
plumb WA_NoSystemBackground from Qt Widgets, as well as a background
color property on QWindow to control the system background, but
in the meantime we'll have to use the FramelessWindowHint heuristic.

The QWidget docs have been updated to reflect this.

Task-number: QTBUG-95042
Pick-to: 6.2
Change-Id: I0d40eecace60883c205ebb8c76cef1092cdf1144
Reviewed-by: Morten Johan Sørvig <morten.sorvig@qt.io>

2 files changed, 14 insertions, 6 deletions

diff --git a/src/plugins/platforms/cocoa/qcocoawindow.mm b/src/plugins/platforms/cocoa/qcocoawindow.mm
index d4c1593936..242d442a19 100644
--- a/src/plugins/platforms/cocoa/qcocoawindow.mm
+++ b/src/plugins/platforms/cocoa/qcocoawindow.mm
@@ -529,10 +529,8 @@ NSUInteger QCocoaWindow::windowStyleMask(Qt::WindowFlags flags)
     if (frameless) {
         // Frameless windows do not display the traffic lights buttons for
         // e.g. minimize, however StyleMaskMiniaturizable is required to allow
-        // programatic minimize. However, for framless tool windows (e.g. dock windows)
-        // we don't want that, as it breaks translucency.
-        if (type != Qt::Tool)
-            styleMask |= NSWindowStyleMaskMiniaturizable;
+        // programmatic minimize.
+        styleMask |= NSWindowStyleMaskMiniaturizable;
     } else if (flags & Qt::CustomizeWindowHint) {
         if (flags & Qt::WindowTitleHint)
             styleMask |= NSWindowStyleMaskTitled;
diff --git a/src/plugins/platforms/cocoa/qnswindow.mm b/src/plugins/platforms/cocoa/qnswindow.mm
index 1fa11e2789..82daccc68c 100644
--- a/src/plugins/platforms/cocoa/qnswindow.mm
+++ b/src/plugins/platforms/cocoa/qnswindow.mm
@@ -324,8 +324,18 @@ OSStatus CGSClearWindowTags(const CGSConnectionID, const CGSWindowID, int *, int
 
 - (NSColor *)backgroundColor
 {
-    return self.styleMask == NSWindowStyleMaskBorderless ?
-        [NSColor clearColor] : [super backgroundColor];
+    // FIXME: Plumb to a WA_NoSystemBackground-like window flag,
+    // or a QWindow::backgroundColor() property. In the meantime
+    // we assume that if you have translucent content, without a
+    // frame then you intend to do all background drawing yourself.
+    const QWindow *window = m_platformWindow ? m_platformWindow->window() : nullptr;
+    if (!self.opaque && window && window->flags().testFlag(Qt::FramelessWindowHint))
+        return [NSColor clearColor];
+
+    // This still allows you to have translucent content with a frame,
+    // where the system background (or color set via NSWindow) will
+    // shine through.
+    return [super backgroundColor];
 }
 
 - (void)sendEvent:(NSEvent*)theEvent