macOS: Use submenuAction: as action for sub-menu menu items

Having the generic qt_itemFired: as action would result in the whole
submenu tree closing if an item with a sub-menu was clicked on. This
is not how native applications behave. They respond by immediately
opening the submenu, or do nothing if the menu is already open.

By using submenuAction: as the selector we achieve the same behavior.

A complication here is that for some reason we defer associating the
submenu NSMenu to an NSMenuItem until QCocoaMenu::setAttachedItem(),
instead of doing it in QCocoaMenuItem::setMenu(), or even as part of
QCocoaMenuItem::sync().

As a result, AppKit's NSMenuValidation logic will conclude that the
item does neither have a submenu, nor a valid target/selector combo
to be validated, and will explicitly disable the item. This can be
debugged by passing -NSTrackMenuValidation YES to the application.

To work around this we explicitly enable the item once we have set
a valid submenu for the item.

Pick-to: 6.5 6.6
Fixes: QTBUG-114199
Change-Id: I7178e7687066b3fe082454c512ec9c7eab3bded4
Reviewed-by: Timur Pocheptsov <timur.pocheptsov@qt.io>
Reviewed-by: Tor Arne Vestbø <tor.arne.vestbo@qt.io>

2 files changed, 18 insertions, 1 deletions

diff --git a/src/plugins/platforms/cocoa/qcocoamenu.mm b/src/plugins/platforms/cocoa/qcocoamenu.mm
index 0f39246a43..4b66d2b610 100644
--- a/src/plugins/platforms/cocoa/qcocoamenu.mm
+++ b/src/plugins/platforms/cocoa/qcocoamenu.mm
@@ -483,6 +483,10 @@ void QCocoaMenu::setAttachedItem(NSMenuItem *item)
     if (m_attachedItem)
         m_attachedItem.submenu = m_nativeMenu;
 
+    // NSMenuItems with a submenu and submenuAction: as the item's action
+    // will not take part in NSMenuValidation, so explicitly enable/disable
+    // the item here. See also QCocoaMenuItem::resolveTargetAction()
+    m_attachedItem.enabled = m_attachedItem.hasSubmenu;
 }
 
 NSMenuItem *QCocoaMenu::attachedItem() const
diff --git a/src/plugins/platforms/cocoa/qcocoamenuitem.mm b/src/plugins/platforms/cocoa/qcocoamenuitem.mm
index 0acae8d679..3a0f71bc50 100644
--- a/src/plugins/platforms/cocoa/qcocoamenuitem.mm
+++ b/src/plugins/platforms/cocoa/qcocoamenuitem.mm
@@ -473,7 +473,20 @@ void QCocoaMenuItem::resolveTargetAction()
         roleAction = @selector(selectAll:);
         break;
     default:
-        roleAction = @selector(qt_itemFired:);
+        if (m_menu) {
+            // Menu items that represent sub menus should have submenuAction: as their
+            // action, so that clicking the menu item opens the sub menu without closing
+            // the entire menu hierarchy. A menu item with this action and a valid submenu
+            // will disable NSMenuValidation for the item, which is normally not an issue
+            // as NSMenuItems are enabled by default. But in our case, we haven't attached
+            // the submenu yet, which results in AppKit concluding that there's no validator
+            // for the item (the target is nil, and nothing responds to submenuAction:), and
+            // will in response disable the menu item. To work around this we explicitly
+            // enable the menu item in QCocoaMenu::setAttachedItem() once we have a submenu.
+            roleAction = @selector(submenuAction:);
+        } else {
+            roleAction = @selector(qt_itemFired:);
+        }
     }
 
     m_native.action = roleAction;