macOS: Map dead keys directly to their terminator when building key map

When a key press comes in we may end up in QAppleKeyMapper::possibleKeys()
as part of checking whether the key press should trigger a QShortcut.
The function builds on QAppleKeyMapper::keyMapForKey(), which provides
a map from the given virtual key to all the possible Qt::Keys that can
be produced by applying different modifier key combinations.

The map is built using the Carbon function UCKeyTranslate, that takes
the current keyboard layout, virtual key, and modifiers, and produces
the resulting characters. The function also maintains a running dead
key state via one of the arguments. When mapping a dead key, the state
variable will be updated to the current dead key state, which then
affects the next call to the function (for the next key press).

The problem is that we're not calling UCKeyTranslate for each key press.
We are calling it in a loop, for a single key press, to build up a map
of all the possible characters produced by varying the modifier keys.
And in doing so, we are passing on the dead key state from one call
to the next, even if these are for different modifiers. The result is
that the first call, for the dead key, results in mapping to \0, as
UCKeyTranslate produces no output, it only modifies the dead key state.
And then the next call, for the next modifier key combination, results
in mapping to a character that incorrectly incorporates the dead key
state (resetting it in the process).

What we really want is to directly map the initial modifier combination
to the dead key terminator character, if one is defined. This is the
character produced if the dead key state is cancelled, for example by
pressing a key that's not defined in the dead key state.

To achieve this we pass kUCKeyTranslateNoDeadKeysMask as the translate
options to UCKeyTranslate, and always reset the dead key state before
every call. Another common way to achieve the same result would be to
call UCKeyTranslate a second time when detecting that the first call
produced a dead key state, for example with a synthetic space key, to
trigger the terminator output. But this can potentially fail if the
space key actually has a defined output in the dead key state.

Fixes: QTBUG-95471
Pick-to: 6.2 6.1
Change-Id: Icdae7639fd9a641a86c9d6615679bd93d380ff5c
Reviewed-by: Morten Johan Sørvig <morten.sorvig@qt.io>

2 files changed, 5 insertions, 4 deletions

diff --git a/src/gui/platform/darwin/qapplekeymapper.mm b/src/gui/platform/darwin/qapplekeymapper.mm
index 29fdae8d7f..c2a2ba7ed6 100644
--- a/src/gui/platform/darwin/qapplekeymapper.mm
+++ b/src/gui/platform/darwin/qapplekeymapper.mm
@@ -437,7 +437,6 @@ bool QAppleKeyMapper::updateKeyboard()
     Q_ASSERT(source);
     m_currentInputSource = source;
     m_keyboardKind = LMGetKbdType();
-    m_deadKeyState = 0;
 
     m_keyMap.clear();
 
@@ -508,12 +507,15 @@ const QAppleKeyMapper::KeyMap &QAppleKeyMapper::keyMapForKey(VirtualKeyCode virt
         auto carbonModifiers = toCarbonModifiers(qtModifiers);
         const UInt32 modifierKeyState = (carbonModifiers >> 8) & 0xFF;
 
+        UInt32 deadKeyState = 0;
         static const UniCharCount maxStringLength = 10;
         static UniChar unicodeString[maxStringLength];
         UniCharCount actualStringLength = 0;
         OSStatus err = UCKeyTranslate(m_keyboardLayoutFormat, virtualKey,
-            kUCKeyActionDown, modifierKeyState, m_keyboardKind, OptionBits(0),
-            &m_deadKeyState, maxStringLength, &actualStringLength, unicodeString);
+            kUCKeyActionDown, modifierKeyState, m_keyboardKind,
+            kUCKeyTranslateNoDeadKeysMask, &deadKeyState,
+            maxStringLength, &actualStringLength,
+            unicodeString);
 
         // Use translated Unicode key if valid
         QChar carbonUnicodeKey;
diff --git a/src/gui/platform/darwin/qapplekeymapper_p.h b/src/gui/platform/darwin/qapplekeymapper_p.h
index 78e71ac37d..9b2b661b7b 100644
--- a/src/gui/platform/darwin/qapplekeymapper_p.h
+++ b/src/gui/platform/darwin/qapplekeymapper_p.h
@@ -102,7 +102,6 @@ private:
     enum { NullMode, UnicodeMode, OtherMode } m_keyboardMode = NullMode;
     const UCKeyboardLayout *m_keyboardLayoutFormat = nullptr;
     KeyboardLayoutKind m_keyboardKind = kKLKCHRuchrKind;
-    mutable UInt32 m_deadKeyState = 0; // Maintains dead key state beween calls to UCKeyTranslate
 
     mutable QHash<VirtualKeyCode, KeyMap> m_keyMap;
 #endif