path: root/src/virtualkeyboard/doc/src/technical-guide.qdoc

// Copyright (C) 2022 The Qt Company Ltd.
// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only

/*!

\page qtvirtualkeyboard-overview.html
\title Qt Virtual Keyboard Overview
\brief Describes the technical details of the Qt Virtual Keyboard module.
\ingroup explanantion
This document provides a technical overview of the Qt Virtual Keyboard plugin.

\section1 Features

Key features of the Qt Virtual Keyboard include:

\list
    \li Customizable keyboard layouts and styles with dynamic switching.
    \li Predictive text input with word selection.
    \li Character preview and alternative character view.
    \li Automatic capitalization and space insertion.
    \li Scalability to different resolutions.
    \li Support for different character sets (Latin, Simplified/Traditional Chinese,
        Hindi, Japanese, Arabic, Hebrew, Korean, and others).
    \li Support for most common input \l {Supported Languages}{languages},
        with possibility to easily extend the language support.
    \li Left-to-right and right-to-left input.
    \li Hardware key support for 2-way and 5-way navigation.
    \li \l {Handwriting Recognition}{Handwriting support}, with gestures for fullscreen input.
    \li Audio feedback.
    \li Cross-platform functionality.
    \li Supports both \l {Integration Method}{Qt Quick and Qt Widgets applications}.
\endlist

\section1 Supported Languages

\include layouts.qdocinc layout-list

\section1 Third-party Plugins

The Qt Virtual Keyboard supports third-party plugins from the following vendors:
\list
    \li \l {Cerence XT9 Advanced Input}{Cerence XT9} advanced input.
    \li \l {Handwriting Recognition#Cerence Handwriting}{Cerence Handwriting} text input.
    \li \l {Handwriting Recognition#MyScript}{MyScript} Text handwriting recognition
\endlist

\l {Building Qt Virtual Keyboard} describes how to integrate these plugins
into the Qt Virtual Keyboard.

\section1 Basic Concepts

The Qt Virtual Keyboard project is a Qt input context plugin that implements
QPlatformInputContextPlugin and QPlatformInputContext interfaces.
These interfaces allow the plugin to be used as a platform input
context plugin in Qt applications.

The plugin itself provides an input framework supporting multiple
input methods as well as a QML UI for the virtual keyboard. The input
framework is extensible through a plugin interface, which allows third-party
input methods and keyboard layouts to be loaded at runtime.

The input framework provides the following main interfaces:

\list
    \li QVirtualKeyboardInputContext: provides contextual information for the virtual keyboard
        and other input components. Acts as an interface to the underlying
        text input component.
    \li QVirtualKeyboardInputEngine: exposes an API to integrate user input events (key
        presses, etc.) and acts as a host for input methods.
    \li QVirtualKeyboardAbstractInputMethod: a base type for C++-based input methods. The input method
        usually handles key events, but can also handle mouse and touch input
        events.
    \li InputMethod: a base type for QML-based input methods. The input method
        usually handles key events, but can also handle mouse and touch input
        events.
\endlist

\section1 Input Context

The input context is used by the keyboard as well as concrete input methods. InputContext
is a singleton instance hosted by QML. An application should not directly interact with
the input context.

\section2 Contextual Information

The input context provides access to contextual information that originates
from the application. This information includes, but is not limited to:

\list
    \li InputContext::cursorPosition
    \li InputContext::cursorRectangle
    \li InputContext::inputMethodHints
    \li InputContext::preeditText
    \li InputContext::selectedText
    \li InputContext::surroundingText
\endlist

\section2 Locale

The Virtual Keyboard Engine generates the list of supported locales from
locale-specific layout directories in \c {layouts/}. Each layout directory
must contain a definition or fallback for the following layout types:
\e dialpad, \e digits, \e handwriting, \e main, \e numbers, and \e symbols.
Definitions are implemented in \c {.qml}-files, fallbacks are defined by a
placeholder file with the \c {.fallback} file extension. The \c {layouts/}
directory must contain a \c {fallback/} sub-directory that contains definitions
for each layout type.

Each layout directory may contain the definition of one or more layout types.
If the locale-specific layout is the same as that of the fallback locale, you
can add a placeholder file for the layout called \c {<layout type>.fallback}.
This instructs the virtual keyboard to use the fallback layout instead.

For example: you may add a locale-specific layout for Finnish, that defines the
main layout type in \c {main.qml}. For the other layout types, you opt for the
fallback mechanism. Your \c {layouts/} tree will look like this:

\badcode
.
├── fallback
│   ├── dialpad.qml
│   ├── digits.qml
│   ├── handwriting.qml
│   ├── main.qml
│   ├── numbers.qml
│   └── symbols.qml
└── fi_FI
    ├── dialpad.fallback
    ├── digits.fallback
    ├── handwriting.fallback
    ├── main.qml
    ├── numbers.fallback
    └── symbols.fallback
\endcode

It's imperative that the \c {layouts/fallback} directory always contain a set
of full implementation files.

The application can specify the initial layout by changing the default locale.
However, this must be done before the application initializes and loads the
input method plugin. If there are no changes to the default locale, the current
system locale is used.

Matching the keyboard locale follows this sequence:

\list
    \li \c {layouts/<language>_<country>}
    \li \c {layouts/<language>_*}
    \li \c {layouts/fallback} -- the default layout here is \e en_GB.
\endlist

First, the locale is matched against the full locale name. If a there isn't a
full match, then only the locale language is matched. Finally, the contents of
\c {layouts/fallback} is used as a fallback when there's also no partial match.

After the locale selection is done, the keyboard updates the input locale and
input direction to match the current layout. The application can receive this
information through the QInputMethod interface.

Internally, the current input locale is also updated to
QVirtualKeyboardInputEngine and the current input method instances.

\section1 Input Engine

The input engine object is owned by InputContext. Like the InputContext, there is
only one instance of the QVirtualKeyboardInputEngine. The input engine contains API functions which
the keyboard uses to map user interactions, such as key press and release events,
to the input method.

For example, virtual keyboard key events are mapped through the following methods:

\list
    \li QVirtualKeyboardInputEngine::virtualKeyPress
    \li QVirtualKeyboardInputEngine::virtualKeyRelease
\endlist

The above-mentioned methods are intended for the integration of the virtual
keyboard, hence the word "virtual" in the methods' names. This also means that
the methods are not suitable for mapping the physical keystrokes. This is a
consequence of the fact that the actual action is performed only when the key
is released.

If the key press is interrupted before the key release event, the keyboard invokes
the QVirtualKeyboardInputEngine::virtualKeyCancel method.

\section2 Input Method

The input method is a concrete implementation of the keypress handler. It's main
function is to handle keypress events and maintain state information for the
user input. It interacts with the text editor through \l QVirtualKeyboardInputContext via
pre-edit text or key events.

The input method instance can be created in various ways, depending on the use
case:

\list
    \li \c {KeyboardLayout::inputMethod}: the keyboard layout can create an input method
        instance solely for this keyboard layout. It should be noted that this
        instance will be destroyed when the keyboard layout changes. Therefore,
        this method is usually limited to very narrow use cases.
    \li \c {KeyboardLayout::createInputMethod()}: the keyboard layout can dynamically
        create an input method that can be used with this layout and also with the
        \l {KeyboardLayout::sharedLayouts} {shared layouts} (e.g. symbol layout).
        This is the preferred way for creating specialized input methods, such as
        those involving complex languages or handwriting.
    \li \c {DefaultInputMethod}: the virtual keyboard attempts to create this type of
        input method at startup. This instance will be used as the default input
        method across all keyboard layouts, unless the keyboard layout uses a
        custom input method. This instance outlives keyboard layout changes across
        languages and is the preferred way of creating and overriding the default
        input method.
\endlist

\section1 Virtual Keyboard Plugins

The \e src/plugins directory of the virtual keyboard contains the existing plugins
for the virtual keyboard. These plugins are standard QML modules loaded implicitly
by the QtQuick.VirtualKeyboard.Plugins QML module.

A plugin may provide keyboard layouts and input methods (usually both). The input method
used by the virtual keyboard depends on which keyboard layout is in use. The keyboard layout
may provide an instance of a custom input method by the KeyboardLayout.createInputMethod()
function. Otherwise, the default input method (DefaultInputMethod) created by the virtual
keyboard is used.

\section2 Adding Keyboard Layouts

The plugin can add keyboard layouts for the virtual keyboard by including
the layout files in the Qt resources of the plugin binary.

The virtual keyboard searches for the keyboard layouts (per language) from a specific
path \e /qt-project.org/imports/QtQuick/VirtualKeyboard/Layouts/<language_COUNTRY>, so it is essential
to use this exact path also in the plugin. Qt resource paths can overlap, meaning that
the plugin can override existing layouts on the virtual keyboard.

It is also possible to override the built-in keyboard layouts by having them loaded directly
from the file system by using the \e QT_VIRTUALKEYBOARD_LAYOUT_PATH environment variable.

\section2 Adding Input Method

The plugin can register an input method that other keyboard layouts can use by
default (such as \c DefaultInputMethod) or an input method that is used privately
in the plugin (by also providing custom keyboard layout, which creates the input
method).

The input method must implement a \l {QVirtualKeyboardAbstractInputMethod} (C++) or InputMethod (QML)
interface and it must be registered as a QML type (\l {QML_NAMED_ELEMENT}) by the plugin.

\section1 Implementing a Custom Input Method

The implementation of input methods starts by deciding which interface is used,
QML or C++. In this example the QML interface is used. The same logic and interfaces
apply with the C++ interface \l {QVirtualKeyboardAbstractInputMethod}. In this case,
the plugin must be linked to the \e VirtualKeyboard module.

The following example shows the minimum functionality that is required
from an input method:

\quotefile CustomInputMethod.qml

The InputMethod::inputModes() method is called by the input engine before
an input mode is set. The method returns a list of input modes available in
the given locale.

An input method is initialized in InputMethod::setInputMode() method with
a locale and input mode. After setting the locale and input mode, the input
method should be ready for use.

InputMethod::reset() is called when an input method needs to be reset. The reset
must only reset the internal state of the input method, and not the user text.

InputMethod::update() is called when the input context is updated and
the input state is possibly out of sync. The input method should commit
the current text.

The keystroke events are handled in InputMethod::keyEvent(). This method handles
a single keystroke event and returns \c true if the event was processed.
Otherwise the keystroke is handled by the default input method.

\section2 Selection Lists

Selection lists are an optional feature that can be integrated into the input
method. The input framework supports various types of lists, such as the word
candidate list. Responsibilities in implementing the lists are handled such
that the input method is responsible for the content and activities, such as the
click behavior. The input framework is responsible for maintaining the list
model and delivery to the user interface.

\section3 Allocating the Selection Lists

Selection lists are allocated when the input method is activated. The
InputMethod::selectionLists() method returns a list of the required selection
list types:

\code
function selectionLists() {
    return [SelectionListModel.Type.WordCandidateList];
}
\endcode

In the above example, the input method allocates the word candidate list for its
use.

\section3 Updating the Selection Lists

When the input method requires the UI to update the contents of a selection
list, it will emit the InputMethod::selectionListChanged signal. Likewise, if
the input method requires the UI to highlight an item in the list, it will emit
the InputMethod::selectionListActiveItemChanged signal.

\code
selectionListChanged(SelectionListModel.Type.WordCandidateList)
selectionListActiveItemChanged(SelectionListModel.Type.WordCandidateList, wordIndex)
\endcode

\section3 Populating Items in the Selection Lists

The items are populated with method callbacks which will provide the number of
items in a list as well as the data for individual items.

The InputMethod::selectionListItemCount callback requests the number of
items in the list identified by the given type.

\code
function selectionListItemCount(type) {
    if (type == SelectionListModel.Type.WordCandidateList) {
        return wordList.length
    }
    return 0
}
\endcode

The InputMethod::selectionListData callback requests the data for items.

\code
function selectionListData(type, index, role) {
    var result = null
    if (type == SelectionListModel.Type.WordCandidateList) {
        switch (role) {
        case SelectionListModel.Role.Display:
            result = wordList[index]
            break
        default:
            break
        }
    }
    return result
}
\endcode

The \a role parameter identifies which data is requested for an item. For
example, the SelectionListModel.Role.Display requests the display text data.

\section3 Responding to User Actions

When the user selects an item in the list, the input method responds to the
event in the InputMethod::selectionListItemSelected method callback.

\code
function selectionListItemSelected(type, index) {
    if (type == SelectionListModel.Type.WordCandidateList) {
        inputContext.commit(wordlist[index])
        update()
    }
}
\endcode

\section1 Integrating Handwriting Recognition

An input method can also use input data from the touch screen or other input devices.

When the input begins, the virtual keyboard calls the input method function
\l { QVirtualKeyboardAbstractInputMethod::traceBegin } { traceBegin }, which returns a new \l {QVirtualKeyboardTrace} {Trace}
object to which the input is collected on behalf of the input method. Similarly, when
the finger or stylus is raised, the event is terminated with the \l { QVirtualKeyboardAbstractInputMethod::traceEnd } { traceEnd } call.
The input method processes the collected data and produces text using the
\l {QVirtualKeyboardInputContext} {InputContext} interface.

There are predefined keyboard layouts for handwriting. However, they are not included
by default, and the handwriting plugin should include them in its own resources. For
examples of how to do this, see the existing plugins for handwriting from \e MyScript
or \e Cerence.

\section2 Data Model for Handwriting Input

Virtual keyboard collects the handwriting data in a special data model QVirtualKeyboardTrace.
Each trace represents a collection of data sampled from one touch (e.g.
a swipe on the screen). There will be as many instances of QVirtualKeyboardTrace as there are
touches on the handwriting input area.

By definition, \e trace is a set of data sampled from one touch. In addition to
the basic point data, it can also include other types of data, such as
the time of each point. The input method can define the desired input channels
in the beginning of a trace event.

The input method does not participate in the actual collection of trace data.
However, the input method has full control over the input since it can
either accept or reject a QVirtualKeyboardTrace (e.g. if there are too many instances to handle).
This also allows for precise control over how many fingers can be used simultaneously.

The input method can collect as many traces as it sees fit and it can begin
processing them when necessary. The processing can even be performed in parallel while
sampling the data, although it is not recommended because of the potential performance
issues. The recommended way is to start processing in a background thread
after a suitable delay from the last input, so that the processing does not affect
negatively to the user interface.

\section2 Trace API for Input Methods

The trace API consists of the following virtual methods, which the input method
must implement in order to receive and process trace input data.

\list
    \li \l { QVirtualKeyboardAbstractInputMethod::patternRecognitionModes } { patternRecognitionModes }
    \li \l { QVirtualKeyboardAbstractInputMethod::traceBegin } { traceBegin }
    \li \l { QVirtualKeyboardAbstractInputMethod::traceEnd } { traceEnd }
\endlist

By implementing these methods, the input method can receive and process data
from a variety of input sources (e.g. keyboard layout or full screen).

The patternRecognitionModes method returns a list of pattern recognition modes,
which are supported by the input method. A pattern recognition mode, such as
\l { QVirtualKeyboardInputEngine::PatternRecognitionMode } { Handwriting },
defines the method by which the input method processes the data.

The trace interaction is started when an input source detects a new contact point, and
calls the traceBegin method for a new trace object. If the input method accepts the
interaction, it creates a new trace object and returns it to the caller. From this point
on, trace data is collected until the traceEnd method is called.

When the traceEnd method is called, the input method may begin processing of the data
contained in the trace object. After processing the data, the input method should destroy
the object. This also removes the trace rendered to the screen.

\section1 Keyboard Layouts

Keyboard layouts are located in the \e src/layouts/builtin
directory. Each subdirectory of the layout directory represents a locale.
The locale directory is a string of the form "language_country", where
language is a lowercase, two-letter ISO 639 language code, and country is an
uppercase, two or three-letter ISO 3166 country code.

\section2 Layout Types

Different keyboard layout types are used in different input modes. The default
layout which is used for regular text input, is called the "main" layout.
The layout type is determined by the layout file name. Therefore, the "main" layout
file is called the "main.qml".

List of supported layout types:

\list
    \li \c main The main layout for normal text input
    \li \c symbols Symbol layout for special characters etc. (activated from main layout)
    \li \c numbers Numeric layout for formatted numbers (activated by Qt::ImhFormattedNumbersOnly)
    \li \c digits Digits only layout (activated by Qt::ImhDigitsOnly)
    \li \c dialpad Dialpad layout for phone number input (activated by Qt::ImhDialableCharactersOnly)
    \li \c handwriting Handwriting layout for handwriting recognition (activated from main layout)
\endlist

\section2 Adding New Keyboard Layouts

The keyboard layout element must be based on the KeyboardLayout QML type.
This type defines the root item of the layout. The root item has the following
optional properties which can be set if necessary:

\table
    \row
        \li \c {property var inputMethod}
        \li Specifies an input method for this layout. If the input method is
            not defined, then the current input method is used.
    \row
        \li \c {property int inputMode}
        \li Specifies an input mode for this layout.
    \row
        \li \c {property real keyWeight}
        \li Specifies the default key weight used for all keys in this keyboard
            layout. The key weight is a proportional value which affects the size
            of individual keys in relation to each other.
\endtable

New rows are added to the keyboard layout by using the KeyboardRow type. The
KeyboardRow can also specify the default key weight for its child elements.
Otherwise, the key weight is inherited from its parent element.

New keys are added to the keyboard row using the Key type or one of the
specialized key types. Below is the list of all key types:

\generatelist groupsbymodule qtvirtualkeyboard-key-types

For example, to add a regular key which sends a key event to the input method:

\code
    import QtQuick
    import QtQuick.VirtualKeyboard
    import QtQuick.VirtualKeyboard.Components

    // file: en_GB/main.qml

    KeyboardLayout {
        keyWeight: 160
        KeyboardRow {
            Key {
                key: Qt.Key_Q
                text: "q"
            }
        }
    }
\endcode

\section2 Key Size Calculation

The keyboard layouts are scalable, which means that there cannot be any fixed
sizes set for any items in the layout. Instead, the key widths are calculated
from key weight in relation to each other and the height by dividing the space
equally among the keyboard rows.

In the above example, the key size is inherited from parent elements in this
order:

Key > KeyboardRow > KeyboardLayout

The effective value for the key weight will be 160. For the sake of the
example, we add another key which specifies a custom key weight:

\code
    import QtQuick
    import QtQuick.VirtualKeyboard
    import QtQuick.VirtualKeyboard.Components

    // file: en_GB/main.qml

    KeyboardLayout {
        keyWeight: 160
        KeyboardRow {
            Key {
                key: Qt.Key_Q
                text: "q"
            }
            Key {
                key: Qt.Key_W
                text: "w"
                keyWeight: 200
            }
        }
    }
\endcode

Now the total key weight of a row is \e {160 + 200 = 360}. When the keyboard
layout is activated, the width of an individual key is calculated as follows:

\e {key width in pixels = key weight / SUM(key weights in a row) * row width in pixels}

This means that the keyboard can be scaled to any size, while the relative key
sizes remain the same.

\section2 Alternative Keys

Key can specify an alternativeKeys property, which results in a popup that
lists alternative keys when the user presses and holds the key.
The alternativeKeys can specify either a string, or a list of strings.
If alternativeKeys is a string, the user can select between the characters in
the string.

\section2 Styles and Layouts

The keyboard layouts cannot specify any visual elements. Instead, the layout is
visualized by the keyboard style. On the other hand, the keyboard style cannot
affect the size of the keyboard layout.

\section2 Keyboard Layouts with Multiple Pages of Keys

Some keyboard layouts, such as symbol layouts, may contain more keys than it is
feasible to present on a single keyboard layout. A solution is to embed multiple
keyboard layouts into the same context by using the KeyboardLayoutLoader.

When the KeyboardLayoutLoader is used as a root item of a keyboard layout, the
actual keyboard layouts are wrapped inside Component elements. The keyboard
layout is activated by assigning the id of an active component to the
sourceComponent property.

For example:

\code
    import QtQuick
    import QtQuick.VirtualKeyboard
    import QtQuick.VirtualKeyboard.Components

    // file: en_GB/symbols.qml

    KeyboardLayoutLoader {
        property bool secondPage
        onVisibleChanged: if (!visible) secondPage = false
        sourceComponent: secondPage ? page2 : page1
        Component {
            id: page1
            KeyboardLayout {
                KeyboardRow {
                    Key {
                        displayText: "1/2"
                        functionKey: true
                        onClicked: secondPage = !secondPage
                    }
                }
            }
        }
        Component {
            id: page2
            KeyboardLayout {
                KeyboardRow {
                    Key {
                        displayText: "2/2"
                        functionKey: true
                        onClicked: secondPage = !secondPage
                    }
                }
            }
        }
    }
\endcode

\section2 Handwriting Keyboard Layout

Each language which supports handwriting recognition must provide a
special keyboard layout named \e handwriting.qml.

This type of keyboard layout must meet the following requirements:
\list
\li contains a TraceInputKey in the keyboard layout
\li provides an instance of HandwritingInputMethod as the input method.
\endlist

The handwriting layout may also include \l ChangeLanguageKey. For this purpose, it is
important to use the \l {ChangeLanguageKey::customLayoutsOnly} {customLayoutsOnly} attribute, which will filter out languages
that do not use handwriting.

Both the main and handwriting layouts should contain a key to activate
and deactivate the handwriting input mode. This can be done by adding a
HandwritingModeKey to the layout.

\section2 Adding Custom Layouts

The virtual keyboard layouts system supports built-in layouts as well
as custom layouts. The built-in layouts are embedded as \l {The Qt Resource System}{Qt Resources}
into the plugin binary. Custom layouts are located in the file system, so
that they can be installed without recompiling the virtual keyboard itself, or they can be located in a resource file.

The selection of layouts at runtime is affected by the \c QT_VIRTUALKEYBOARD_LAYOUT_PATH environment variable.

In case the environment variable is not set, or contains an invalid
directory, the virtual keyboard falls back to the default built-in layouts.

To prevent the built-in layouts from being built into the virtual keyboard
plugin when using custom layouts, add \c -no-vkb-layouts option to the \c configure script.
For more information, see \l [DOC] {Advanced Usage} {Configuration Options}.

\section1 Keyboard Styles

The virtual keyboard styling system supports built-in styles as well
as custom styles. The built-in styles are embedded as Qt Resources
into the plugin binary and the custom styles are located in the file system
and can be installed without recompiling the virtual keyboard itself.

The selection of the runtime style is affected by an environment
variable QT_VIRTUALKEYBOARD_STYLE, which can be set to the name of
the built-in style, e.g. "retro", or any of the custom styles installed
into the Styles directory:

\code
$$[QT_INSTALL_QML]/QtQuick/VirtualKeyboard/Styles
\endcode

In case the environment variable is not set, or contains an invalid
style name, the virtual keyboard falls back in the default built-in style.

\section2 Adding Custom Styles

The process of creating a new style begins by creating a new subdirectory
for the style in a QML import path under the URL-based directory structure
QtQuick/VirtualKeyboard/Styles/. See \l[QtQml]{QML Import Path}
for information about QML import paths. The directory
name can not contain spaces or special characters other than underscore.
Also, the directory name can not be the same as one of the built-in style,
which currently includes "default" and "retro".

A good starting point for creating a new style is to use an existing
built-in style as a template and edit it. You can find the built-in
styles from the virtual keyboard sources directory
src/styles/builtin. Copy one of the directories containing
a built-in style into the \e Styles directory and rename it to "test".
The directory structure should now be as follows:

\code
test/default_style.qrc
test/style.qml
test/images
test/images/backspace.png
test/images/check.png
test/images/enter.png
test/images/globe.png
test/images/hidekeyboard.png
test/images/search.png
test/images/shift.png
\endcode

The QRC configuration file, which is unnecessary in this case, can
be safely removed.

\note The style.qml file should not be renamed, or otherwise the virtual
      keyboard cannot load the style.

Next, open the style.qml in your favorite editor and set the
resourcePrefix property to an empty string. The resource prefix
is not needed as the resources are contained in the same directory
as the style.qml file.

Also, to make it more obvious to see that the custom style is actually
being loaded and used, set the keyboard background to a different color:

\code
keyboardBackground: Rectangle {
    color: "gray"
}
\endcode

The final step is to run the example application with your custom style:

\code
QT_VIRTUALKEYBOARD_STYLE=test virtualkeyboard
\endcode

\section1 Using Qt Virtual Keyboard with QQuickWidget

When using Qt Virtual Keyboard in a \l QQuickWidget on a touch device,
it is necessary to set the \l Qt::WA_AcceptTouchEvents attribute via
\l QWidget::setAttribute(). Without this attribute set, events from
a touch device will be converted into synthesized mouse events.

*/