1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
|
// Copyright (C) 2022 Intel Corporation.
// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR GFDL-1.3-no-invariants-only
/*!
\page ipc.html
\title Inter-Process Communication
\ingroup groups
\ingroup frameworks-technologies
\keyword ipc
\brief An overview of Qt's inter-process communication functionality
Qt supports many ways of communicating with other processes running in the
same system or in different systems. There are basically three types of
inter-process communication mechanisms:
\list 1
\li Synchronization primitives
\li Exchanging of arbitrary byte-level data
\li Passing structured messages
\endlist
\section1 Synchronization primitives
Qt only provides one class for explicit inter-process synchronization:
\l{QSystemSemaphore}. A QSystemSemaphore is like a \l{QSemaphore} that is
accessible by multiple processes in the same system. It is globally
identified by a "key", which in Qt is represented by the \l{QNativeIpcKey}
class. Additionally, depending on the OS, Qt may support multiple different
backends for sharing memory; see the \l{Native IPC Keys} documentation for
more information and limitations.
It is possible to use regular thread-synchronization primitives such as
mutexes, wait conditions, and read-write locks, located in memory that is
shared between processes. Qt does not provide any class to support this,
but applications can use low-level operations on certain operating systems.
Other Qt classes may be used to provide higher-level locking, like
\l{QLockFile}, or by acquiring a unique, system-wide resource. Such
techniques include \l{QTcpServer}{TCP} or \l{QUdpSocket}{UDP} ports or
well-known names in \l{QDBusConnection::registerService}{D-Bus}.
\section1 Byte-level data sharing
Using byte-level data, applications can implement any communication
protocol they may choose. Sharing of byte data can be stream-oriented
(serialized) or can allow random access (a similar condition to
QFileDevice::isSequential()).
For serial communication, Qt provides a number of different classes and
even full modules:
\list
\li Pipes and FIFOs: \l QFile
\li Child processes: \l QProcess
\li Sockets: \l QTcpSocket, \l QUdpSocket (in \l{Qt Network})
\li HTTP(S): \l QNetworkAccessManager (in \l{Qt Network}) and
\l QHttpServer (in \l{Qt HTTP Server})
\li CoAP(S): \l QCoapClient (in \l{Qt CoAP})
\endlist
For random-access data sharing within the same system, Qt provides
\l{QSharedMemory}. See the \l{Shared Memory} documentation for detailed
information.
\section1 Structured message passing
Qt also provides a number of techniques to exchange structured messages
with other processes. Applications can build on top of the byte-level
solutions above, such as by using \l QJsonDocument or \l QXmlStreamReader /
\l QXmlStreamWriter over HTTP to perform JSONRPC or XMLRPC, respectively,
or \l QCborValue with QtCoAP.
Dedicated Qt modules for structured messages and remote procedure-calling
include:
\list
\li \l{Qt D-Bus}
\li \l{Qt Remote Objects}
\li \l{Qt WebSockets}
\endlist
*/
/*!
\page shared-memory.html
\title Shared Memory
\keyword ipc
\keyword shared memory
\brief Overview of the techniques for sharing memory between processes
Qt provides two techniques to share memory with other processes in the same
system: \l{QSharedMemory} and memory-mapped files using \l{QFile}. Memory
that is shared with other processes is often referred to as a "segment",
and although it may have been implemented as specific segments on
processors with segmented memory models in the past, this is not the case
in any modern operating system. Shared memory segments are simply regions
of memory that the operating system will ensure are available to all
processes participating.
\note The address at which the segment is located in memory will almost
always be different for each process that is participating in the sharing.
Therefore, applications must take care to share only position-independent
data, such as primitive C++ types or arrays of such types.
\section1 Sharing memory using QSharedMemory
QSharedMemory provides a simple API to create a shared memory segment of a
given size or attach to one that was created by another process.
Additionally, it provides a pair of methods to \l{QSharedMemory::}{lock}
and \l{QSharedMemory::}{unlock} the whole segment, using an internal
\l{QSystemSemaphore}.
Shared memory segments and system semaphores are globally identified in the
system through a "key", which in Qt is represented by the \l{QNativeIpcKey}
class. Additionally, depending on the OS, Qt may support multiple different
backends for sharing memory; see the \l{Native IPC Keys} documentation for
more information and limitations.
QSharedMemory is designed to share memory only within the same privilege
level (that is, not with untrusted other processes, such as those started
by other users). For backends that support it, QSharedMemory will create
segments such that only processes with the same privilege level can attach.
\section1 Sharing memory via memory-mapped files
Most files can be mapped to memory using QFile::map() and, if the
\l{QFileDevice::MapPrivateOption}{MapPrivateOption} option is not specified,
any writes to the mapped segment will be observed by all other processes
that have mapped the same file. Exceptions to files that can be mapped to
memory include remote files found in network shares or those located in
certain filesystems. Even if the operating system does allow mapping remote
files to memory, I/O operations on the file will likely be cached and
delayed, thus making true memory sharing impossible.
This solution has the major advantages of being independent of any backend
API and of being simpler to interoperate with from non-Qt applications.
Since \l{QTemporaryFile} is a \l{QFile}, applications can use that class to
achieve clean-up semantics and to create unique shared memory segments too.
To achieve locking of the shared memory segment, applications will need to
deploy their own mechanisms. One way may be to use \l QLockFile. Another
and less costly solution is to use QBasicAtomicInteger or \c{std::atomic} in
a pre-determined offset in the segment itself. Higher-level locking
primitives may be available on some operating systems; for example, on
Linux, applications can set the "pshared" flag in the mutex attribute
passed to \c{pthread_mutex_create()} to indicate that the mutex resides in
a shared memory segment.
Be aware that the operating system will likely attempt to commit to
permanent storage any writes made to the shared memory. This may be desired
or it may be a performance penalty if the file itself was meant to be
temporary. In that case, applications should locate a RAM-backed
filesystem, such as \c{tmpfs} on Linux (see
QStorageInfo::fileSystemType()), or pass a flag to the native file-opening
function to inform the OS to avoid committing the contents to storage.
It is possible to use file-backed shared memory to communicate with
untrusted processes, in which case the application should exercise great
care. The files may be truncated/shrunk and cause applications accessing
memory beyond the file's size to crash.
\section2 Linux hints on memory-mapped files
On modern Linux systems, while the \c{/tmp} directory is often a \c{tmpfs}
mount point, that is not a requirement. However, the \c{/dev/shm} directory
is required to be a \c{tmpfs} and exists for the very purpose of sharing
memory. Do note that it is world-readable and writable (like \c{/tmp} and
\c{/var/tmp}), so applications must be careful of the contents revealed
there. Another alternative is to use the XDG Runtime Directory (see
QStandardPaths::writableLocation() and \l{QStandardPaths::RuntimeLocation}),
which on Linux systems using systemd is a user-specific \c{tmpfs}.
An even more secure solution is to create a "memfd" using \c{memfd_create(2)}
and use interprocess communication to pass the file descriptor, like
\l{QDBusUnixFileDescriptor} or by letting the child process of a \l{QProcess}
inherit it. "memfds" can also be sealed against being shrunk, so they are
safe to be used when communicating with processes with a different privilege
level.
\section2 FreeBSD hints on memory-mapped files
FreeBSD also has \c{memfd_create(2)} and can pass file descriptors to other
processes using the same techniques as Linux. It does not have temporary
filesystems mounted by default.
\section2 Windows hints on memory-mapped files
On Windows, the application can request the operating system avoid saving
the file's contents on permanent storage. This request is performed by
passing the \c{FILE_ATTRIBUTE_TEMPORARY} flag in the
\c{dwFlagsAndAttributes} parameter to the \c{CreateFile} Win32 function,
the \c{_O_SHORT_LIVED} flag to \c{_open()} low-level function, or by
including the modifier "T" to the
\c{fopen()} C runtime function.
There's also a flag to inform the operating system to delete the file when
the last handle to it is closed (\c{FILE_FLAG_DELETE_ON_CLOSE},
\c{_O_TEMPORARY}, and the "D" modifier), but do note that all processes
attempting to open the file must agree on using this flag or not using it. A
mismatch will likely cause a sharing violation and failure to open the file.
*/
/*!
\page native-ipc-keys.html
\title Native IPC Keys
\keyword ipc
\keyword shared memory
\keyword system semaphore
\brief An overview of keys for QSharedMemory and QSystemSemaphore
The \l QSharedMemory and \l QSystemSemaphore classes identify their
resource using a system-wide identifier known as a "key". The low-level key
value as well as the key type are encapsulated in Qt using the \l
QNativeIpcKey class. That class also provides the proper means of
exchanging the key with other processes, by way of
QNativeIpcKey::toString() and QNativeIpcKey::fromString().
Qt currently supports three distinct backends for those two classes, which
match the values available in the \l{QNativeIpcKey::Type} enumeration.
\list
\li POSIX Realtime extensions (IEEE 1003.1b, POSIX.1b)
\li X/Open System Interfaces (XSI) or System V (SVr4), though also now part of POSIX
\li Windows primitives
\endlist
As the name indicates, the Windows primitives are only available on the
Windows operating system, where they are the default backend. The other two
are usually both available on Unix operating systems. The following table
provides an overview of typical availability since Qt 6.6:
\table
\header \li Operating system \li POSIX \li System V \li Windows
\row \li Android \li \li \li
\row \li INTEGRITY \li \li \li
\row \li QNX \li Yes \li \li
\row \li \macos \li Yes \li Usually (1) \li
\row \li Other Apple OSes \li Yes \li \li
\row \li Other Unix systems \li Yes \li Yes \li
\row \li Windows \li Rarely (2) \li \li Yes
\endtable
\note 1 Sandboxed \macos applications, which include all applications
distributed via the Apple App Store, may not use System V objects.
\note 2 Some GCC-compatible C runtimes on Windows provide POSIX-compatible
shared memory support, but this is rare. It is always absent with the
Microsoft compiler.
To determine whether a given key type is supported, applications should
call QSharedMemory::isKeyTypeSupported() and
QSystemSemaphore::isKeyTypeSupported().
QNativeIpcKey also provides support for compatibility with Qt applications
prior to its introduction. The following sections detail the limitations of
the backends, the contents of the string keys themselves, and
compatibility.
\section1 Cross-platform safe key format
QNativeIpcKey::setNativeKey() and QNativeIpcKey::nativeKey() handle the
low-level native key, which may be used with the native APIs and shared
with other, non-Qt processes (see below for the API). This format is not
usually cross-platform, so both QSharedMemory and QSystemSemaphore provide
a function to translate a cross-platform identifier string to the native
key: QSharedMemory::platformSafeKey() and
QSystemSemaphore::platformSafeKey().
The length of the cross-platform key on most platforms is the same as that
of a file name, but is severely limited on Apple platforms to only 30
usable bytes (be mindful of UTF-8 encoding if using characters outside the
US-ASCII range). The format of the key is also similar to that of a file
path component, meaning it should not contain any characters not allowed in
file names, in particular those that separate path components (slash and
backslash), with the exception of sandboxed applications on Apple operating
systems. The following are good examples of cross-platform keys: "myapp",
"org.example.myapp", "org.example.myapp-12345".
\b{Apple sandbox limitations:} if the application is running inside of a
sandbox in an Apple operating system, the key must be in a very specific
format: \c {<application group identifier>/<custom identifier>}. Sandboxing
is implied for all applications distributed through the Apple App Store.
See Apple's documentation
\l{https://developer.apple.com/library/archive/documentation/Security/Conceptual/AppSandboxDesignGuide/AppSandboxInDepth/AppSandboxInDepth.html#//apple_ref/doc/uid/TP40011183-CH3-SW24}
{here} and \l{https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_application-groups}
{here} for more information, including how to obtain the application's group identifier.
\section1 Native key format
This section details the format of the native keys of the supported
backends.
\section3 POSIX Realtime
Native keys resemble file names and may contain any character that file
names do, except for a slash. POSIX requires the first character in the key
name to be a slash and leaves undetermined whether any additional slashes
are permitted. On most operating systems, the key length is the same as a
file name, but it is limited to 32 characters on Apple operating systems
(this includes the first slash and the terminating null, so only 30 usable
characters are possible).
The following are good examples of native POSIX keys: "/myapp",
"/org.example.myapp", "/org.example.myapp-12345".
QSharedMemory::platformSafeKey() and QSystemSemaphore::platformSafeKey()
simply prepend the slash. On Apple operating systems, they also truncate
the result to the available size.
\section3 Windows
Windows key types are NT
\l{https://learn.microsoft.com/en-us/windows/win32/sync/object-namespaces}{kernel
object names} and may be up to \c{MAX_PATH} (260) characters in length.
They look like relative paths (that is, they don't start with a backslash
or a drive letter), but unlike file names on Windows, they are
case-sensitive.
The following are good examples of native Windows keys: "myapp",
"org.example.myapp", "org.example.myapp-12345".
QSharedMemory::platformSafeKey() and QSystemSemaphore::platformSafeKey()
insert a prefix to disambiguate shared memory and system semaphores,
respectively.
\section3 X/Open System Interfaces (XSI) / System V
System V keys take the form of the name of a file in the system, and thus
have the exact same limitations as file paths do. Both QSharedMemory and
QSystemSemaphore will create this file if it does not exist when creating
the object. If auto-removal is disabled, it may also be shared between
QSharedMemory and QSystemSemaphore without conflict and can be any extant
file (for example, it can be the process executable itself, see
QCoreApplication::applicationFilePath()). The path should be an absolute
one to avoid mistakes due to different current directories.
QSharedMemory::platformSafeKey() and QSystemSemaphore::platformSafeKey()
always return an absolute path. If the input was already absolute, they
will return their input unchanged. Otherwise, they will prepend a suitable
path where the application usually has permission to create files in.
\section1 Ownership
Shared memory and system semaphore objects need to be created before use,
which is accomplished with QSharedMemory::create() or by passing
QSystemSemaphore::Create to the constructor, respectively.
On Unix systems, the Qt classes that created the object will be responsible
for cleaning up the object in question. Therefore, if the application with
that C++ object exits uncleanly (a crash, qFatal(), etc.), the object may
be left behind. If that happens, applications may fail to create the
object again and should instead attach to an existing one. For example, for
QSharedMemory:
\code
if (!shm.create(4096) && shm.error() == QSharedMemory::AlreadyExists)
shm.attach();
\endcode
Re-attaching to a QSystemSemaphore is probably unwise, as the token counter
in it is probably in an unknown state and therefore may lead to deadlocks.
\section3 POSIX Realtime
POSIX Realtime object ownership is patterned after files, in the sense that
they exist independent of any process using them or not. Qt is unable to
determine if the object is still in use, so auto-removal will remove it
even then, which will make attaching to the same object impossible but
otherwise not affecting existing attachments.
Prior to Qt 6.6, Qt never cleaned up POSIX Realtime objects, except on QNX.
\section3 X/Open System Interfaces (XSI) / System V
There are two resources managed by the Qt classes: the file the key refers
to and the object itself. QSharedMemory manages the object cooperatively:
the last attachment is responsible for removing the object itself and then
removing the key file. QSystemSemaphore will remove the object if and only
if it was passed QSystemSemaphore::Create; additionally, if it created the
key file, it will remove that too.
Since Qt 6.6, it is possible to ask either class not to clean up.
\section3 Windows
The operating system owns the object and will clean up after the last
handle to the object is closed.
\section1 Interoperability with old Qt applications
The QNativeIpcKey class was introduced in Qt 6.6. Prior to this version,
QSharedMemory and QSystemSemaphore backends were determined at the time of
Qt's own build. For Windows systems, it was always the Windows backend. For
Unix systems, it defaulted to the System V backend if the configuration
script determined it was available. If it was not available, it fell back
to the POSIX one (since Qt 4.8). The POSIX backend could be explicitly
selected using the \c{-feature-ipc_posix} option to the Qt configure
script; if it was enabled, the \c{QT_POSIX_IPC} macro would be defined.
Qt 6.6 retains the configure script option but it no longer controls the
availability of the backends. Instead, it changes what
QNativeIpcKey::legacyDefaultTypeForOs() will return. Applications that need
to retain compatibility must use this key type exclusively to guarantee
interoperability.
The API in both QSharedMemory and QSystemSemaphore had the concept of a
cross-platform key, which is now deprecated in favor of using
QSharedMemory::legacyNativeKey() and QSystemSemaphore::legacyNativeKey().
Those two functions produce the same native key as the deprecated functions
did in prior versions. If the old code was for example:
\code
QSharedMemory shm("org.example.myapplication");
QSystemSemaphore sem("org.example.myapplication");
\endcode
It can be updated to be:
\code
QSharedMemory shm(QSharedMemory::legacyNativeKey("org.example.myapplication"));
QSystemSempahore sem(QSystemSemaphore::legacyNativeKey("org.example.myapplication"));
\endcode
If the two applications exchanged native keys, there is no need to update
code such as:
\code
QSharedMemory shm;
shm.setNativeKey(key);
\endcode
Though if the older application did accept a native key, the new one may
opt to use \c{platformSafeKey()} with a second argument of
QNativeIpcKey::legacyDefaultTypeForOs().
\section3 X/Open System Interfaces (XSI) / System V
Never use existing files for QSharedMemory keys, as the old Qt application
may attempt to remove it. Instead, let QSharedMemory create it.
\section1 Interoperability with non-Qt applications
Interoperability with non-Qt applications is possible, with some limitations:
\list
\li Creation of shared memory segments must not race
\li QSharedMemory support for locking the segment is unavailable
\endlist
Communication with non-Qt applications must always be through the native
key.
QSharedMemory always maps the entire segment to memory. The non-Qt
application may choose to only map a subset of it to memory, with no ill
effects.
\section3 POSIX Realtime
POSIX shared memory can be opened using
\l{https://pubs.opengroup.org/onlinepubs/9699919799/functions/shm_open.html}{shm_open()}
and POSIX system semaphores can be opened using
\l{https://pubs.opengroup.org/onlinepubs/9699919799/functions/sem_open.html}{sem_open()}.
Both of those functions take a \c name parameter that is the result of
QNativeIpcKey::nativeKey(), encoded for file names using
QFile::encodeName() / QFile::decodeName().
\section3 Windows
Windows shared memory objects can be opened using
\l{https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-createfilemappingw}{CreateFileMappingW}
and Windows system semaphore objects can be opened using
\l{https://learn.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-createsemaphorew}{CreateSemaphoreW}.
Despite the name of both functions starting with "Create", they are able
to attach to existing objects.
The \c lpName parameter to those functions is the result of
QNativeIpcKey::nativeKey(), without transformation.
If the foreign application uses the non-Unicode version of those functions
(ending in "A"), the name can be converted to and from 8-bit using QString.
\section3 X/Open System Interfaces (XSI) / System V
System V shared memory can be obtained using
\l{https://pubs.opengroup.org/onlinepubs/9699919799/functions/shmget.html}{shmget()}
and System V system semaphores can be obtained using
\l{https://pubs.opengroup.org/onlinepubs/9699919799/functions/semget.html}{semget()}.
The \c{key} parameter to either of those functions is the result of the
\l{https://pubs.opengroup.org/onlinepubs/9699919799/functions/ftok.html}{ftok()}
function when passed the file name obtained from QNativeIpcKey::nativeKey()
with an \c id of 81 or 0x51 (the ASCII capital letter 'Q').
System V semaphore objects may contain multiple semaphores, but
QSystemSemaphore only uses the first one (number 0 for \c{sem_num}).
Both QSharedMemory and QSystemSemaphore default to removing the object
using the \c{IPC_RMID} operation to \c{shmctl()} and \c{semctl()}
respectively if they are the last attachment.
*/
|