Use a dedicated launcher process to run all ProcessCommands

On Linux (and very likely other Unices as well), QProcess::start()
incurs a certain overhead, because forking off the child process
duplicates some of the parent process' resources. This overhead appears
to be proportional to the amount of memory allocated by the parent
process, presumably due to page table entries getting copied. This has
consequences for qbs, particularly when being used from an IDE such as
Qt Creator, which has a higher memory footprint than the command line
tool. When using a high job count, as is typical on machines with lots
of CPU cores or in a distributed compilation environment, the following
problems were observed:
    - High CPU load in the starting process (Qt Creator). Profiling
      showed that most of the time was spent in fork() and related
      functions.
    - As a result, the number of parallel jobs stalled at a value
      well below the requested one, slowing down the build.
    - In some cases, QProcess::start() failed altogether, emitting
      a message such as "fork() failed: Could not allocate memory".
We solve these issues by outsourcing the starting of ProcessCommands to
a dedicated launcher tool with modest memory requirements. For each qbs
process, we have one instance of this tool running while a build job is
going on. Communication with qbs happens via QLocalSocket. The protocol
is encapsulated in a QProcess replacement, so almost no changes to
existing code were necessary.
No performance regressions were observed when using lower job counts.
This patch will also enable us to properly support the incredibuild tool
in IDEs such as Qt Creator, which you do not want to start via
ibconsole.

[ChangeLog] Improved scalability of parallel builds on Linux by starting
Process commands via a dedicated launcher process.

Change-Id: I8966c00a2d67a94c3a380f0e089d61eda122209e
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@qt.io>
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>

1 files changed, 3 insertions, 3 deletions

diff --git a/src/lib/corelib/buildgraph/processcommandexecutor.h b/src/lib/corelib/buildgraph/processcommandexecutor.h
index d04f305c3..b96f6a0f1 100644
--- a/src/lib/corelib/buildgraph/processcommandexecutor.h
+++ b/src/lib/corelib/buildgraph/processcommandexecutor.h
@@ -42,8 +42,8 @@
 
 #include "abstractcommandexecutor.h"
 
-#include <QtCore/qprocess.h>
-#include <QtCore/qprocess.h>
+#include <tools/qbsprocess.h>
+
 #include <QtCore/qstring.h>
 
 namespace qbs {
@@ -87,7 +87,7 @@ private:
     QStringList m_arguments;
     QString m_shellInvocation;
 
-    QProcess m_process;
+    QbsProcess m_process;
     QProcessEnvironment m_buildEnvironment;
     QProcessEnvironment m_commandEnvironment;
     QString m_responseFileName;