QProcess/Linux: fix file descriptor leak in case of failed child start

If the child failed to start, execChild() would (unnecessarily) store -1
in childStartedPipe[1] after writing the failure message to it and
closing that pipe. This had worked for the previous 20 years of QProcess
existence, because that was run in the child process. However, with 6.5
commit e1a787a76ed462e4ed49db78a40c6d7e272182d7 (cherry-picked to 6.4)
we implemented vfork-like behavior, meaning the child would share memory
with the parent and ran before the parent, so startProcess() failed to
close it:

    // parent
    // close the ends we don't use and make all pipes non-blocking
    qt_safe_close(childStartedPipe[1]);

Also updated the docs to account for the fact that this is a vfork()
environment and, moreover, not using the vfork() function from glibc on
Linux.

[ChangeLog][QtCore][QProcess] Fixed a file descriptor leak in QProcess
when running on Linux 5.4 or later, if the executable being run failed
to start (for example, the file for the executable didn't exist).

Pick-to: 6.4.3 6.4 6.5
Task-number: QTBUG-111243
Change-Id: I7f354474adce419ca6c2fffd17481002e4853cc3
Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@gmx.de>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>

1 files changed, 7 insertions, 3 deletions

diff --git a/src/corelib/io/qprocess_unix.cpp b/src/corelib/io/qprocess_unix.cpp
index 449fd14c5c..7607901021 100644
--- a/src/corelib/io/qprocess_unix.cpp
+++ b/src/corelib/io/qprocess_unix.cpp
@@ -358,7 +358,7 @@ bool QProcessPrivate::openChannel(Channel &channel)
     }
 }
 
-void QProcessPrivate::commitChannels()
+void QProcessPrivate::commitChannels() const
 {
     // copy the stdin socket if asked to (without closing on exec)
     if (stdinChannel.pipe[0] != INVALID_Q_PIPE)
@@ -516,7 +516,12 @@ void QProcessPrivate::startProcess()
         ::fcntl(stderrChannel.pipe[0], F_SETFL, ::fcntl(stderrChannel.pipe[0], F_GETFL) | O_NONBLOCK);
 }
 
-void QProcessPrivate::execChild(const char *workingDir, char **argv, char **envp)
+// IMPORTANT:
+//
+// This function is called in a vfork() context on some OSes (notably, Linux
+// with forkfd), so it MUST NOT modify any non-local variable because it's
+// still sharing memory with the parent process.
+void QProcessPrivate::execChild(const char *workingDir, char **argv, char **envp) const
 {
     ::signal(SIGPIPE, SIG_DFL);         // reset the signal that we ignored
 
@@ -556,7 +561,6 @@ void QProcessPrivate::execChild(const char *workingDir, char **argv, char **envp
 report_errno:
     error.code = errno;
     qt_safe_write(childStartedPipe[1], &error, sizeof(error));
-    childStartedPipe[1] = -1;
 }
 
 bool QProcessPrivate::processStarted(QString *errorMessage)