path: root/doc
diff options
authorChristian Kandeler <christian.kandeler@qt.io>2018-07-24 12:44:28 +0200
committerChristian Kandeler <christian.kandeler@qt.io>2018-08-08 11:18:45 +0000
commit0bc341e3dcba1f561a685d72cfa0f4b1a7f697dd (patch)
treebc79d5bc3f8bbbda56b72b0825c70bb71127d863 /doc
parent43e3b7ad7e5e68bae839feb74cdbcde48424c065 (diff)
Add support for job pools
Commands can now be assigned to an arbitrary job pool and a limit for the number of concurrently running jobs in such pools can be provided in a number of ways: - via the build command line: qbs --job-limits linker:1 - via the settings: qbs config preferences.jobLimit.linker 1 - in a project file: JobLimit { jobPool: "linker"; jobCount: 1 } We provide two job pools ourselves with the cpp module: "compiler" and "linker". [ChangeLog] Added the concept of job pools for limiting concurrent execution of commands by type Task-number: QBS-743 Change-Id: Ib3f361dbc73093e342bf0eba0daf2079a2b3a8ce Reviewed-by: Oswald Buddenhagen <oswald.buddenhagen@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
Diffstat (limited to 'doc')
8 files changed, 207 insertions, 2 deletions
diff --git a/doc/howtos.qdoc b/doc/howtos.qdoc
index dd1b45130..7982ae6d4 100644
--- a/doc/howtos.qdoc
+++ b/doc/howtos.qdoc
@@ -44,6 +44,7 @@
\li \l{How do I create application bundles and frameworks on iOS, macOS, tvOS, and watchOS?}
\li \l{How do I apply C/C++ preprocessor macros to only a subset of the files in my product?}
\li \l{How do I make the state of my Git repository available to my source files?}
+ \li \l{How do I limit the number of concurrent jobs for the linker only?}
\section1 How do I build a Qt-based project?
@@ -366,4 +367,52 @@
This value is also available via the \l{vcs::repoState}{vcs.repoState}
+ \section1 How do I limit the number of concurrent jobs for the linker only?
+ \target job-pool-howto
+ While it is usually desirable to run as many compiler jobs as there are CPU cores,
+ the same is not true for linker jobs. The reason is that linkers are typically
+ I/O bound rather than CPU bound. When building large libraries, they also tend
+ to use up enormous amounts of memory. Therefore, we'd like to make sure that
+ only a few linkers are running at the same time without limiting other types
+ of jobs. In \QBS, this is achieved via \e{job pools}. There are several ways
+ to make use of them.
+ Firstly, you can provide a limit via the command line:
+ \code
+ $ qbs --job-limits linker:4
+ \endcode
+ The above call instructs \QBS to run at most four linker instances at the same
+ time, while leaving the general number of concurrent jobs at the default
+ value, which is derived from the number of CPU cores.
+ The \c linker string on the command line refers to the job pool of the same
+ name, which the \l{cpp-job-pools}{cpp module} assigns to all its commands that
+ invoke a linker.
+ Secondly, you can set a limit via the settings, either generally
+ or for a specific profile:
+ \code
+ $ qbs config preferences.jobLimit.linker 4
+ $ qbs config profiles.myprofile.preferences.jobLimit.linker 2
+ \endcode
+ And finally, you can also set the limit per project or per product, using a
+ \l JobLimit item:
+ \code
+ Product {
+ name: "my_huge_library"
+ JobLimit {
+ jobPool: "linker"
+ jobCount: 1
+ }
+ // ...
+ }
+ \endcode
+ The above construct ensures that this specific library is never linked at
+ the same time as any other binary in the project.
+ Job limits set on the command line override those from the settings, which in turn
+ override the ones defined within a project. Use the \c{--enforce-project-job-limits}
+ option to give the job limits defined via \c JobLimit items maximum precedence.
diff --git a/doc/reference/cli/builtin/cli-build.qdoc b/doc/reference/cli/builtin/cli-build.qdoc
index 8e4b8ed44..cffb19d49 100644
--- a/doc/reference/cli/builtin/cli-build.qdoc
+++ b/doc/reference/cli/builtin/cli-build.qdoc
@@ -68,6 +68,7 @@
\target build-force-probe-execution
\include cli-options.qdocinc force-probe-execution
\include cli-options.qdocinc jobs
+ \include cli-options.qdocinc job-limits
\include cli-options.qdocinc keep-going
\include cli-options.qdocinc less-verbose
\include cli-options.qdocinc log-level
diff --git a/doc/reference/cli/cli-options.qdocinc b/doc/reference/cli/cli-options.qdocinc
index 189a3526a..e6d909963 100644
--- a/doc/reference/cli/cli-options.qdocinc
+++ b/doc/reference/cli/cli-options.qdocinc
@@ -249,6 +249,22 @@
//! [jobs]
+//! [job-limits]
+ \section2 \c {--job-limits <pool1>:<limit1>[,<pool2>:<limit2>...]}
+ Sets pool-specific job limits. See \l{job-pool-howto}{here} for more information on
+ job pools.
+ \section2 \c {--enforce-project-job-limits}
+ Normally, job limits defined in project files via the \l JobLimit item get overridden
+ by those set on the command line. If this option is passed, they get maximum priority
+ instead. Use it if there are product-specific limits that make more sense for
+ that part of the code base than the generic ones you'd like to apply globally.
+//! [jobs]
//! [keep-going]
\section2 \c --keep-going|-k
diff --git a/doc/reference/commands.qdoc b/doc/reference/commands.qdoc
index 8f0392c32..eb93e8f7a 100644
--- a/doc/reference/commands.qdoc
+++ b/doc/reference/commands.qdoc
@@ -103,6 +103,13 @@
All other values are mapped to the default color.
+ \li \c jobPool
+ \li string
+ \li empty
+ \li Determines which job pool the command will use. An empty
+ string, which is the default, stands for the global job pool.
+ See \l{JobLimit}{here} and \l{job-pool-howto}{here} for more information on job pools.
+ \row
\li \c silent
\li bool
\li false
diff --git a/doc/reference/items/language/group.qdoc b/doc/reference/items/language/group.qdoc
index dae647060..caeffcfaa 100644
--- a/doc/reference/items/language/group.qdoc
+++ b/doc/reference/items/language/group.qdoc
@@ -28,7 +28,7 @@
\contentspage list-of-language-items.html
\previouspage FileTagger
- \nextpage Module
+ \nextpage JobLimit
\qmltype Group
\inqmlmodule QbsLanguageItems
\ingroup list-of-items
diff --git a/doc/reference/items/language/joblimit.qdoc b/doc/reference/items/language/joblimit.qdoc
new file mode 100644
index 000000000..66697ba3e
--- /dev/null
+++ b/doc/reference/items/language/joblimit.qdoc
@@ -0,0 +1,107 @@
+** Copyright (C) 2018 The Qt Company Ltd.
+** Contact: https://www.qt.io/licensing/
+** This file is part of Qbs.
+** Commercial License Usage
+** Licensees holding valid commercial Qt licenses may use this file in
+** accordance with the commercial license agreement provided with the
+** Software or, alternatively, in accordance with the terms contained in
+** a written agreement between you and The Qt Company. For licensing terms
+** and conditions see https://www.qt.io/terms-conditions. For further
+** information use the contact form at https://www.qt.io/contact-us.
+** GNU Free Documentation License Usage
+** Alternatively, this file may be used under the terms of the GNU Free
+** Documentation License version 1.3 as published by the Free Software
+** Foundation and appearing in the file included in the packaging of
+** this file. Please review the following information to ensure
+** the GNU Free Documentation License version 1.3 requirements
+** will be met: https://www.gnu.org/licenses/fdl-1.3.html.
+ \contentspage list-of-language-items.html
+ \previouspage Group
+ \nextpage Module
+ \qmltype JobLimit
+ \inqmlmodule QbsLanguageItems
+ \ingroup list-of-items
+ \keyword QML.JobLimit
+ \brief Restricts concurrent execution of jobs in a given pool.
+ In addition to the global limit on concurrently running commands, a project might
+ want to restrict concurrent execution of certain types of commands even further,
+ for instance because they are not well-suited to share certain types of resources.
+ In the following example, we define a rule that runs a tool of which at most one
+ instance can be running for the same project at any given time:
+ \code
+ Rule {
+ // ...
+ prepare: {
+ var cmd = new Command("my-exclusive-tool", [project.buildDirectory]);
+ cmd.description = "running the exclusive tool";
+ cmd.jobPool = "exclusive_tool";
+ return cmd;
+ }
+ }
+ JobLimit {
+ jobPool: "exclusive_tool"
+ jobCount: 1
+ }
+ \endcode
+ \c JobLimit items can appear inside \l Product, \l Project and \l Module items.
+ In the case of collisions, that is, items matching the same job pool but setting
+ different values, the ones defined inside products have the highest precedence,
+ and the ones inside modules have the lowest. Items defined in sub-projects have
+ higher precedence than those defined in parent projects. For items with the same
+ precedence level, the most restrictive one is chosen, that is, the one with the
+ lowest job number greater than zero.
+ \see {How do I limit the number of concurrent jobs for the linker only?}
+ \qmlproperty bool JobLimit::condition
+ Determines whether the job limit is active.
+ If this property is set to \c false, the job limit is ignored.
+ \defaultvalue \c true
+ \qmlproperty string JobLimit::jobCount
+ The maximum number of commands in the given \l{jobPool}{job pool} that can run
+ concurrently.
+ A value of zero means "unlimited", negative values are not allowed.
+ \note The global job limit always applies: For instance, if you set this
+ property to 100 for some job pool, and "-j 8" was given on the
+ command line, then no more than eight instances of commands from
+ the respective job pool will run at any time.
+ This property must always be set.
+ \nodefaultvalue
+ \qmlproperty string JobLimit::jobPool
+ The job pool to which apply the limit.
+ This property must always be set to a non-empty value.
+ \nodefaultvalue
diff --git a/doc/reference/items/language/module.qdoc b/doc/reference/items/language/module.qdoc
index bfaedacc9..e5472983f 100644
--- a/doc/reference/items/language/module.qdoc
+++ b/doc/reference/items/language/module.qdoc
@@ -26,7 +26,7 @@
\contentspage list-of-language-items.html
- \previouspage Group
+ \previouspage JobLimit
\nextpage Parameter
\qmltype Module
\inqmlmodule QbsLanguageItems
diff --git a/doc/reference/modules/cpp-module.qdoc b/doc/reference/modules/cpp-module.qdoc
index 0818af28d..386ac7553 100644
--- a/doc/reference/modules/cpp-module.qdoc
+++ b/doc/reference/modules/cpp-module.qdoc
@@ -217,6 +217,31 @@
This file tag only has an effect with GCC-like toolchains. The linker needs to be
+ \section2 Relevant Job Pools
+ \target cpp-job-pools
+ \table
+ \header
+ \li Pool
+ \li Since
+ \li Description
+ \row
+ \li \c{"assembler"}
+ \li 1.13
+ \li The job pool used by rules that run the toolchain's assembler. This is only
+ relevant for direct invocations of the assembler binary, not for running it
+ indirectly via the compiler.
+ \row
+ \li \c{"compiler"}
+ \li 1.13
+ \li The job pool used by rules that run a compiler. All language variants use
+ the same pool.
+ \row
+ \li \c{"linker"}
+ \li 1.13
+ \li The job pool used by rules that run a linker.
+ \endtable