Introduce module providers

If a dependency is not found, we now search for a matching module
provider that can generate one for us.
We also provide a generic fall-back provider which uses pkg-config to
locate the dependency (but could be extended to incorporate other
methods in the future). This is the most important part of this change
for practical purposes, as it makes hundreds of popular libraries
available for use in qbs projects without users having to write any
boilerplate code.
In a future patch, a module provider could also be used to implement the
functionality of the qtprofilesetup library, relieving users of the need
to create a profile for building Qt applications.

[ChangeLog] The Depends item now falls back to pkg-config to locate
dependencies whose names do not correspond to a qbs module.

Fixes: QBS-1107
Change-Id: Ifd4f05c237cf58cd9fe707c3da648d3dbb33e82b
Reviewed-by: Leena Miettinen <riitta-leena.miettinen@qt.io>
Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>

9 files changed, 281 insertions, 13 deletions

diff --git a/doc/howtos.qdoc b/doc/howtos.qdoc
index 7ef1fc086..a2595cffd 100644
--- a/doc/howtos.qdoc
+++ b/doc/howtos.qdoc
@@ -41,6 +41,7 @@
     \li \l{How do I make sure my generated sources are getting compiled?}
     \li \l{How do I run my autotests?}
     \li \l{How do I create a module for a third-party library?}
+    \li \l{How do I build against libraries that provide pkg-config files?}
     \li \l{How do I create application bundles and frameworks on iOS, macOS, tvOS, and watchOS?}
     \li \l{How do I apply C/C++ preprocessor macros to only a subset of the files in my product?}
     \li \l{How do I make the state of my Git repository available to my source files?}
@@ -315,6 +316,20 @@
     static library; see the \l{How do I make my app build against my library?} section for an
     example.
 
+    \section1 How do I build against libraries that provide pkg-config files?
+
+    Just add a \l Depends item that matches the name of the pkg-config module, and \QBS
+    will automatically employ \l{https://www.freedesktop.org/wiki/Software/pkg-config}{pkg-config}
+    to find the headers and libraries if no matching \QBS module can be found. For instance,
+    to build against the OpenSSL library, you would write this:
+    \code
+    Depends { name: "openssl" }
+    \endcode
+    That's it. The pkg-config behavior can be fine-tuned via the \l pkgconfig module,
+    but normally you will not need to pull it in explicitly.
+
+    Internally, this functionality is implemented via \l {Module Providers}
+
     \section1 How do I apply C/C++ preprocessor macros to only a subset of the files in my product?
 
     Use a \l{Group} item to define a subset of project files. To add
diff --git a/doc/qbs.qdoc b/doc/qbs.qdoc
index 14787ffbf..44678bc40 100644
--- a/doc/qbs.qdoc
+++ b/doc/qbs.qdoc
@@ -68,6 +68,7 @@
                \li  \l{Generators}
                \li  \l{Multiplexing}
                \li  \l{Custom Modules and Items}
+               \li  \l{Module Providers}
            \endlist
        \li  \l{How-tos}
        \li  \l{Reference}
@@ -791,6 +792,7 @@
         \li  \l{Generators}
         \li  \l{Multiplexing}
         \li  \l{Custom Modules and Items}
+        \li  \l{Module Providers}
     \endlist
 
 */
@@ -1351,7 +1353,7 @@
     \contentspage index.html
     \previouspage multiplexing.html
     \page custom-modules.html
-    \nextpage howtos.html
+    \nextpage module-providers.html
 
     \title Custom Modules and Items
 
@@ -1421,6 +1423,62 @@
 
 /*!
     \contentspage index.html
+    \previouspage custom-modules.html
+    \page module-providers.html
+    \nextpage howtos.html
+
+    \title Module Providers
+
+    There are use cases for which a pre-defined module is not flexible enough.
+    For instance, the overall set of modules related to a certain task might depend
+    on some information present on the local platform.
+
+    \note Module providers are an advanced concept that you will rarely need to use directly.
+    Reading this section is not required for most people's everyday work.
+
+    \section1 How \QBS Uses Module Providers
+
+    If \QBS encounters a \l Depends item whose name does not match a known module,
+    it checks whether such a module can be generated. This procedure works as follows:
+    \list 1
+        \li All \l{Project::qbsSearchPaths}{search paths} are scanned for a file called
+            \c {module-providers/<name>/provider.qbs}, where \c <name> is the name of the dependency
+            as specified in the \c Depends item. Multi-component names such as "a.b" are turned
+            into nested directories, and each of them is scanned, starting with the deepest path.
+            For instance, if the dependency's name is \c {a.b}, then \QBS will look for
+            \c {a/b/provider.qbs} and then \c {a/provider.qbs}.
+        \li If such a file is found, it needs to contain a \l ModuleProvider item. This item
+            is instantiated, which potentially leads to the creation of one or more modules,
+            and \QBS retrieves the search paths to find these modules from the item.
+            The details are described in the \l ModuleProvider documentation.
+        \li If a matching module provider was found and provided new search paths,
+            a second attempt will be made to locate the dependency using the new paths.
+            The search for a matching module provider ends as soon as one was found, regardless
+            of whether it created any modules or not.
+        \li If no matching module provider was found in any of the search paths, \QBS will fall back
+            to a generic module provider, which creates a module that attempts to locate the
+            dependency via \c pkg-config.
+            This fallback mechanism can be disabled in the respective
+            \l{Depends::enableFallback}{Depends} item or globally via the
+            \l{no-fallback-module-provider}{--no-fallback-module-provider} option.
+    \endlist
+
+    \section1 Parameterizing Module Providers
+
+    You can pass information to module providers from the command line, via profiles or
+    from within a product, in a similar way as you would do for modules. For instance, the
+    following invocation of \QBS passes information to two module providers \c a and \c b:
+    \code
+    $ qbs moduleProviders.a.p1:true moduleProviders.a.p2:true moduleProviders.b.p:false
+    \endcode
+    \QBS will set the properties of the respective module providers accordingly.
+    In the above example, module provider \c a needs to declare two boolean properties \c p1
+    and \c p2, and they will be set to \c true and \c false, respectively.
+
+*/
+
+/*!
+    \contentspage index.html
     \previouspage shell.html
     \page generators.html
     \nextpage multiplexing.html
diff --git a/doc/reference/cli/builtin/cli-build.qdoc b/doc/reference/cli/builtin/cli-build.qdoc
index cffb19d49..7844dceea 100644
--- a/doc/reference/cli/builtin/cli-build.qdoc
+++ b/doc/reference/cli/builtin/cli-build.qdoc
@@ -79,6 +79,8 @@
     \include cli-options.qdocinc products-specified
     \include cli-options.qdocinc settings-dir
     \include cli-options.qdocinc show-progress
+    \target no-fallback-module-provider
+    \include cli-options.qdocinc no-fallback-module-provider
     \include cli-options.qdocinc wait-lock
 
     \section1 Parameters
diff --git a/doc/reference/cli/builtin/cli-resolve.qdoc b/doc/reference/cli/builtin/cli-resolve.qdoc
index 7170856f2..b13f3de3d 100644
--- a/doc/reference/cli/builtin/cli-resolve.qdoc
+++ b/doc/reference/cli/builtin/cli-resolve.qdoc
@@ -56,6 +56,7 @@
     \include cli-options.qdocinc more-verbose
     \include cli-options.qdocinc settings-dir
     \include cli-options.qdocinc show-progress
+    \include cli-options.qdocinc no-fallback-module-provider
 
     \section1 Parameters
 
diff --git a/doc/reference/cli/cli-options.qdocinc b/doc/reference/cli/cli-options.qdocinc
index 254444dcb..2111d8a2d 100644
--- a/doc/reference/cli/cli-options.qdocinc
+++ b/doc/reference/cli/cli-options.qdocinc
@@ -459,6 +459,16 @@
 
 //! [show-progress]
 
+//! [no-fallback-module-provider]
+
+    \section2 \c --no-fallback-module-provider
+
+    If this option is set, then \QBS will not fall back to a pkg-config based
+    \l{Module Providers}{module provider} if a dependency is not found.
+
+//! [no-fallback-module-provider]
+
+
 //! [setup-tools-system]
 
     \section2 \c {--system}
diff --git a/doc/reference/items/language/depends.qdoc b/doc/reference/items/language/depends.qdoc
index b57c4a79f..8a3e23ba9 100644
--- a/doc/reference/items/language/depends.qdoc
+++ b/doc/reference/items/language/depends.qdoc
@@ -38,25 +38,21 @@
     A Depends item can appear inside a \l{Product} or \l{Module} item.
 
     For example, the following product will load the \l{cpp} module. In
-    addition, it will try to load modules that may or may not exist, and in the
-    latter case use a fallback.
+    addition, it will try to load modules that may or may not exist, and
+    pass this information on to the compiler.
 
     \code
     Product {
         Depends { name: "cpp" }
         Depends {
-            name: "awesome_module"
+            name: "optional_module"
             versionAtLeast: "2.0"
             required: false
         }
-        Depends {
-            name: "adequate_module"
-            condition: !awesome_module.present
-            required: false
-        }
-        Depends {
-            name: "inferior_module"
-            condition: !awesome_module.present && !adequate_module.present
+
+        Properties {
+            condition: optional_module.present
+            cpp.defines: "HAS_OPTIONAL_MODULE"
         }
 
         // ...
@@ -190,3 +186,12 @@
 
     \nodefaultvalue
 */
+
+/*!
+    \qmlproperty bool Depends::enableFallback
+
+    Whether to fall back to a pkg-config based \l{Module Providers}{module provider}
+    if the dependency is not found.
+
+    \defaultvalue \c true
+*/
diff --git a/doc/reference/items/language/module.qdoc b/doc/reference/items/language/module.qdoc
index e5472983f..7dd5249b2 100644
--- a/doc/reference/items/language/module.qdoc
+++ b/doc/reference/items/language/module.qdoc
@@ -27,7 +27,7 @@
 /*!
     \contentspage list-of-language-items.html
     \previouspage JobLimit
-    \nextpage Parameter
+    \nextpage ModuleProvider
     \qmltype Module
     \inqmlmodule QbsLanguageItems
     \ingroup list-of-items
diff --git a/doc/reference/items/language/moduleprovider.qdoc b/doc/reference/items/language/moduleprovider.qdoc
new file mode 100644
index 000000000..ddbc25959
--- /dev/null
+++ b/doc/reference/items/language/moduleprovider.qdoc
@@ -0,0 +1,108 @@
+/****************************************************************************
+**
+** Copyright (C) 2017 The Qt Company Ltd.
+** Contact: https://www.qt.io/licensing/
+**
+** This file is part of Qbs.
+**
+** $QT_BEGIN_LICENSE:FDL$
+** Commercial License Usage
+** Licensees holding valid commercial Qt licenses may use this file in
+** accordance with the commercial license agreement provided with the
+** Software or, alternatively, in accordance with the terms contained in
+** a written agreement between you and The Qt Company. For licensing terms
+** and conditions see https://www.qt.io/terms-conditions. For further
+** information use the contact form at https://www.qt.io/contact-us.
+**
+** GNU Free Documentation License Usage
+** Alternatively, this file may be used under the terms of the GNU Free
+** Documentation License version 1.3 as published by the Free Software
+** Foundation and appearing in the file included in the packaging of
+** this file. Please review the following information to ensure
+** the GNU Free Documentation License version 1.3 requirements
+** will be met: https://www.gnu.org/licenses/fdl-1.3.html.
+** $QT_END_LICENSE$
+**
+****************************************************************************/
+/*!
+    \contentspage list-of-language-items.html
+    \previouspage Module
+    \nextpage Parameter
+    \qmltype ModuleProvider
+    \inqmlmodule QbsLanguageItems
+    \ingroup list-of-items
+    \keyword QML.ModuleProvider
+
+    \brief Creates modules on demand.
+
+    The \c ModuleProvider item implements the module creation part of the procedure described
+    in the \l {Module Providers} overview. It is always located in a file called \c provider.qbs.
+
+    The actual module creation is done on the right-hand side of the
+    \l{ModuleProvider::relativeSearchPaths}{relativeSearchPaths} property.
+
+    Here is a complete minimal example of a module provider. It just creates an empty module.
+    If you put this item into the file \c {module-providers/mymodule/provider.qbs}
+    in your project source directory, you will be able to successfully build a product which
+    contains a dependency on the module \c mymodule.
+    \code
+        import qbs.File
+        import qbs.FileInfo
+        import qbs.TextFile
+
+        ModuleProvider {
+            relativeSearchPaths: {
+                var moduleDir = FileInfo.joinPaths(outputBaseDir, "modules", name);
+                File.makePath(moduleDir);
+                var moduleFilePath = FileInfo.joinPaths(moduleDir, name + ".qbs");
+                var moduleFile = new TextFile(moduleFilePath, TextFile.WriteOnly);
+                moduleFile.writeLine("Module {");
+                moduleFile.writeLine("}");
+                moduleFile.close();
+                return "";
+            }
+        }
+    \endcode
+*/
+
+/*!
+    \qmlproperty string ModuleProvider::name
+
+    The name of the module provider.
+
+    This property is set by \QBS. For simple dependency names, it is the name of the dependency
+    as specified in the \l Depends item. If the dependency name consists of multiple components,
+    the value is the name up until (and including) the component that corresponds to the directory
+    the provider was found in. For instance, if the dependency is \c {x.m1} and the provider was
+    found in \c {module-providers/x/m1/provider.qbs}, then \c name is \c {x.m1}.
+    If the provider was found in \c {module-providers/x/provider.qbs}, then \c name is \c x.
+*/
+
+/*!
+    \qmlproperty string ModuleProvider::outputBaseDir
+
+    The path under which the new modules should be created when \l relativeSearchPaths
+    is evaluated. The path is unique for the current provider in the given configuration.
+
+    This property is set by \QBS.
+*/
+
+/*!
+    \qmlproperty stringList ModuleProvider::relativeSearchPaths
+
+    This property gets evaluated by \QBS to retrieve new search paths with which
+    to re-attempt the module look-up.
+
+    It is here where you need to put the code that creates the new module files.
+    Use the directory structure explained in \l {Custom Modules and Items}.
+    That is, the file for a module called \c m will be located in a directory \c {modules/m/},
+    anchored at \l outputBaseDir.
+
+    The return value is the list of search paths required to find the new module,
+    relative to \l outputBaseDir. In most cases, only a single search path will be required,
+    in which case a single-element list containing an empty string should be returned
+    (or just the empty string, because of \QBS' auto-conversion feature).
+
+    The returned list can also be empty, which means that the module provider was not able
+    to generate any modules in this environment.
+*/
diff --git a/doc/reference/modules/pkgconfig-module.qdoc b/doc/reference/modules/pkgconfig-module.qdoc
new file mode 100644
index 000000000..898349628
--- /dev/null
+++ b/doc/reference/modules/pkgconfig-module.qdoc
@@ -0,0 +1,69 @@
+/****************************************************************************
+**
+** Copyright (C) 2019 The Qt Company Ltd.
+** Contact: https://www.qt.io/licensing/
+**
+** This file is part of Qbs.
+**
+** $QT_BEGIN_LICENSE:FDL$
+** Commercial License Usage
+** Licensees holding valid commercial Qt licenses may use this file in
+** accordance with the commercial license agreement provided with the
+** Software or, alternatively, in accordance with the terms contained in
+** a written agreement between you and The Qt Company. For licensing terms
+** and conditions see https://www.qt.io/terms-conditions. For further
+** information use the contact form at https://www.qt.io/contact-us.
+**
+** GNU Free Documentation License Usage
+** Alternatively, this file may be used under the terms of the GNU Free
+** Documentation License version 1.3 as published by the Free Software
+** Foundation and appearing in the file included in the packaging of
+** this file. Please review the following information to ensure
+** the GNU Free Documentation License version 1.3 requirements
+** will be met: https://www.gnu.org/licenses/fdl-1.3.html.
+** $QT_END_LICENSE$
+**
+****************************************************************************/
+
+/*!
+    \contentspage index.html
+    \qmltype pkgconfig
+    \inqmlmodule QbsModules
+    \since 1.13
+
+    \brief Allows to configure the pkg-config tool.
+
+    The \c pkgconfig module is used to fine-tune the behavior of the \c {pkg-config} tool,
+    which is
+    \l{How do I build against libraries that provide pkg-config files?}{potentially employed}
+    when looking up dependencies.
+*/
+
+/*!
+    \qmlproperty string pkgconfig::executableFilePath
+
+    The path to the \c {pkg-config} executable.
+
+    \defaultvalue auto-detected
+*/
+
+/*!
+    \qmlproperty stringList pkgconfig::libDirs
+
+    Set this if you need to overwrite the default search directories. The values
+    given here will be forwarded to the tool via the \c PKG_CONFIG_LIBDIR environment
+    variable.
+    \note You do not need to set this for cross-compilation in order to point
+          \c {pkg-config} to the sysroot. \QBS does that for you.
+
+    \nodefaultvalue
+*/
+
+/*!
+    \qmlproperty bool pkgconfig::staticMode
+
+    If this property is \c true, then calls to \c{pkg-config} will include the
+    \c{--static} option. Set this if your product is to be linked statically.
+
+    \defaultvalue \c false
+*/