Add ConanfileProbe item for querying conan recipe files

Conan is a popular C/C++ package manager. Conan packages including their
configuration and dependencies are usually described as conanfile.txt/.py.
ConanfileProbe runs 'conan install -g json' and parses the resulting
conanbuildinfo.json. The resulting JS object tree contains relevant information
about the dependencies and can be used to set module properties in Product,
Profile or even Module items.

Change-Id: Ied6b917f061dac67fb2260eab099bcce4037750d
Reviewed-by: Christian Kandeler <christian.kandeler@qt.io>

8 files changed, 488 insertions, 0 deletions

diff --git a/.travis.yml b/.travis.yml
index 84924fd8d..8065c71c6 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -134,6 +134,7 @@ jobs:
         - ./scripts/install-qt.sh -d ${QT_INSTALL_DIR} --version ${QTCREATOR_VERSION} qtcreator
         - choco install python
         - pip install git+https://github.com/frerich/clcache.git@cae73d8255d78db8ba11e23c51fd2c9a89e7475b
+        - pip install conan
       before_script:
         - clcache -s
       after_script:
diff --git a/doc/reference/items/probe/conanfile-probe.qdoc b/doc/reference/items/probe/conanfile-probe.qdoc
new file mode 100644
index 000000000..c18e9b322
--- /dev/null
+++ b/doc/reference/items/probe/conanfile-probe.qdoc
@@ -0,0 +1,250 @@
+/****************************************************************************
+**
+** Copyright (C) 2020 Richard Weickelt
+** Contact: https://www.qt.io/licensing/
+**
+** This file is part of Qbs.
+**
+** $QT_BEGIN_LICENSE:FDL$
+** Commercial License Usage
+** Licensees holding valid commercial Qt licenses may use this file in
+** accordance with the commercial license agreement provided with the
+** Software or, alternatively, in accordance with the terms contained in
+** a written agreement between you and The Qt Company. For licensing terms
+** and conditions see https://www.qt.io/terms-conditions. For further
+** information use the contact form at https://www.qt.io/contact-us.
+**
+** GNU Free Documentation License Usage
+** Alternatively, this file may be used under the terms of the GNU Free
+** Documentation License version 1.3 as published by the Free Software
+** Foundation and appearing in the file included in the packaging of
+** this file. Please review the following information to ensure
+** the GNU Free Documentation License version 1.3 requirements
+** will be met: https://www.gnu.org/licenses/fdl-1.3.html.
+** $QT_END_LICENSE$
+**
+****************************************************************************/
+
+/*!
+    \contentspage list-of-probes.html
+    \qmltype ConanfileProbe
+    \ingroup list-of-probes
+    \ingroup list-of-items
+    \inqmlmodule QbsLanguageItems
+    \keyword QML.ConanfileProbe
+
+    \brief Extracts information about dependencies from a Conan recipe file.
+
+    The \c ConanfileProbe interfaces \Qbs to the \l{https://conan.io/}{Conan
+    package manager}. It runs
+    \l{https://docs.conan.io/en/latest/reference/commands/consumer/install.html}{conan
+    install} on a Conan recipe file such as \c conanfile.py or \c conanfile.txt
+    and extracts all available meta information about package dependencies using
+    the \l{https://docs.conan.io/en/latest/reference/generators/json.html}{json
+    generator}. The output may be used to set up \l{Profile} items or module
+    properties in products. \c ConanfileProbe can also be used to run other
+    Conan generators and to retrieve their output.
+
+    \section1 Examples
+
+    In the following examples we assume that our project contains a \c conanfile.py.
+    This file describes all dependencies of our project. The dependency packages are
+    expected to export meta information to be consumed by our project.
+
+    \section2 Including Files Generated by Conan
+
+    Conan has a built-in
+    \l{https://docs.conan.io/en/latest/reference/generators/qbs.html}{qbs
+    generator} that creates a project file containing dummy products. This is
+    the easiest way to access dependencies, but also the least flexible one. It
+    requires each Conan package to export correct meta information and works only
+    if the dependency is a library.
+
+    \qml
+    import qbs.Probes
+
+    Project {
+        Probes.ConanfileProbe {
+            id: conan
+            conanfilePath: project.sourceDirectory + "/conanfile.py"
+            generators: "qbs"
+        }
+
+        references: conan.generatorOutputPath + "/conanbuildinfo.qbs"
+
+        CppApplication {
+            type: "application"
+            files: "main.cpp"
+            Depends { name: "mylib" }
+        }
+    }
+    \endqml
+
+    \section2 Setting Module Properties in Products
+
+    When a product depends on a Conan package that does not have a
+    dedicated \l{List of Modules}{module}, package meta information may be
+    directly fed into the \l{cpp} module.
+
+    This approach is very flexible.
+
+    \qml
+    import qbs.Probes
+
+    CppApplication {
+        Probes.ConanfileProbe {
+            id: conan
+            conanfilePath: product.sourceDirectory + "/conanfile.py"
+            options: ({opt1: "True"; opt2: "TheValue"})
+        }
+        cpp.includePaths: conan.dependencies["myLib"].include_paths
+        cpp.libraryPaths: conan.dependencies["myLib"].lib_paths
+        cpp.dynamicLibraries: conan.dependencies["mylib"].libs
+    }
+    \endqml
+
+    \section2 Setting Up a Profile
+
+    When multiple products depend on one or more Conan packages, the dependency
+    information may be combined in a \l{Profile}. This is especially useful when
+    \Qbs modules are available for some of the packages, but some of their
+    properties need to be initialized. Otherwise, it would have to be done
+    manually in global profiles.
+
+    \qml
+    import qbs.Probes
+
+    Project {
+        Probes.ConanfileProbe {
+            id: conan
+            conanfilePath: project.sourceDirectory + "/conanfile.py"
+        }
+        Profile {
+            name: "arm-gcc"
+            cpp.toolchainInstallPath: conan.dependencies["arm-none-eabi-gcc"].rootpath + "/bin"
+            cpp.toolchainPrefix: "arm-linux-gnueabi-"
+            qbs.toolchainType: "gcc"
+        }
+    }
+    \endqml
+
+    This allows fully automated dependency management, including compiler
+    toolchains and is very useful when teams work in heterougeneous
+    environments.
+
+*/
+
+/*!
+    \qmlproperty stringList ConanfileProbe::additionalArguments
+
+    Additional command line arguments that are appended to the \c{conan install}
+    command.
+
+    \defaultvalue []
+*/
+
+/*!
+    \qmlproperty path ConanfileProbe::conanfilePath
+
+    Path to a \c conanfile.py or \c conanfile.txt that is used by this probe.
+
+    This property cannot be set at the same time as \l{ConanfileProbe::}{packageReference}.
+
+    \nodefaultvalue
+*/
+
+/*!
+    \qmlproperty var ConanfileProbe::dependencies
+
+    This property contains the same information as
+    \l{ConanfileProbe::}{json}.dependencies, but instead of an array, \c
+    dependencies is a map with package names as keys for convenient access.
+
+    \readonly
+    \nodefaultvalue
+*/
+
+/*!
+    \qmlproperty path ConanfileProbe::executable
+
+    The name of or the path to the Conan executable.
+
+    \defaultvalue "conan.exe" on Windows, "conan" otherwise
+*/
+
+/*!
+    \qmlproperty path ConanfileProbe::generatedFilesPath
+
+    The path of the folder where Conan generators store their files. Each
+    instance of this probe creates a unique folder under
+    \l{Project::buildDirectory}{Project.buildDirectory}. The folder name is a
+    hash of the arguments supplied to \c{conan install}.
+
+    \readonly
+    \nodefaultvalue
+*/
+
+/*!
+    \qmlproperty stringList ConanfileProbe::generators
+
+    Conan generators to be executed by this probe. The
+    \l{https://docs.conan.io/en/latest/reference/generators/json.html}{JSON
+    generator} is always enabled. Generated files are written to the
+    \l{ConanfileProbe::generatedFilesPath}{generatedFilesPath} folder.
+
+    \sa {https://docs.conan.io/en/latest/reference/generators.html}{Available
+    generators}
+
+    \defaultvalue ["json"]
+*/
+
+/*!
+    \qmlproperty var ConanfileProbe::json
+
+    The parsed output of Conan's
+    \l{https://docs.conan.io/en/latest/reference/generators/json.html}{JSON
+    generator} as a JavaScript object.
+
+    \readonly
+    \nodefaultvalue
+*/
+
+/*!
+    \qmlproperty var ConanfileProbe::options
+
+    Options applied to \c{conan install} via the \c{-o} flag.
+    This property is an object in the form \c{key:value}.
+
+    Example:
+    \qml
+    options: ({someOpt: "True", someOtherOpt: "TheValue"})
+    \endqml
+
+    \nodefaultvalue
+*/
+
+/*!
+    \qmlproperty string ConanfileProbe::packageReference
+
+    Reference of a Conan package in the form \c{name/version@user/channel}.
+    Use this property if you want to probe an existing package in the local
+    cache or on a remote.
+
+    This property cannot be set at the same time as \l{ConanfileProbe::}{conanfilePath}.
+
+    \nodefaultvalue
+*/
+
+/*!
+    \qmlproperty var ConanfileProbe::settings
+
+    Settings applied to \c{conan install} via the \c{-s} flag.
+    This property is an object in the form \c{key:value}.
+
+    Example:
+    \qml
+    settings: ({os: "Linux", compiler: "gcc"})
+    \endqml
+
+    \nodefaultvalue
+*/
diff --git a/share/qbs/imports/qbs/Probes/ConanfileProbe.qbs b/share/qbs/imports/qbs/Probes/ConanfileProbe.qbs
new file mode 100644
index 000000000..e97e45f09
--- /dev/null
+++ b/share/qbs/imports/qbs/Probes/ConanfileProbe.qbs
@@ -0,0 +1,130 @@
+/****************************************************************************
+**
+** Copyright (C) 2020 Richard Weickelt
+** Contact: http://www.qt.io/licensing
+**
+** This file is part of Qbs.
+**
+** Commercial License Usage
+** Licensees holding valid commercial Qt licenses may use this file in
+** accordance with the commercial license agreement provided with the
+** Software or, alternatively, in accordance with the terms contained in
+** a written agreement between you and The Qt Company. For licensing terms and
+** conditions see http://www.qt.io/terms-conditions. For further information
+** use the contact form at http://www.qt.io/contact-us.
+**
+** GNU Lesser General Public License Usage
+** Alternatively, this file may be used under the terms of the GNU Lesser
+** General Public License version 2.1 or version 3 as published by the Free
+** Software Foundation and appearing in the file LICENSE.LGPLv21 and
+** LICENSE.LGPLv3 included in the packaging of this file.  Please review the
+** following information to ensure the GNU Lesser General Public License
+** requirements will be met: https://www.gnu.org/licenses/lgpl.html and
+** http://www.gnu.org/licenses/old-licenses/lgpl-2.1.html.
+**
+** In addition, as a special exception, The Qt Company gives you certain additional
+** rights.  These rights are described in The Qt Company LGPL Exception
+** version 1.1, included in the file LGPL_EXCEPTION.txt in this package.
+**
+****************************************************************************/
+
+import qbs.Process
+import qbs.File
+import qbs.FileInfo
+import qbs.TextFile
+import qbs.Utilities
+
+Probe {
+    // Inputs
+    property stringList additionalArguments: []
+    property path conanfilePath
+    property path packageReference
+    property path executable: "conan" + (qbs.hostOS.contains("windows") ? ".exe": "")
+    property stringList generators: ["json"];
+    property var options
+    property var settings
+
+    // Output
+    property var dependencies
+    property path generatedFilesPath
+    property var json
+
+    // Internal
+    // Ensure that the probe is re-run automatically whenever conanfile changes
+    // by making a file system property part of the probe's signature.
+    property int _conanfileLastModified: conanfilePath ? File.lastModified(conanfilePath) : 0
+    property path _projectBuildDirectory: project.buildDirectory
+
+    configure: {
+        if (conanfilePath && packageReference)
+            throw("conanfilePath and packageReference must not be defined at the same time.");
+
+        if (!conanfilePath && !packageReference)
+            throw("Either conanfilePath or packageReference must be defined.");
+
+        var reference = packageReference || FileInfo.cleanPath(conanfilePath);
+        console.info("Probing '" + reference + "'. This might take a while...");
+        if (conanfilePath && !File.exists(reference))
+                throw("The conanfile '" + reference + "' does not exist.");
+
+        var args = [
+            "install", reference,
+        ];
+
+        if (options) {
+            if (typeof options !== "object")
+                throw("The property 'options' must be an object.");
+            Object.keys(options).forEach(function(key,index) {
+                args.push("-o");
+                args.push(key + "=" + options[key]);
+            });
+        }
+
+        if (settings) {
+            if (typeof settings !== "object")
+                throw("The property 'settings' must be an object.");
+            Object.keys(settings).forEach(function(key,index) {
+                args.push("-s");
+                args.push(key + "=" + settings[key]);
+            });
+        }
+
+        if (!generators.contains("json"))
+            generators.push("json");
+
+        for (var i = 0; i < generators.length; i++)
+            args = args.concat(["-g", generators[i]]);
+
+        for (var i = 0; i < additionalArguments.length; i++)
+            args.push(additionalArguments[i]);
+
+        generatedFilesPath = FileInfo.cleanPath(_projectBuildDirectory +
+                                                  "/genconan/" +
+                                                  Utilities.getHash(args.join()));
+
+        args = args.concat(["-if", generatedFilesPath]);
+        var p = new Process();
+        try {
+            p.exec(executable, args, true);
+        } finally {
+            p.close();
+        }
+
+        if (generators.contains("json")) {
+            if (!File.exists(generatedFilesPath + "/conanbuildinfo.json"))
+                throw("No conanbuildinfo.json has been generated.");
+
+            var jsonFile = new TextFile(generatedFilesPath + "/conanbuildinfo.json", TextFile.ReadOnly);
+            json = JSON.parse(jsonFile.readAll());
+            jsonFile.close();
+
+            dependencies = {};
+            for (var i = 0; i < json.dependencies.length; ++i) {
+                var dep = json.dependencies[i];
+                dependencies[dep.name] = dep;
+            }
+        }
+
+        found = true;
+    }
+}
diff --git a/tests/auto/blackbox/testdata/conanfile-probe/testapp/conanfile-probe-project.qbs b/tests/auto/blackbox/testdata/conanfile-probe/testapp/conanfile-probe-project.qbs
new file mode 100644
index 000000000..ab1c68385
--- /dev/null
+++ b/tests/auto/blackbox/testdata/conanfile-probe/testapp/conanfile-probe-project.qbs
@@ -0,0 +1,22 @@
+import qbs.Probes
+import qbs.TextFile
+
+Project {
+
+    Probes.ConanfileProbe {
+        id: conan
+        conanfilePath: path + "/conanfile.py"
+        options: ({opt: "True"})
+        settings: ({os: "AIX"})
+    }
+
+    property var check: {
+        tf = new TextFile(buildDirectory + "/results.json", TextFile.WriteOnly);
+        var o = {
+            json: conan.json.deps_env_info["ENV_VAR"],
+            dependencies: conan.dependencies["testlib"].libs,
+            generatedFilesPath: conan.generatedFilesPath
+        };
+        tf.write(JSON.stringify(o));
+    }
+}
diff --git a/tests/auto/blackbox/testdata/conanfile-probe/testapp/conanfile.py b/tests/auto/blackbox/testdata/conanfile-probe/testapp/conanfile.py
new file mode 100644
index 000000000..630cf0283
--- /dev/null
+++ b/tests/auto/blackbox/testdata/conanfile-probe/testapp/conanfile.py
@@ -0,0 +1,25 @@
+from conans import ConanFile
+
+class TestApp(ConanFile):
+    name = "testapp"
+    description = "Our project package, to be inspected by the Qbs ConanfileProbe"
+    license = "none"
+    version = "6.6.6"
+
+    settings = "os"
+    options = {"opt": [True, False]}
+    default_options = {"opt": False}
+
+    requires = "testlib/1.2.3@qbs/testing"
+
+    def configure(self):
+        self.options["testlib"].opt = self.options.opt
+
+    def source(self):
+        pass
+
+    def build(self):
+        pass
+
+    def package(self):
+        pass
diff --git a/tests/auto/blackbox/testdata/conanfile-probe/testlib/conanfile.py b/tests/auto/blackbox/testdata/conanfile-probe/testlib/conanfile.py
new file mode 100644
index 000000000..983c22599
--- /dev/null
+++ b/tests/auto/blackbox/testdata/conanfile-probe/testlib/conanfile.py
@@ -0,0 +1,25 @@
+from conans import ConanFile
+
+class Testlib(ConanFile):
+    name = "testlib"
+    description = "Represents an arbitrary package, for instance on bintray"
+    license = "none"
+    version = "1.2.3"
+
+    settings = "os"
+    options = {"opt": [True, False]}
+    default_options = {"opt": False}
+
+    def source(self):
+        pass
+
+    def build(self):
+        pass
+
+    def package(self):
+        pass
+
+    def package_info(self):
+        self.cpp_info.libs = ["testlib1","testlib2"]
+        self.env_info.ENV_VAR = "TESTLIB_ENV_VAL"
+        self.user_info.user_var = "testlib_user_val"
diff --git a/tests/auto/blackbox/tst_blackbox.cpp b/tests/auto/blackbox/tst_blackbox.cpp
index 121f60dd8..a30d57be9 100644
--- a/tests/auto/blackbox/tst_blackbox.cpp
+++ b/tests/auto/blackbox/tst_blackbox.cpp
@@ -1737,6 +1737,40 @@ void TestBlackbox::cxxLanguageVersion_data()
                             std::make_pair(QString("msvc-new"), QString("/std:"))});
 }
 
+void TestBlackbox::conanfileProbe()
+{
+    QString executable = findExecutable({"conan"});
+    if (executable.isEmpty())
+        QSKIP("conan is not installed or not available in PATH.");
+
+    // We first build a dummy package testlib and use that as dependency
+    // in the testapp package.
+    QDir::setCurrent(testDataDir + "/conanfile-probe/testlib");
+    QStringList arguments { "create", "-o", "opt=True", "-s", "os=AIX", ".",
+                            "testlib/1.2.3@qbs/testing" };
+    QProcess conan;
+    conan.start(executable, arguments);
+    QVERIFY(waitForProcessSuccess(conan));
+
+    QDir::setCurrent(testDataDir + "/conanfile-probe/testapp");
+    QCOMPARE(runQbs(QbsRunParameters("resolve", {"--force-probe-execution"})), 0);
+
+    QFile file(relativeBuildDir() + "/results.json");
+    QVERIFY(file.open(QIODevice::ReadOnly));
+    QVariantMap actualResults = QJsonDocument::fromJson(file.readAll()).toVariant().toMap();
+    const auto generatedFilesPath = actualResults.take("generatedFilesPath").toString();
+    // We want to make sure that generatedFilesPath is under the project directory,
+    // but we don't care about the actual name.
+    QVERIFY(directoryExists(relativeBuildDir() + "/genconan/"
+                            + QFileInfo(generatedFilesPath).baseName()));
+
+    const QVariantMap expectedResults = {
+        { "json", "TESTLIB_ENV_VAL" },
+        { "dependencies", QVariantList{"testlib1", "testlib2"} },
+    };
+    QCOMPARE(actualResults, expectedResults);
+}
+
 void TestBlackbox::cpuFeatures()
 {
     QDir::setCurrent(testDataDir + "/cpu-features");
diff --git a/tests/auto/blackbox/tst_blackbox.h b/tests/auto/blackbox/tst_blackbox.h
index 2b83b14e3..c1b20956d 100644
--- a/tests/auto/blackbox/tst_blackbox.h
+++ b/tests/auto/blackbox/tst_blackbox.h
@@ -83,6 +83,7 @@ private slots:
     void conflictingArtifacts();
     void cxxLanguageVersion();
     void cxxLanguageVersion_data();
+    void conanfileProbe();
     void cpuFeatures();
     void dependenciesProperty();
     void deprecatedProperty();