Allow building shiboken2 and PySide2 as separate wheels

Actually this creates 3 wheel packages:
- shiboken2 (the python module and libshiboken shared library)
- shiboken2-generator (contains the generator executable, libclang and
  dependent Qt libraries)
- PySide2 (the PySide2 modules and Qt shared libraries, and tools like
  rcc, uic)

Calling the setup.py script will not do the actual build now (in the
sense of calling CMake, make, etc.). Instead it will spawn new
processes (via subprocess.call) calling the same setup.py script,
but with different arguments. These "sub-invocations" will do the
actual building. Thus, the "top-level invocation" will decide which
packages to build and delegate that to the "sub-invocations" of
setup.py.

A new optional command line argument is introduced called
"--build-type" which defaults to "all", and can also be set to
"shiboken2", "shiboken2-generator" and "pyside2". A user can choose
which packages to build using this option. The "top-level invocation"
uses this option to decide how many "sub-invocations" to execute.

A new command line argument called "--internal-build-type"
takes the same values as the one above. It defines which package
will actually be built in the new spawned "sub-invocation" process.
The "top-level invocation" sets this automatically for each
"sub-invocation" depending on the value of "--build-type".

This option is also useful for developers that may want to debug the
python building code in the "sub-invocation".
Developers can set this manually via the command line, and
thus avoid the process spawning indirection.

A new class Config is introduced to facilitate storage of
the various state needed for building a single package.

A new class SetupRunner is introduced that takes care of the
"--build-type" and "--internal-build-type" argument handling
and delegation of "sub-invocations".

A new class Options is introduced to 'hopefully', in the future, streamline
the mess of option handling that we currently have.

setup.py now is now simplified to mostly just call
SetupRunner.run_setup().

Certain refactorings were done to facilitate further clean-up of the
build code, the current code is definitely not the end all be all.

Various other changes that were needed to implement the wheel
separation:
- a new cmake_helpers directory is added to share common cmake
  code between packages.
- the custom popenasync.py file is removed in favor of using
  subprocess.call in as many places as possible, and thus
  avoid 10 different functions for process creation.
- Manifest.in is removed, because copying to the setuptools
  build dir is now done directly by prepare_packages functions.
- because prepare_packages copies directly to the setuptools
  build dir, avoiding the pyside_package dir, we do less copying
  of big Qt files now.
- versioning of PySide2 and shiboken2 packages is now separate.
  shiboken2 and shiboken2-generator share the same versions for
  now though.
- shiboken2 is now listed as a required package for PySide2, to
  facilitate pip requirements.txt dependencies.
- coin_build_instructions currently needs to install an unreleased
  version of wheel, due to a bug that breaks installation of
  generated wheel files.
- added separate command line options to pyside2_config.py for
  shiboken2-module and shiboken2-generator.
- adapted samplebinding and scriptableapplication projects due to
  shiboken being a separate package.
- adapted pyside2-tool and shiboken2-tool python scripts for setup
  tools entry points.
- made some optimizations not to invoke cmake for shiboken2-generator
  when doing a top-level "all" build.
- fixed unnecessary rpaths not to be included on Linux (mainly the
  Qt rpaths).

Task-nubmer: PYSIDE-749
Change-Id: I0336043955624c1d12ed254802c442608cced5fb
Reviewed-by: Christian Tismer <tismer@stackless.com>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>

1 files changed, 165 insertions, 0 deletions

diff --git a/build_scripts/setup_runner.py b/build_scripts/setup_runner.py
new file mode 100644
index 000000000..a1526793e
--- /dev/null
+++ b/build_scripts/setup_runner.py
@@ -0,0 +1,165 @@
+#############################################################################
+##
+## Copyright (C) 2018 The Qt Company Ltd.
+## Contact: https://www.qt.io/licensing/
+##
+## This file is part of Qt for Python.
+##
+## $QT_BEGIN_LICENSE:LGPL$
+## Commercial License Usage
+## Licensees holding valid commercial Qt licenses may use this file in
+## accordance with the commercial license agreement provided with the
+## Software or, alternatively, in accordance with the terms contained in
+## a written agreement between you and The Qt Company. For licensing terms
+## and conditions see https://www.qt.io/terms-conditions. For further
+## information use the contact form at https://www.qt.io/contact-us.
+##
+## GNU Lesser General Public License Usage
+## Alternatively, this file may be used under the terms of the GNU Lesser
+## General Public License version 3 as published by the Free Software
+## Foundation and appearing in the file LICENSE.LGPL3 included in the
+## packaging of this file. Please review the following information to
+## ensure the GNU Lesser General Public License version 3 requirements
+## will be met: https://www.gnu.org/licenses/lgpl-3.0.html.
+##
+## GNU General Public License Usage
+## Alternatively, this file may be used under the terms of the GNU
+## General Public License version 2.0 or (at your option) the GNU General
+## Public license version 3 or any later version approved by the KDE Free
+## Qt Foundation. The licenses are as published by the Free Software
+## Foundation and appearing in the file LICENSE.GPL2 and LICENSE.GPL3
+## included in the packaging of this file. Please review the following
+## information to ensure the GNU General Public License requirements will
+## be met: https://www.gnu.org/licenses/gpl-2.0.html and
+## https://www.gnu.org/licenses/gpl-3.0.html.
+##
+## $QT_END_LICENSE$
+##
+#############################################################################
+
+import sys, os, textwrap
+
+from build_scripts.config import config
+from build_scripts.main import get_package_version, get_setuptools_extension_modules
+from build_scripts.main import cmd_class_dict
+from build_scripts.options import OPTION_BUILD_TYPE, OPTION_INTERNAL_BUILD_TYPE
+from build_scripts.utils import run_process
+
+from setuptools import setup
+
+
+class SetupRunner(object):
+    def __init__(self, orig_argv):
+        self.invocations_list = []
+
+        # Keep the original args around in case we ever need to pass
+        # modified arguments to the sub invocations.
+        self.orig_argv = orig_argv
+        self.sub_argv = list(orig_argv)
+
+        self.setup_script_dir = os.getcwd()
+
+    @staticmethod
+    def cmd_line_argument_is_in_args(argument, args):
+        """ Check if command line argument was passed in args. """
+        return any(arg for arg in list(args) if "--" + argument in arg)
+
+    @staticmethod
+    def remove_cmd_line_argument_in_args(argument, args):
+        """ Remove command line argument from args. """
+        return [arg for arg in list(args) if "--" + argument not in arg]
+
+    @staticmethod
+    def construct_cmd_line_argument(name, value=None):
+        """ Constructs a command line argument given name and value. """
+        if not value:
+            return "--{}".format(name)
+        return "--{}={}".format(name, value)
+
+    @staticmethod
+    def construct_internal_build_type_cmd_line_argument(internal_build_type):
+        return SetupRunner.construct_cmd_line_argument("internal-build-type", internal_build_type)
+
+    def add_setup_internal_invocation(self, build_type, reuse_build=False):
+        """ Enqueues a script sub-invocation to be executed later. """
+        internal_build_type_arg = self.construct_internal_build_type_cmd_line_argument(build_type)
+        setup_cmd = [sys.executable] + self.sub_argv + [internal_build_type_arg]
+
+        # Add --reuse-build option if requested and not already present.
+        if reuse_build and not self.cmd_line_argument_is_in_args("reuse-build", self.sub_argv):
+            setup_cmd.append(self.construct_cmd_line_argument("reuse-build"))
+        self.invocations_list.append(setup_cmd)
+
+    def run_setup(self):
+        """
+        Decide what kind of build is requested and then execute it.
+        In the top-level invocation case, the script
+            will spawn setup.py again (possibly multiple times).
+        In the internal invocation case, the script
+            will run setuptools.setup().
+        """
+
+        # Prepare initial config.
+        config.init_config(build_type=OPTION_BUILD_TYPE,
+                           internal_build_type=OPTION_INTERNAL_BUILD_TYPE,
+                           cmd_class_dict=cmd_class_dict,
+                           package_version=get_package_version(),
+                           ext_modules=get_setuptools_extension_modules(),
+                           setup_script_dir=self.setup_script_dir)
+
+        # This is an internal invocation of setup.py, so start actual
+        # build.
+        if config.is_internal_invocation():
+            if config.internal_build_type not in config.get_allowed_internal_build_values():
+                raise RuntimeError("Invalid '{}' option given to --internal-build-type. "
+                                   .format(config.internal_build_type))
+            self.run_setuptools_setup()
+            return
+
+        # This is a top-level invocation of setup.py, so figure out what
+        # modules we will build and depending on that, call setup.py
+        # multiple times with different arguments.
+        if config.build_type not in config.get_allowed_top_level_build_values():
+            raise RuntimeError("Invalid '{}' option given to --build-type. "
+                               .format(config.build_type))
+
+        # Build everything: shiboken2, shiboken2-generator and PySide2.
+        if config.is_top_level_build_all():
+            self.add_setup_internal_invocation(config.shiboken_module_option_name)
+
+            # Reuse the shiboken build for the generator package instead
+            # of rebuilding it again.
+            self.add_setup_internal_invocation(config.shiboken_generator_option_name,
+                                               reuse_build=True)
+
+            self.add_setup_internal_invocation(config.pyside_option_name)
+
+        elif config.is_top_level_build_shiboken_module():
+            self.add_setup_internal_invocation(config.shiboken_module_option_name)
+
+        elif config.is_top_level_build_shiboken_generator():
+            self.add_setup_internal_invocation(config.shiboken_generator_option_name)
+
+        elif config.is_top_level_build_pyside():
+            self.add_setup_internal_invocation(config.pyside_option_name)
+
+        for cmd in self.invocations_list:
+            cmd_as_string = " ".join(cmd)
+            print("\nRunning process: {}\n".format(cmd_as_string))
+            exit_code = run_process(cmd)
+            if exit_code != 0:
+                msg = textwrap.dedent("""
+                    setup.py invocation failed with exit code: {}.\n\n
+                    setup.py invocation was: {}
+                    """).format(exit_code, cmd_as_string)
+                raise RuntimeError(msg)
+
+    @staticmethod
+    def run_setuptools_setup():
+        """
+        Runs setuptools.setup() once in a single setup.py
+        sub-invocation.
+        """
+
+        kwargs = config.setup_kwargs
+        setup(**kwargs)