Allow building shiboken2 and PySide2 as separate wheels

Actually this creates 3 wheel packages:
- shiboken2 (the python module and libshiboken shared library)
- shiboken2-generator (contains the generator executable, libclang and
  dependent Qt libraries)
- PySide2 (the PySide2 modules and Qt shared libraries, and tools like
  rcc, uic)

Calling the setup.py script will not do the actual build now (in the
sense of calling CMake, make, etc.). Instead it will spawn new
processes (via subprocess.call) calling the same setup.py script,
but with different arguments. These "sub-invocations" will do the
actual building. Thus, the "top-level invocation" will decide which
packages to build and delegate that to the "sub-invocations" of
setup.py.

A new optional command line argument is introduced called
"--build-type" which defaults to "all", and can also be set to
"shiboken2", "shiboken2-generator" and "pyside2". A user can choose
which packages to build using this option. The "top-level invocation"
uses this option to decide how many "sub-invocations" to execute.

A new command line argument called "--internal-build-type"
takes the same values as the one above. It defines which package
will actually be built in the new spawned "sub-invocation" process.
The "top-level invocation" sets this automatically for each
"sub-invocation" depending on the value of "--build-type".

This option is also useful for developers that may want to debug the
python building code in the "sub-invocation".
Developers can set this manually via the command line, and
thus avoid the process spawning indirection.

A new class Config is introduced to facilitate storage of
the various state needed for building a single package.

A new class SetupRunner is introduced that takes care of the
"--build-type" and "--internal-build-type" argument handling
and delegation of "sub-invocations".

A new class Options is introduced to 'hopefully', in the future, streamline
the mess of option handling that we currently have.

setup.py now is now simplified to mostly just call
SetupRunner.run_setup().

Certain refactorings were done to facilitate further clean-up of the
build code, the current code is definitely not the end all be all.

Various other changes that were needed to implement the wheel
separation:
- a new cmake_helpers directory is added to share common cmake
  code between packages.
- the custom popenasync.py file is removed in favor of using
  subprocess.call in as many places as possible, and thus
  avoid 10 different functions for process creation.
- Manifest.in is removed, because copying to the setuptools
  build dir is now done directly by prepare_packages functions.
- because prepare_packages copies directly to the setuptools
  build dir, avoiding the pyside_package dir, we do less copying
  of big Qt files now.
- versioning of PySide2 and shiboken2 packages is now separate.
  shiboken2 and shiboken2-generator share the same versions for
  now though.
- shiboken2 is now listed as a required package for PySide2, to
  facilitate pip requirements.txt dependencies.
- coin_build_instructions currently needs to install an unreleased
  version of wheel, due to a bug that breaks installation of
  generated wheel files.
- added separate command line options to pyside2_config.py for
  shiboken2-module and shiboken2-generator.
- adapted samplebinding and scriptableapplication projects due to
  shiboken being a separate package.
- adapted pyside2-tool and shiboken2-tool python scripts for setup
  tools entry points.
- made some optimizations not to invoke cmake for shiboken2-generator
  when doing a top-level "all" build.
- fixed unnecessary rpaths not to be included on Linux (mainly the
  Qt rpaths).

Task-nubmer: PYSIDE-749
Change-Id: I0336043955624c1d12ed254802c442608cced5fb
Reviewed-by: Christian Tismer <tismer@stackless.com>
Reviewed-by: Qt CI Bot <qt_ci_bot@qt-project.org>

1 files changed, 72 insertions, 47 deletions

diff --git a/build_scripts/platforms/macos.py b/build_scripts/platforms/macos.py
index 936f4ca90..49f02754d 100644
--- a/build_scripts/platforms/macos.py
+++ b/build_scripts/platforms/macos.py
@@ -37,12 +37,29 @@
 ##
 #############################################################################
 
-import fnmatch, os
+import fnmatch
+import os
 from ..utils import copydir, copyfile, macos_fix_rpaths_for_library
+from ..config import config
 
-def prepare_standalone_package_macos(self, executables, vars):
+
+def prepare_standalone_package_macos(self, vars):
     built_modules = vars['built_modules']
 
+    constrain_modules = None
+    copy_plugins = True
+    copy_qml = True
+    copy_translations = True
+    copy_qt_conf = True
+
+    if config.is_internal_shiboken_generator_build():
+        constrain_modules = ["Core", "Network", "Xml", "XmlPatterns"]
+        constrain_frameworks = ['Qt' + name + '.framework' for name in constrain_modules]
+        copy_plugins = False
+        copy_qml = False
+        copy_translations = False
+        copy_qt_conf = False
+
     # Directory filter for skipping unnecessary files.
     def general_dir_filter(dir_name, parent_full_path, dir_full_path):
         if fnmatch.fnmatch(dir_name, "*.dSYM"):
@@ -52,6 +69,7 @@ def prepare_standalone_package_macos(self, executables, vars):
     # Filter out debug plugins and qml plugins in the
     # debug_and_release config.
     no_copy_debug = True
+
     def file_variant_filter(file_name, file_full_path):
         if self.qtinfo.build_type != 'debug_and_release':
             return True
@@ -59,17 +77,16 @@ def prepare_standalone_package_macos(self, executables, vars):
             return False
         return True
 
-    # <qt>/lib/* -> <setup>/PySide2/Qt/lib
+    # <qt>/lib/* -> <setup>/{st_package_name}/Qt/lib
     if self.qt_is_framework_build():
-        framework_built_modules = [
-            'Qt' + name + '.framework' for name in built_modules]
-
-        def framework_dir_filter(dir_name, parent_full_path,
-                dir_full_path):
+        def framework_dir_filter(dir_name, parent_full_path, dir_full_path):
             if '.framework' in dir_name:
-                if dir_name.startswith('QtWebEngine') and not \
-                            self.is_webengine_built(built_modules):
+                if (dir_name.startswith('QtWebEngine') and
+                        not self.is_webengine_built(built_modules)):
+                    return False
+                if constrain_modules and dir_name not in constrain_frameworks:
                     return False
+
             if dir_name in ['Headers', 'fonts']:
                 return False
             if dir_full_path.endswith('Versions/Current'):
@@ -84,6 +101,7 @@ def prepare_standalone_package_macos(self, executables, vars):
         # Filter out debug frameworks in the
         # debug_and_release config.
         no_copy_debug = True
+
         def framework_variant_filter(file_name, file_full_path):
             if self.qtinfo.build_type != 'debug_and_release':
                 return True
@@ -93,7 +111,7 @@ def prepare_standalone_package_macos(self, executables, vars):
                 return False
             return True
 
-        copydir("{qt_lib_dir}", "{pyside_package_dir}/PySide2/Qt/lib",
+        copydir("{qt_lib_dir}", "{st_build_dir}/{st_package_name}/Qt/lib",
             recursive=True, vars=vars,
             ignore=["*.la", "*.a", "*.cmake", "*.pc", "*.prl"],
             dir_filter_function=framework_dir_filter,
@@ -104,7 +122,7 @@ def prepare_standalone_package_macos(self, executables, vars):
         # from Versions/5/Helpers, thus adding two more levels of
         # directory hierarchy.
         if self.is_webengine_built(built_modules):
-            qt_lib_path = "{pyside_package_dir}/PySide2/Qt/lib".format(
+            qt_lib_path = "{st_build_dir}/{st_package_name}/Qt/lib".format(
                 **vars)
             bundle = "QtWebEngineCore.framework/Helpers/"
             bundle += "QtWebEngineProcess.app"
@@ -120,9 +138,11 @@ def prepare_standalone_package_macos(self, executables, vars):
         if 'WebKit' not in built_modules:
             ignored_modules.extend(['libQt5WebKit*.dylib'])
         accepted_modules = ['libQt5*.5.dylib']
+        if constrain_modules:
+            accepted_modules = ["libQt5" + module + "*.5.dylib" for module in constrain_modules]
 
         copydir("{qt_lib_dir}",
-            "{pyside_package_dir}/PySide2/Qt/lib",
+            "{st_build_dir}/{st_package_name}/Qt/lib",
             filter=accepted_modules,
             ignore=ignored_modules,
             file_filter_function=file_variant_filter,
@@ -130,53 +150,58 @@ def prepare_standalone_package_macos(self, executables, vars):
 
         if self.is_webengine_built(built_modules):
             copydir("{qt_lib_execs_dir}",
-                "{pyside_package_dir}/PySide2/Qt/libexec",
+                "{st_build_dir}/{st_package_name}/Qt/libexec",
                 filter=None,
                 recursive=False,
                 vars=vars)
 
             copydir("{qt_prefix_dir}/resources",
-                "{pyside_package_dir}/PySide2/Qt/resources",
+                "{st_build_dir}/{st_package_name}/Qt/resources",
                 filter=None,
                 recursive=False,
                 vars=vars)
 
             # Fix rpath for WebEngine process executable.
-            pyside_package_dir = vars['pyside_package_dir']
-            qt_libexec_path = "{}/PySide2/Qt/libexec".format(pyside_package_dir)
+            qt_libexec_path = "{st_build_dir}/{st_package_name}/Qt/libexec".format(**vars)
             binary = "QtWebEngineProcess"
             final_path = os.path.join(qt_libexec_path, binary)
             rpath = "@loader_path/../lib"
             macos_fix_rpaths_for_library(final_path, rpath)
 
-            # Copy the qt.conf file to libexec.
-            copyfile(
-                "{build_dir}/pyside2/PySide2/qt.conf",
-                "{pyside_package_dir}/PySide2/Qt/libexec",
-                vars=vars)
+            if copy_qt_conf:
+                # Copy the qt.conf file to libexec.
+                copyfile(
+                    "{build_dir}/pyside2/{st_package_name}/qt.conf",
+                    "{st_build_dir}/{st_package_name}/Qt/libexec",
+                    vars=vars)
+
+    if copy_plugins:
+        # <qt>/plugins/* -> <setup>/{st_package_name}/Qt/plugins
+        copydir("{qt_plugins_dir}",
+            "{st_build_dir}/{st_package_name}/Qt/plugins",
+            filter=["*.dylib"],
+            recursive=True,
+            dir_filter_function=general_dir_filter,
+            file_filter_function=file_variant_filter,
+            vars=vars)
+
 
-    # <qt>/plugins/* -> <setup>/PySide2/Qt/plugins
-    copydir("{qt_plugins_dir}",
-        "{pyside_package_dir}/PySide2/Qt/plugins",
-        filter=["*.dylib"],
-        recursive=True,
-        dir_filter_function=general_dir_filter,
-        file_filter_function=file_variant_filter,
-        vars=vars)
-
-    # <qt>/qml/* -> <setup>/PySide2/Qt/qml
-    copydir("{qt_qml_dir}",
-        "{pyside_package_dir}/PySide2/Qt/qml",
-        filter=None,
-        recursive=True,
-        force=False,
-        dir_filter_function=general_dir_filter,
-        file_filter_function=file_variant_filter,
-        vars=vars)
-
-    # <qt>/translations/* -> <setup>/PySide2/Qt/translations
-    copydir("{qt_translations_dir}",
-        "{pyside_package_dir}/PySide2/Qt/translations",
-        filter=["*.qm", "*.pak"],
-        force=False,
-        vars=vars)
+    if copy_qml:
+        # <qt>/qml/* -> <setup>/{st_package_name}/Qt/qml
+        copydir("{qt_qml_dir}",
+            "{st_build_dir}/{st_package_name}/Qt/qml",
+            filter=None,
+            recursive=True,
+            force=False,
+            dir_filter_function=general_dir_filter,
+            file_filter_function=file_variant_filter,
+            vars=vars)
+
+    if copy_translations:
+        # <qt>/translations/* ->
+        # <setup>/{st_package_name}/Qt/translations
+        copydir("{qt_translations_dir}",
+            "{st_build_dir}/{st_package_name}/Qt/translations",
+            filter=["*.qm", "*.pak"],
+            force=False,
+            vars=vars)