setup.py: Add support for cross-building

setup.py can now be used to cross-compile PySide to a target Linux
distribution from a Linux host.
For example you could cross-compile PySide targeting an arm64
Raspberry Pi4 sysroot on an Ubuntu x86_64 host machine.

Cross-compiling PySide has a few requirements:
 - a sysroot to cross-compile against, with a pre-installed Qt,
   Python interpreter, library and development packages (which
   provides C++ headers)
 - a host Qt installation of the same version that is in the target
   sysroot
 - a host Python installation, preferably of the same version as the
   target one (to run setup.py)
 - a working cross-compiling toolchain (cross-compiler, linker, etc)
 - a custom written CMake toolchain file
 - CMake version 3.17+
 - Qt version 6.3+

The CMake toolchain file is required to set up all the relevant
cross-compilation information: where the sysroot is, where the
toolchain is, the compiler name, compiler flags, etc.

Once are requirements are met, to cross-compile one has to specify a
few additional options when calling setup.py: the path to the cmake
toolchain file, the path to the host Qt installation
and the target python platform name.

An example setup.py invocation to build a wheel for an armv7 machine
might look like the following:

  python setup.py bdist_wheel --parallel=8 --ignore-git --reuse-build
  --cmake-toolchain-file=$PWD/rpi/toolchain_armv7.cmake
  --qt-host-path=/opt/Qt/6.3.0/gcc_64
  --plat-name=linux_armv7l
  --limited-api=yes
  --standalone

Sample platform names that can be used are: linux_armv7, linux_aarch64.

If the auto-detection code fails to find the target Python or Qt
installation, one can specify their location by providing the
  --python-target-path=<path>
and
  --qt-target-path=<path>
options to setup.py.

If the automatic build of the host shiboken code generator fails,
one can specify the path to a custom built host shiboken via the
--shiboken-host-path option.

Documentation about the build process and a sample CMake
toolchain file will be added in a separate change.


Implementation details.

Internally, setup.py will build a host shiboken executable using
the provided host Qt path, and then use it for the cross-build.
This is achieved via an extra setup.py sub-invocation with some
heuristics on which options should be passed to the sub-invocation.
The host shiboken is not included in the target wheels.

Introspection of where the host / target Qt and Python are located
is done via CMake compile tests, because we can't query information
from a qmake that is built for a different architecture / platform.

When limited API is enabled, we modify the wheel name to contain the
manylinux2014 tag, despite the wheel not fully qualifying for that
tag.

When copying the Qt libraries / plugins from the target sysroot in a
standalone build, we need to adjust all their rpaths to match the
destination directory layout of the wheel.

Fixes: PYSIDE-802
Task-number: PYSIDE-1033
Change-Id: I6e8c51ef5127d85949de650396d615ca95194db0
Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io>
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>

1 files changed, 75 insertions, 3 deletions

diff --git a/build_scripts/utils.py b/build_scripts/utils.py
index 39cff6743..2daad642e 100644
--- a/build_scripts/utils.py
+++ b/build_scripts/utils.py
@@ -48,6 +48,8 @@ import subprocess
 import fnmatch
 import itertools
 import glob
+import tempfile
+from collections import defaultdict
 
 import urllib.request as urllib
 
@@ -244,12 +246,16 @@ def init_msvc_env(platform_arch, build_type):
     log.info("Done initializing MSVC env")
 
 
-def platform_cmake_options():
+def platform_cmake_options(as_tuple_list=False):
     result = []
     if sys.platform == 'win32':
         # Prevent cmake from auto-detecting clang if it is in path.
-        result.append("-DCMAKE_C_COMPILER=cl.exe")
-        result.append("-DCMAKE_CXX_COMPILER=cl.exe")
+        if as_tuple_list:
+            result.append(("CMAKE_C_COMPILER", "cl.exe"))
+            result.append(("CMAKE_CXX_COMPILER", "cl.exe"))
+        else:
+            result.append("-DCMAKE_C_COMPILER=cl.exe")
+            result.append("-DCMAKE_CXX_COMPILER=cl.exe")
     return result
 
 
@@ -1250,3 +1256,69 @@ def parse_cmake_conf_assignments_by_key(source_dir):
             d[key] = value
     return d
 
+
+def configure_cmake_project(project_path,
+                            cmake_path,
+                            build_path=None,
+                            temp_prefix_build_path=None,
+                            cmake_args=None,
+                            cmake_cache_args=None,
+                            ):
+    clean_temp_dir = False
+    if not build_path:
+        # Ensure parent dir exists.
+        if temp_prefix_build_path:
+            os.makedirs(temp_prefix_build_path, exist_ok=True)
+
+        project_name = Path(project_path).name
+        build_path = tempfile.mkdtemp(prefix=f"{project_name}_", dir=temp_prefix_build_path)
+
+        if 'QFP_SETUP_KEEP_TEMP_FILES' not in os.environ:
+            clean_temp_dir = True
+
+    cmd = [cmake_path, '-G', 'Ninja', '-S', project_path, '-B', build_path]
+
+    if cmake_args:
+        cmd.extend(cmake_args)
+
+    for arg, value in cmake_cache_args:
+        cmd.extend([f'-D{arg}={value}'])
+
+    cmd_string = ' '.join(cmd)
+    # FIXME Python 3.7: Use subprocess.run()
+    proc = subprocess.Popen(cmd,
+                            stdout=subprocess.PIPE,
+                            stderr=subprocess.PIPE,
+                            shell=False,
+                            cwd=build_path,
+                            universal_newlines=True)
+    output, error = proc.communicate()
+    proc.wait()
+    return_code = proc.returncode
+
+    if return_code != 0:
+        raise RuntimeError(f"\nFailed to configure CMake project \n "
+                           f"'{project_path}' \n with error: \n {error}\n "
+                           f"Return code: {return_code}\n"
+                           f"Configure args were:\n  {cmd_string}")
+
+    if clean_temp_dir:
+        rmtree(build_path)
+
+    return output
+
+
+def parse_cmake_project_message_info(output):
+    # Parse the output for anything prefixed
+    # '-- qfp:<category>:<key>: <value>' as created by the message()
+    # calls in a given CMake project and store it in a python dict.
+    result = defaultdict(lambda: defaultdict(str))
+    pattern = re.compile(r"^-- qfp:(.+?):(.+?):(.*)$")
+    for line in output.splitlines():
+        found = pattern.search(line)
+        if found:
+            category = found.group(1).strip()
+            key = found.group(2).strip()
+            value = found.group(3).strip()
+            result[category][key] = str(value)
+    return result