setup.py: Add support for cross-building

setup.py can now be used to cross-compile PySide to a target Linux distribution from a Linux host. For example you could cross-compile PySide targeting an arm64 Raspberry Pi4 sysroot on an Ubuntu x86_64 host machine. Cross-compiling PySide has a few requirements: - a sysroot to cross-compile against, with a pre-installed Qt, Python interpreter, library and development packages (which provides C++ headers) - a host Qt installation of the same version that is in the target sysroot - a host Python installation, preferably of the same version as the target one (to run setup.py) - a working cross-compiling toolchain (cross-compiler, linker, etc) - a custom written CMake toolchain file - CMake version 3.17+ - Qt version 6.3+ The CMake toolchain file is required to set up all the relevant cross-compilation information: where the sysroot is, where the toolchain is, the compiler name, compiler flags, etc. Once are requirements are met, to cross-compile one has to specify a few additional options when calling setup.py: the path to the cmake toolchain file, the path to the host Qt installation and the target python platform name. An example setup.py invocation to build a wheel for an armv7 machine might look like the following: python setup.py bdist_wheel --parallel=8 --ignore-git --reuse-build --cmake-toolchain-file=$PWD/rpi/toolchain_armv7.cmake --qt-host-path=/opt/Qt/6.3.0/gcc_64 --plat-name=linux_armv7l --limited-api=yes --standalone Sample platform names that can be used are: linux_armv7, linux_aarch64. If the auto-detection code fails to find the target Python or Qt installation, one can specify their location by providing the --python-target-path=<path> and --qt-target-path=<path> options to setup.py. If the automatic build of the host shiboken code generator fails, one can specify the path to a custom built host shiboken via the --shiboken-host-path option. Documentation about the build process and a sample CMake toolchain file will be added in a separate change. Implementation details. Internally, setup.py will build a host shiboken executable using the provided host Qt path, and then use it for the cross-build. This is achieved via an extra setup.py sub-invocation with some heuristics on which options should be passed to the sub-invocation. The host shiboken is not included in the target wheels. Introspection of where the host / target Qt and Python are located is done via CMake compile tests, because we can't query information from a qmake that is built for a different architecture / platform. When limited API is enabled, we modify the wheel name to contain the manylinux2014 tag, despite the wheel not fully qualifying for that tag. When copying the Qt libraries / plugins from the target sysroot in a standalone build, we need to adjust all their rpaths to match the destination directory layout of the wheel. Fixes: PYSIDE-802 Task-number: PYSIDE-1033 Change-Id: I6e8c51ef5127d85949de650396d615ca95194db0 Reviewed-by: Cristian Maureira-Fredes <cristian.maureira-fredes@qt.io> Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
author: Alexandru Croitor <alexandru.croitor@qt.io> 2021-09-29 19:01:51 +0200
committer: Alexandru Croitor <alexandru.croitor@qt.io> 2022-02-04 15:51:04 +0100
commit: 57866a57586d401c784f809f9f7994b0e4623706 (patch)
tree: 48b9d5a7d1a03d1edd15359858adcefd18430467 /build_scripts/options.py
parent: 14e4527cc427ce8c5e7c1758a95a1bbce0498471 (diff)
1 files changed, 183 insertions, 34 deletions
diff --git a/build_scripts/options.py b/build_scripts/options.py
index 045eb05d0..6ce53b982 100644
--- a/build_scripts/options.py
+++ b/build_scripts/options.py
@@ -39,11 +39,13 @@
 
 try:
     from setuptools._distutils import log
+    from setuptools import Command
 except ModuleNotFoundError:
     # This is motivated by our CI using an old version of setuptools
     # so then the coin_build_instructions.py script is executed, and
     # import from this file, it was failing.
     from distutils import log
+    from distutils.cmd import Command
 from shutil import which
 import sys
 import os
@@ -51,6 +53,7 @@ import warnings
 from pathlib import Path
 
 from .qtinfo import QtInfo
+from .utils import memoize
 
 
 _AVAILABLE_MKSPECS = ["ninja", "msvc", "mingw"] if sys.platform == "win32" else ["ninja", "make"]
@@ -63,6 +66,9 @@ Additional options:
   ---macos-use-libc++                  Use libc++ on macOS
   --snapshot-build                     Snapshot build
   --package-timestamp                  Package Timestamp
+  --cmake-toolchain-file               Path to CMake toolchain to enable cross-compiling
+  --shiboken-host-path                 Path to host shiboken package when cross-compiling
+  --qt-host-path                       Path to host Qt installation when cross-compiling
 """
 
 
@@ -164,7 +170,7 @@ def _jobs_option_value():
 
 
 # Declare options which need to be known when instantiating the DistUtils
-# commands.
+# commands or even earlier during SetupRunner.run().
 OPTION = {
     "BUILD_TYPE": option_value("build-type"),
     "INTERNAL_BUILD_TYPE": option_value("internal-build-type"),
@@ -179,7 +185,11 @@ OPTION = {
     "PACKAGE_TIMESTAMP": option_value("package-timestamp"),
     # This is used automatically by distutils.command.install object, to
     # specify the final installation location.
-    "FINAL_INSTALL_PREFIX": option_value("prefix", remove=False)
+    "FINAL_INSTALL_PREFIX": option_value("prefix", remove=False),
+    "CMAKE_TOOLCHAIN_FILE": option_value("cmake-toolchain-file"),
+    "SHIBOKEN_HOST_PATH": option_value("shiboken-host-path"),
+    "SHIBOKEN_HOST_PATH_QUERY_FILE": option_value("internal-shiboken-host-path-query-file"),
+    "QT_HOST_PATH": option_value("qt-host-path")
     # This is used to identify the template for doc builds
 }
 _deprecated_option_jobs = option_value('jobs')
@@ -191,7 +201,7 @@ if _deprecated_option_jobs:
 class DistUtilsCommandMixin(object):
     """Mixin for the DistUtils build/install commands handling the options."""
 
-    _finalized = False
+    _static_class_finalized_once = False
 
     mixin_user_options = [
         ('avoid-protected-hack', None, 'Force --avoid-protected-hack'),
@@ -217,9 +227,16 @@ class DistUtilsCommandMixin(object):
         ('qtpaths=', None, 'Path to qtpaths'),
         ('qmake=', None, 'Path to qmake (deprecated, use qtpaths)'),
         ('qt=', None, 'Qt version'),
+        ('qt-target-path=', None,
+         'Path to device Qt installation (use Qt libs when cross-compiling)'),
         ('cmake=', None, 'Path to CMake'),
         ('openssl=', None, 'Path to OpenSSL libraries'),
+
+        # FIXME: Deprecated in favor of shiboken-target-path
         ('shiboken-config-dir=', None, 'shiboken configuration directory'),
+
+        ('shiboken-target-path=', None, 'Path to target shiboken package'),
+        ('python-target-path=', None, 'Path to target Python installation / prefix'),
         ('make-spec=', None, 'Qt make-spec'),
         ('macos-arch=', None, 'macOS architecture'),
         ('macos-sysroot=', None, 'macOS sysroot'),
@@ -230,7 +247,13 @@ class DistUtilsCommandMixin(object):
         ('qt-conf-prefix=', None, 'Qt configuration prefix'),
         ('qt-src-dir=', None, 'Qt source directory'),
         ('no-qt-tools', None, 'Do not copy the Qt tools'),
-        ('pyside-numpy-support', None, 'libpyside: Add (experimental) numpy support')
+        ('pyside-numpy-support', None, 'libpyside: Add (experimental) numpy support'),
+        ('internal-cmake-install-dir-query-file-path=', None,
+         'Path to file where the CMake install path of the project will be saved'),
+
+        # We redeclare plat-name as an option so it's recognized by the
+        # install command and doesn't throw an error.
+        ('plat-name=', None, 'The platform name for which we are cross-compiling'),
         ]
 
     def __init__(self):
@@ -259,9 +282,17 @@ class DistUtilsCommandMixin(object):
         self.qmake = None
         self.has_qmake_option = False
         self.qt = '5'
+        self.qt_host_path = None
+        self.qt_target_path = None
         self.cmake = None
         self.openssl = None
         self.shiboken_config_dir = None
+        self.shiboken_host_path = None
+        self.shiboken_host_path_query_file = None
+        self.shiboken_target_path = None
+        self.python_target_path = None
+        self.is_cross_compile = False
+        self.cmake_toolchain_file = None
         self.make_spec = None
         self.macos_arch = None
         self.macos_sysroot = None
@@ -273,16 +304,62 @@ class DistUtilsCommandMixin(object):
         self.qt_src_dir = None
         self.no_qt_tools = False
         self.pyside_numpy_support = False
+        self.plat_name = None
+        self.internal_cmake_install_dir_query_file_path = None
+        self._per_command_mixin_options_finalized = False
+
+        # When initializing a command other than the main one (so the
+        # first one), we need to copy the user options from the main
+        # command to the new command options dict. Then
+        # Distribution.get_command_obj will pick up the copied options
+        # ensuring that all commands that inherit from
+        # the mixin, get our custom properties set by the time
+        # finalize_options is called.
+        if DistUtilsCommandMixin._static_class_finalized_once:
+            current_command: Command = self
+            dist = current_command.distribution
+            main_command_name = dist.commands[0]
+            main_command_opts = dist.get_option_dict(main_command_name)
+            current_command_name = current_command.get_command_name()
+            current_command_opts = dist.get_option_dict(current_command_name)
+            mixin_options_set = self.get_mixin_options_set()
+            for key, value in main_command_opts.items():
+                if key not in current_command_opts and key in mixin_options_set:
+                    current_command_opts[key] = value
+
+    @staticmethod
+    @memoize
+    def get_mixin_options_set():
+        keys = set()
+        for (name, _, _) in DistUtilsCommandMixin.mixin_user_options:
+            keys.add(name.rstrip("=").replace("-", "_"))
+        return keys
+
 
     def mixin_finalize_options(self):
-        # Bail out on 2nd call to mixin_finalize_options() since that is the
-        # build command following the install command when invoking
-        # setup.py install
-        if not DistUtilsCommandMixin._finalized:
-            DistUtilsCommandMixin._finalized = True
+        # The very first we finalize options, record that.
+        if not DistUtilsCommandMixin._static_class_finalized_once:
+            DistUtilsCommandMixin._static_class_finalized_once = True
+
+        # Ensure we finalize once per command object, rather than per
+        # setup.py invocation. We want to have the option values
+        # available in all commands that derive from the mixin.
+        if not self._per_command_mixin_options_finalized:
+            self._per_command_mixin_options_finalized = True
             self._do_finalize()
 
     def _do_finalize(self):
+        # is_cross_compile must be set before checking for qtpaths/qmake
+        # because we DON'T want those to be found when cross compiling.
+        # Currently when cross compiling, qt-target-path MUST be used.
+        using_cmake_toolchain_file = False
+        cmake_toolchain_file = None
+        if OPTION["CMAKE_TOOLCHAIN_FILE"]:
+            self.is_cross_compile = True
+            using_cmake_toolchain_file = True
+            cmake_toolchain_file = OPTION["CMAKE_TOOLCHAIN_FILE"]
+            self.cmake_toolchain_file = cmake_toolchain_file
+
         if not self._determine_defaults_and_check():
             sys.exit(-1)
         OPTION['AVOID_PROTECTED_HACK'] = self.avoid_protected_hack
@@ -320,12 +397,62 @@ class DistUtilsCommandMixin(object):
             OPTION['QMAKE'] = qmake_abs_path
         OPTION['HAS_QMAKE_OPTION'] = self.has_qmake_option
         OPTION['QT_VERSION'] = self.qt
+        self.qt_host_path = OPTION['QT_HOST_PATH']
+        OPTION['QT_TARGET_PATH'] = self.qt_target_path
+
+        qt_target_path = None
+        if self.qt_target_path:
+            qt_target_path = self.qt_target_path
+
+        # We use the CMake project to find host Qt if neither qmake or
+        # qtpaths is available. This happens when building the host
+        # tools in the overall cross-building process.
+        use_cmake = False
+        if using_cmake_toolchain_file or \
+                (not self.qmake and not self.qtpaths and self.qt_target_path):
+            use_cmake = True
+
         QtInfo().setup(qtpaths_abs_path, self.cmake, qmake_abs_path,
-                       self.has_qmake_option)
+                       self.has_qmake_option,
+                       use_cmake=use_cmake,
+                       qt_target_path=qt_target_path,
+                       cmake_toolchain_file=cmake_toolchain_file)
+
+        try:
+            QtInfo().prefix_dir
+        except Exception as e:
+            if not self.qt_target_path:
+                log.error(
+                    "\nCould not find Qt. You can pass the --qt-target-path=<qt-dir> option as a "
+                    "hint where to find Qt. Error was:\n\n\n")
+            else:
+                log.error(
+                    f"\nCould not find Qt via provided option --qt-target-path={qt_target_path} "
+                    "Error was:\n\n\n")
+            raise e
 
         OPTION['CMAKE'] = os.path.abspath(self.cmake)
         OPTION['OPENSSL'] = self.openssl
         OPTION['SHIBOKEN_CONFIG_DIR'] = self.shiboken_config_dir
+        if self.shiboken_config_dir:
+            _warn_deprecated_option('shiboken-config-dir', 'shiboken-target-path')
+
+        self.shiboken_host_path = OPTION['SHIBOKEN_HOST_PATH']
+        self.shiboken_host_path_query_file = OPTION['SHIBOKEN_HOST_PATH_QUERY_FILE']
+
+        if not self.shiboken_host_path and self.shiboken_host_path_query_file:
+            try:
+                queried_shiboken_host_path = Path(self.shiboken_host_path_query_file).read_text()
+                self.shiboken_host_path = queried_shiboken_host_path
+                OPTION['SHIBOKEN_HOST_PATH'] = queried_shiboken_host_path
+            except Exception as e:
+                log.error(
+                    f"\n Could not find shiboken host tools via the query file: "
+                    f"{self.shiboken_host_path_query_file:} Error was:\n\n\n")
+                raise e
+
+        OPTION['SHIBOKEN_TARGET_PATH'] = self.shiboken_target_path
+        OPTION['PYTHON_TARGET_PATH'] = self.python_target_path
         OPTION['MAKESPEC'] = self.make_spec
         OPTION['MACOS_ARCH'] = self.macos_arch
         OPTION['MACOS_SYSROOT'] = self.macos_sysroot
@@ -338,6 +465,15 @@ class DistUtilsCommandMixin(object):
         OPTION['NO_QT_TOOLS'] = self.no_qt_tools
         OPTION['PYSIDE_NUMPY_SUPPORT'] = self.pyside_numpy_support
 
+        if not self._extra_checks():
+            sys.exit(-1)
+
+    def _extra_checks(self):
+        if self.is_cross_compile and not self.plat_name:
+            log.error(f"No value provided to --plat-name while cross-compiling.")
+            return False
+        return True
+
     def _find_qtpaths_in_path(self):
         if not self.qtpaths:
             self.qtpaths = which("qtpaths")
@@ -354,30 +490,43 @@ class DistUtilsCommandMixin(object):
             log.error(f"'{self.cmake}' does not exist.")
             return False
 
-        # Enforce usage of qmake in QtInfo if it was given explicitly.
-        if self.qmake:
-            self.has_qmake_option = True
-            _warn_deprecated_option('qmake', 'qtpaths')
-
-        # If no option was given explicitly, prefer to find qtpaths
-        # in PATH.
-        if not self.qmake and not self.qtpaths:
-            self._find_qtpaths_in_path()
-
-        # If no tool was specified and qtpaths was not found in PATH,
-        # ask to provide a path to qtpaths.
-        if not self.qtpaths and not self.qmake:
-            log.error("No value provided to --qtpaths option. Please provide one to find Qt.")
-            return False
-
-        # Validate that the given tool path exists.
-        if self.qtpaths and not os.path.exists(self.qtpaths):
-            log.error(f"The specified qtpaths path '{self.qtpaths}' does not exist.")
-            return False
-
-        if self.qmake and not os.path.exists(self.qmake):
-            log.error(f"The specified qmake path '{self.qmake}' does not exist.")
-            return False
+        # When cross-compiling, we only accept the qt-target-path
+        # option and don't rely on auto-searching in PATH or the other
+        # qtpaths / qmake options.
+        # We also don't do auto-searching if qt-target-path is passed
+        # explicitly. This is to help with the building of host tools
+        # while cross-compiling.
+        if not self.is_cross_compile and not self.qt_target_path:
+            # Enforce usage of qmake in QtInfo if it was given explicitly.
+            if self.qmake:
+                self.has_qmake_option = True
+                _warn_deprecated_option('qmake', 'qtpaths')
+
+            # If no option was given explicitly, prefer to find qtpaths
+            # in PATH.
+            if not self.qmake and not self.qtpaths:
+                self._find_qtpaths_in_path()
+
+            # If no tool was specified and qtpaths was not found in PATH,
+            # ask to provide a path to qtpaths.
+            if not self.qtpaths and not self.qmake and not self.qt_target_path:
+                log.error("No value provided to --qtpaths option. Please provide one to find Qt.")
+                return False
+
+            # Validate that the given tool path exists.
+            if self.qtpaths and not os.path.exists(self.qtpaths):
+                log.error(f"The specified qtpaths path '{self.qtpaths}' does not exist.")
+                return False
+
+            if self.qmake and not os.path.exists(self.qmake):
+                log.error(f"The specified qmake path '{self.qmake}' does not exist.")
+                return False
+        else:
+            # Check for existence, but don't require if it's not set. A
+            # check later will be done to see if it's needed.
+            if self.qt_target_path and not os.path.exists(self.qt_target_path):
+                log.error(f"Provided --qt-target-path='{self.qt_target_path}' path does not exist.")
+                return False
 
         if not self.make_spec:
             self.make_spec = _AVAILABLE_MKSPECS[0]
author	Alexandru Croitor <alexandru.croitor@qt.io>	2021-09-29 19:01:51 +0200
committer	Alexandru Croitor <alexandru.croitor@qt.io>	2022-02-04 15:51:04 +0100
commit	57866a57586d401c784f809f9f7994b0e4623706 (patch)
tree	48b9d5a7d1a03d1edd15359858adcefd18430467 /build_scripts/options.py
parent	14e4527cc427ce8c5e7c1758a95a1bbce0498471 (diff)