aboutsummaryrefslogtreecommitdiffstats
path: root/packaging-tools
Commit message (Collapse)AuthorAgeFilesLines
* pylint: Update to version 2.17.3, apply required changesHEADmasterPatrik Teivonen2023-05-092-25/+34
| | | | | | | | Upgrade pylint package version in Pipfile to be compatible with 3.11. Fix or ignore checks that no longer pass as result of this change. Change-Id: I1b311a54d817e4f20879a32cf85de6e406817ce4 Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
* release_repo_updater.spawn_remote_background_task: exit with exit codePatrik Teivonen2023-05-021-1/+1
| | | | | | | | | | When logging script output with pipe, also return with the original exit code to avoid failing silently. The release_repo_updater.py script should not continue if this happens. Fixes: QTBUG-106022 Change-Id: I291ef04762680dd7bda7cb3f0c3addf6726fa6d4 Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
* Add generic functions to read files and configurations from SFTP remotePatrik Teivonen2023-05-025-93/+397
| | | | | | | | | | | | | -Move SFTP functionality sign_windows_installer -> read_remote_config -Remove old functions for getting the remote .ini from an URL -Add more generic functions to transfer files and read .ini configs -Add command line options for new functionality -Add unit tests for new functionality -Avoid writing decrypted private key to file system -Catch the new exception type PackagingError where relevant Change-Id: I4b2c4341c2fc1795847fa0c193bcbc47869c61ad Reviewed-by: Akseli Salovaara <akseli.salovaara@qt.io>
* sign_installer: Adjust logging verbosity level for codesignPatrik Teivonen2023-04-281-3/+3
| | | | | | | | Log the output from the codesign tool. Change verbosity level to 2 (3 made no difference). Change-Id: Id1f26adc2884a004a0e795bc8ff2a13eeae0988b Reviewed-by: Akseli Salovaara <akseli.salovaara@qt.io>
* update_component_translations.py: Fix regression, log skipped ts filesPatrik Teivonen2023-04-271-13/+10
| | | | | | | | | | | Fix regressions: -Fix appending correct translation <locale>.qm file name -Remove only </Package> or </Translations> tags from the line. Log it when locales are skipped (Korean, Chinese, untranslated). Change-Id: Ia2616c437e505908789dba1f3d5ff649c30bb92f Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* Pythonize update_component_translations.shPatrik Teivonen2023-04-253-126/+154
| | | | | | | | | The current script relied on realpath tool not available out of the box on macOS and Windows. Since we need to run repository builds on at least macOS in the future, it is better to have a Python script. Change-Id: I3cfe3789b16cb4635e18bd20e0beaace8fafff8a Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
* QtCreator: Sign on WindowsEike Ziller2023-04-172-3/+17
| | | | | | | | | | - make sign_windows_installer able to recurse directories - pass the signing command to the Qt Creator build script Task-number: QTCREATORBUG-25740 Task-number: QTCREATORBUG-28909 Change-Id: Idd2e96de36623833fc794cf5bee717b43b673efd Reviewed-by: Cristian Adam <cristian.adam@qt.io>
* sign_installer.py: Use 'file' for detecting Mach-O binariesv6.2.8-lts-packagingPatrik Teivonen2023-04-061-3/+2
| | | | | | | | | | Python's macholib library did not recognize all types of Mach-O headers Use 'file' utility to check for the file type instead. There is also 'python-magic' library but it relies on 'libmagic' which is not installed on macOS by default. Change-Id: Ib300590969415a65c187f096a7b6c80e9e8f8715 Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
* sign_installer: Use ditto when copying files for notarizationPatrik Teivonen2023-04-051-6/+4
| | | | | | | | | The signature seems to get lost for some items when using the copy command from shutil. Use ditto to preserve extended file attrs and signatures. Change-Id: Ie220675fba56f59270054467c964a118da42a8cb Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* sign_installer.py: Ignore symlinks in looking for signable contentPatrik Teivonen2023-04-051-1/+13
| | | | | | | Broken symlinks in content fail to resolve Change-Id: I271ce6e3127835ed0c29a06a9cc5e5ed8077efec Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* sign_installer: Adjust what counts as signable macOS payload contentPatrik Teivonen2023-04-051-7/+4
| | | | | | | | | | | | Remove unreliable file suffix checks, For example, ".so" files can contain non-native code, so they don't have to be always signed. Non-Mach-O executables and libraries seem to be treated as other data by macOS code signing and notarization systems so we should ignore those. Change-Id: Ib203a25ab7fd321e182033f4eb5e97b0656dddbd Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* sign_installer.py: Skip notarization if there is no payload to notarizePatrik Teivonen2023-04-051-0/+3
| | | | | Change-Id: Id3fe65e26c8abe14b7ff02f0010a8ff7b627bdd9 Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* Sign macOS installer disk images before notarizationPatrik Teivonen2023-04-032-30/+43
| | | | | | | | | | | | | | | release_repo_updater.py (offline installers): Previously, only the app bundle was signed prior notarization on macOS. Codesign also the outer container (in this case, the .dmg disk image) sign_installer.py (applies to online installers): Accept any file types for signing (not just .app bundles). Add CLI option to skip .dmg generation (--skip-dmg). If .dmg creation is enabled, also sign the output .dmg container. Task-number: QTBUG-112476 Change-Id: I26ef613d5eeb02da9115d6092631e107aec5784a Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* create_installer: Remove obsolete debug files patchingPatrik Teivonen2023-03-302-172/+1
| | | | | | | | | Debug files are already stripped and processed during an export. We don't need this functionality in packaging anymore. Change-Id: I7ce5aabdc7813ad01711ddf77e2aece946449d6d Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* release_repo_updater: Fix offline task buildPatrik Teivonen2023-03-301-1/+1
| | | | | Change-Id: I89f76221c7752e13313c6e4eb37d2e4a7bec717d Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
* release_task_reader: Improve task parsing logging outputv6.5.0-packagingPatrik Teivonen2023-03-296-67/+89
| | | | | | | | Allow parse_config to specify multiple task types e.g. IFW and QBSP. Then tasks excluded by the filters get printed out only once. Change-Id: I9b310227c1579425ae1edacdfb3809df2727058c Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
* Add option to sign and notarize installer payloadv6.5.0-rc1-packagingPatrik Teivonen2023-03-138-64/+303
| | | | | | | | | | | | | | | Offline installer/Online repo content should be notarized to avoid Gatekeeper issues on latest versions of macOS. Before compressing the final payload component archives, (re-)sign, notarize, and finally staple the relevant types of nested install content before distribution. Steps will be performed if '--notarize-payload' argument is passed onto create_installer.py script or when 'notarize_payload' is specified and set to yes|true|1 in the IFW task config. Change-Id: Icb5c482674af9421fa08e58f4e27043a6766e5c7 Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
* Ensure that libclang is built with -fPIC/PIEv6.4.3-packagingAlexey Edelev2023-03-101-0/+1
| | | | | Change-Id: I2f63521785625e39ac6ed65b9d09311de2cac168 Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
* create_installer.py: Move LRELEASE_TOOL configuration to argparsePatrik Teivonen2023-03-091-4/+10
| | | | | | | | Allow specifying download uri for the lrelease tool binary from CLI. Then we don't need to have this in the environment. Change-Id: Ia95dbc49667b70f590a76ecdd90c66ee530e6d51 Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* create_installer.py: Fix argparse configurationPatrik Teivonen2023-03-091-16/+22
| | | | | | | | | | | | Add missing 'default' / 'required' options for arguments. Fix error when generating help: - ValueError: unsupported format character 'V' (0x56) at index 16 Set long path check for Windows systems only Change-Id: If944735b0cc8df89ec11a7ac0d2723ddf46204e7 Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* bldinstallercommon.py: Refactor RPATH/RUNPATH patchingPatrik Teivonen2023-03-0710-159/+235
| | | | | | | | | | | | Simplify RPATH/RUNPATH patching functionality. Make calculate_relpath() use pathlib. Remove unreliable length check, chrpath fails with the error msg. Improve unit tests, add some test ELF binaries. Fix certain input not producing correct relative output paths. Task-number: QTBUG-106027 Change-Id: I33f89f934215d456b428713471db1fe5d89abbd8 Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
* Fix flake8, isort, pylint linter warnings when running git hooksv5.15.13-lts-packagingPatrik Teivonen2023-03-012-5/+16
| | | | | Change-Id: Id4ca869c36b68336429b7c2c8abc4e6a8dcfcbb8 Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* Qtcreator_libclang: Support for LLVM 16.0.0-rc2Cristian Adam2023-02-227-559/+39
| | | | | | | | | | | | | Set HAVE_CLANG_REPL_SUPPORT to OFF, which fixes a linking error with Visual C++ due to the high number of symbols (>65k) Qt 6.4 has a different naming scheme. Use ClangCodeModel plugin tests for the PGO training, which use clangd. Change-Id: I9c823d8bcdeaa8d7ce638624f6de285fc3fbf9c5 Reviewed-by: Eike Ziller <eike.ziller@qt.io>
* Pylint: Enable R0911, R0916, R1702, reduce number of nested blocksPatrik Teivonen2023-02-222-36/+38
| | | | | | | | Enable additional passing checks: R0911, R0916 Fix too-many-nested-blocks and enable: R1702 Change-Id: If6853425a2b3a62409f6512cea585fca4d88eb03 Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* create_installer: Log build task args for easier reproduction of buildsPatrik Teivonen2023-02-222-4/+45
| | | | | | | | Generate and log the CLI arguments required to reproduce the task for the create_installer.py script. This helps with testing builds locally. Change-Id: If789516756524f47381811b08dfba0948a05436a Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* Support creating online repositories from QBSP filesIikka Eklund2023-02-224-99/+437
| | | | | | | | | | | | | | | | | | | | | Add a possibility to create an online repositories from QBSP files. The QBSP file is basically a compressed repository. Utilize the contents from the archive. Introduce a "BuildStrategy" base class that defines an interface to execute the "repository build". The existing functionality maps to IFWRepoBuildStrategy i.e. when building the whole repository from scratch. The QBSPRepoBuildStrategy defines a new way to "build" the repository from an existing .qbsp file. Adapt the release_task_reader.py to understand job configurations for QBSP based repo contents. Task-number: QTQAINFRA-5294 Change-Id: I0544ac21251d8f6eb80b4f6eac2f49af0a401919 Reviewed-by: Patrik Teivonen <patrik.teivonen@qt.io> Reviewed-by: Samuli Piippo <samuli.piippo@qt.io> Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* bldinstallercommon: improve dir strip functionality, add unit testsv6.5.0-beta3-packagingPatrik Teivonen2023-02-144-33/+83
| | | | | | | | | | | - Switch os.path style paths -> pathlib.Path - Switch move function: move_tree -> shutil.move - Add missing unit tests - Improve logging, error messages - Add docstring Change-Id: Ie8c0abe3af0163127dbfe2a016b0471bf9705b0a Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* create_installer: Fetch non-extractable payload directly to correct dirPatrik Teivonen2023-02-133-30/+92
| | | | | | | | | | | | | Remove unnecessary shutil.move for non-extractable payload, download directly to install or data directory depending on the payload. Add sanity check + test for duplicate archive names in a component. We still need to use temporary directory with extractable payload, as the source payload archives can have identical names. Change-Id: Ia9b74d2ef6d718ae89266580b2d326a6ccf5b9e4 Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
* Fix some linter errors, broken Pipfile import, update readmePatrik Teivonen2023-02-107-19/+23
| | | | | | | | | Change minimum Python requirement to 3.6.2 reflecting the current state of the repository. Make sure linter hook passes on this version and fix some errors. Change-Id: I4f569cbda286a2fc878f8d8e5de5d61890a7b301 Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* Add operations for Debian repo creation and publishingIikka Eklund2023-02-106-15/+689
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | While the debian_repo_release.py implements a client wrapper for basic Debian repo operations we need a way to use these opertions in a meaningful way to actually manage releases. The first version of release_repo_updater_deb.py implements a transaction to: - create a repository - populate the repository with content - create snapshot - publish the snapshot i.e. make it visible The transaction is implemented as a command pattern so that if any of the steps should fail an undo can be performed for the executed steps in correct order. Implement a function to crawl URL paths recursively to scan for .deb package content in a concurrent way. Implement a function to download given files concurrently in chunks to minimize memory usage. Add a test to create and publish a repo with test content into filesystem endpoint using local Aptly server instance. Task-number: QTIFW-2813 Change-Id: I6d5bf8ea0c066c35e7c0cfe407c81234327d5c8 Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* release_task_reader.py: fix offline task validation issueAntti Kokko2023-02-072-2/+12
| | | | | | | | Offline tasks settings do not include 'repo_path'. Change-Id: Icb8154cbc16ac5f254f1d2fbb4389baed920da2c Reviewed-by: Patrik Teivonen <patrik.teivonen@qt.io> Reviewed-by: Akseli Salovaara <akseli.salovaara@qt.io>
* Add a summary of sdkcomponent errors at the end of a dry runPatrik Teivonen2023-02-073-0/+29
| | | | | | | | | | When running release_repo_updater.py and create_installer.py scripts, capture a summary of sdkcomponent validation errors during a dry run. Task-number: QTBUG-110633 Change-Id: I5ce637f84bcae580717a9b106cb6ce802ddc6dae Reviewed-by: Antti Kokko <antti.kokko@qt.io> Reviewed-by: Jani Heikkinen <jani.heikkinen@qt.io>
* release_task_reader.py: add validation for mandatory task settingsAntti Kokko2023-02-072-10/+81
| | | | | | | | | | | | Raise 'ReleaseTaskError' if 'repo_path:' or 'config_file:' is not found for ifw tasks. Check that 'repo_path:' exists for debian tasks. Add and update unit tests for task parsing. Change-Id: I57ec9e5031f44aad7be8d04f3e3c4546e9574a90 Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
* release_task_reader.py: Fix argparse CLI task type argumentPatrik Teivonen2023-02-011-2/+2
| | | | | | | "values" -> "choices" Change-Id: I3349d16119c5937b0128d9d280d5d9c9b645fddd Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* Support Debian task type in packaging configurationsIikka Eklund2023-01-304-159/+344
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Refactor the ReleaseTask class as a base class for two specialized cases: - IFWReleaseTask for tasks that use Qt Installer Framework to produce installers or repositories - DebReleaseTask for tasks that produce Debian repositories Support key-value substitution also in the configuration files that are parsed by ReleaseTask classes. Remove unused functionality from ReleaseTask class and change the various getter methods as @property getters: better data hiding and more Pythonic access to container attributes. Construct the task types via an factory: ReleaseTaskFactory To make the task parsing more type specific and straightforward introduce a breaking change in the task specification syntax: [task.foo.bar.abc...] -> [task.ifw.foo.bar.abc...] or [task.deb.foo.bar.abc...] Use "ifw" specifier for tasks that should use Qt Installer Framework. Use "deb" specifier for tasks that should produce Debian repositories. The release configuration files need to be updated at the same time of this change. Task-number: QTIFW-2813 Change-Id: I5c12ae3705a9e89a16b536ceef5382130860e0b0 Reviewed-by: Patrik Teivonen <patrik.teivonen@qt.io> Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* Remove win32api dependency, adapt create_installer.pyPatrik Teivonen2023-01-263-21/+52
| | | | | | | | | | | -When path limit disabled in Windows VMs, short pathnames no longer needed -Changes in create_installer.py, release_repo_updater.py: -Log warning in InstallerTask when running with path limit enabled -Force long path support when calling via the CLI interface -Add CLI option --disable-path-limit-check to bypass the check Change-Id: Ifed8f491ca4cdcb5b166365faccff8782059cecd Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
* dum_debug_infos.py: improve error outputTim Jenssen2023-01-251-3/+12
| | | | | Change-Id: I30bc1675f66b3f11c2291c705cb48ef98f5c9271 Reviewed-by: Tim Jenssen <tim.jenssen@qt.io>
* dum_debug_infos.py: fix create .sym file namesTim Jenssen2023-01-251-1/+1
| | | | | | | | | Using pathlib changed the pathes from / to \ on Windows so the replace did not work anymore. Change-Id: Ia4a69aa3a167b868cc675683fd57b111bfba6d4c Reviewed-by: Patrik Teivonen <patrik.teivonen@qt.io> Reviewed-by: Tim Jenssen <tim.jenssen@qt.io>
* fix install_qt still talks about Qt 5Tim Jenssen2023-01-251-1/+1
| | | | | Change-Id: I584a6de83729174c66bf73f330fd4db9ed3662ba Reviewed-by: Tim Jenssen <tim.jenssen@qt.io>
* PL103: os.makedirs(path) should be replaced by path.mkdir(parents=True)v6.5.0-beta2-packagingPatrik Teivonen2023-01-2415-44/+37
| | | | | Change-Id: Ie72f6555f99eeb8e499011d43d3035424ed9647f Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
* PL102: os.mkdir("foo") should be replaced by foo_path.mkdir()Patrik Teivonen2023-01-241-1/+1
| | | | | Change-Id: Idcf807ffbb785066cfa5086efde2c755c0ad19fe Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* PL122: os.path.splitext("foo.bar") should be replaced by path.suffixPatrik Teivonen2023-01-2411-76/+70
| | | | | | | | | os.path.split and os.path.splitext to pathlib equivalents adjust bldinstallercommon.calculate_relpath unit tests Change-Id: Ieb49864ed4f804cfdff61492edfb622ce1c4a9dc Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* sdkcomponent: Compress to a single archive when using pattern match uriPatrik Teivonen2023-01-243-91/+124
| | | | | | | | | | | | | Change behavior when using pattern match in payload uri: - Do not support extract operation with pattern matching - Compress all matching content to a single payload archive - Include source directory structure in the final archive to avoid name conflicts - Require to declare "archive_name" in configs if using patterns - Adjust create_installer, unit tests Change-Id: I7b89010d1d872f8b3bff8e1d496147675a9a6df1 Reviewed-by: Jani Heikkinen <jani.heikkinen@qt.io>
* release_repo_updater: add supported archive formats for aws syncAntti Kokko2023-01-161-4/+6
| | | | | | | | | | | | | Add tar, tar.bz2, tar.gz, tar.xz and zip to supported archive formats. Rename s3 sync function: old: sync_production_7z_to_s3 new: sync_production_payload_to_s3 Change-Id: I172635ffc77f8cc677d9006ead0773cfddd50354 Reviewed-by: Patrik Teivonen <patrik.teivonen@qt.io> Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
* PL119: os.path.basename("foo/bar") should be replaced by bar_path.namePatrik Teivonen2023-01-1310-26/+31
| | | | | | | | | Get basename of a path using pathlib.Path.name. Enable PL119 in flake8. Update copyright year for changed files. Change-Id: Ice7cd7e9246950dd39321b108bb68c9466f62189 Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* PL101: os.chmod("foo", perm) should be replaced by Path.chmod(perm)Patrik Teivonen2023-01-132-6/+6
| | | | | | | | Use pathlib to change file system permissions. Enable PL101 in flake8 Change-Id: Ibaadea6d0768d89770ef0c05ca5f1c3ac9cb0133 Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* test_release_repo_updater: Fix repogen path in unit test assertionPatrik Teivonen2023-01-131-2/+3
| | | | | | | | | | Previous change a0abcb9311c7bb02263bc51ef0189648dbdeec24 made upload_ifw_to_remote() return a relative path. Update unit test assertion with correct path and make unit tests pass. Change-Id: I9ea6aa68a926a5f30a8d3d68057b1b3a4c1038db Reviewed-by: Antti Kokko <antti.kokko@qt.io>
* PL111: os.path.expanduser("~") should be replaced by Path.home()Patrik Teivonen2023-01-134-16/+15
| | | | | | | | | | | | Enable PL111 check in flake8-pathlib. Replace os.path.expanduser with Path.home() Remove remote_server_home argument from upload_ifw_to_remote() as rsync uses paths relative to the (remote server's) home directory by default. If the remote server's home directory is different from the local one, it will fail. Change-Id: I31a8dee61889c749b240aa6af9c440e9371acebf Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
* Add Aptly repository manager client adapter for Debian repositoriesIikka Eklund2023-01-126-0/+826
| | | | | | | | | | | | | | | | | | | | Implement an adapter wrapping the aptly-api-client Python interface for easier access to the functionality. Add basic support to: - create repository - add files to the repository - create a snapshot - publish and sign the snapshot to given endpoint and prefix - list operations for the resources - delete operations for the resources Also add an command line interface for testing purposes. Tests are enabled if 'aptly' is found from PATH. Task-number: QTIFW-2813 Change-Id: I062e0967ec025f532a93e3466cae829bdd9e63c8 Reviewed-by: Jani Heikkinen <jani.heikkinen@qt.io>
* QtCreator: Sign online/Qt offline bundle on macOSEike Ziller2023-01-121-38/+36
| | | | | | | | | | | | | | | | | | We need to sign the Qt Creator app that we install with the online and Qt offline installers, to avoid an issue with access permissions on M1 Macs. For that the installer may no longer write into the app bundle. The last remaining thing that the installer wrote, was the optional plugins. Instead, add a relative plugin path (expanding to "<Qt>/Tools/QtCreator <version>/PlugIns") to the install settings of Qt Creator, and sign that bundle. The installer has to install the optional plugins there, instead of into the bundle. For consistency, do the same with the commercial Qt Creator offline installers. Task-number: QTCREATORBUG-26705 Change-Id: I728ebf28b658b23743afb563ccc5cdf96639fa7e Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
generated by cgit v1.2.3 (git 2.25.1) at 2024-05-08 10:54:15 +0000