| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Offline installer/Online repo content should be notarized
to avoid Gatekeeper issues on latest versions of macOS.
Before compressing the final payload component archives,
(re-)sign, notarize, and finally staple the relevant types of
nested install content before distribution. Steps will be
performed if '--notarize-payload' argument is passed onto
create_installer.py script or when 'notarize_payload' is
specified and set to yes|true|1 in the IFW task config.
Change-Id: Icb5c482674af9421fa08e58f4e27043a6766e5c7
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Simplify RPATH/RUNPATH patching functionality.
Make calculate_relpath() use pathlib.
Remove unreliable length check, chrpath fails with the error msg.
Improve unit tests, add some test ELF binaries.
Fix certain input not producing correct relative output paths.
Task-number: QTBUG-106027
Change-Id: I33f89f934215d456b428713471db1fe5d89abbd8
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Remove unnecessary shutil.move for non-extractable payload,
download directly to install or data directory depending on the payload.
Add sanity check + test for duplicate archive names in a component.
We still need to use temporary directory with extractable payload, as
the source payload archives can have identical names.
Change-Id: Ia9b74d2ef6d718ae89266580b2d326a6ccf5b9e4
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
| |
Change minimum Python requirement to 3.6.2 reflecting the
current state of the repository. Make sure linter hook passes
on this version and fix some errors.
Change-Id: I4f569cbda286a2fc878f8d8e5de5d61890a7b301
Reviewed-by: Antti Kokko <antti.kokko@qt.io>
|
|
|
|
|
|
|
|
|
|
| |
When running release_repo_updater.py and create_installer.py scripts,
capture a summary of sdkcomponent validation errors during a dry run.
Task-number: QTBUG-110633
Change-Id: I5ce637f84bcae580717a9b106cb6ce802ddc6dae
Reviewed-by: Antti Kokko <antti.kokko@qt.io>
Reviewed-by: Jani Heikkinen <jani.heikkinen@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Change behavior when using pattern match in payload uri:
- Do not support extract operation with pattern matching
- Compress all matching content to a single payload archive
- Include source directory structure in the final archive to avoid name
conflicts
- Require to declare "archive_name" in configs if using patterns
- Adjust create_installer, unit tests
Change-Id: I7b89010d1d872f8b3bff8e1d496147675a9a6df1
Reviewed-by: Jani Heikkinen <jani.heikkinen@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Issues:
-Target install paths may include forward slashes even on Windows, so
'get_archive_install_dir' will not join the paths properly.
-When archive target install base specified, may not be repackaged
with correct paths
-Use pathlib.Path in 'get_archive_install_dir' (no need to normalize)
-Make paths relative to root install dir by stripping both types of
leading/trailing separators
-Remove check for missing install dir, and use root dir as default
-Adapt unit tests, create_installer.py
Change-Id: I35f66233b34bf57da94317412049ac15c018b203
Reviewed-by: Antti Kokko <antti.kokko@qt.io>
|
|
|
|
|
|
|
|
| |
Speed up the uri resolvation by resolving child dirs concurrently.
Task-number: QTBUG-105693
Change-Id: I44fadc6de4097f43c57a5dd6df275bf94e0fc5e3
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
| |
Add the possibility to specify fnmatch patterns in payload uris
All the matching payload content will be added to the component
Task-number: QTBUG-105693
Change-Id: Id1ac69e3ffbd2b8be4bcac11ad889bc57a1db5c0
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add --allow-broken-components option for partial installer creation.
Skip downloading ifw tools + payload, online repo creation in dry run.
Add more generic uri_exists function to replace is_content_url_valid.
Add two dry run modes (--dry-run=configs|payload):
- Config validation mode: check only the configuration for errors
- Payload URI validation mode: in addition to the above,
checks if payload archive uris exist on the server (or file system)
Make these dry run options available in release_repo_updater.py and
create_installer.py with a separate parser.
Change-Id: Ie77cf57cbae3302ed3b9f7cd5d3a61351f0e406d
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This reverts commit e3502143b1ee8febd162eab0531c2d8a774af617.
Reason for revert: Fixed:
- Download file format should derive from archive_url
- Final save format determined from archive_name
- Rename extract_archive -> disable_extract_archive
- Do not treat payload with disable_extract_archive as raw artifact
(Still needs to be packed to apply new install directory structure)
- Sanity check and unit test adjustments
- Download payload to its own temporary directory before processing it
(prevents name conflicts with other payload and static data)
- Support non-tarballed .gz, .xz, .bz2 in bldinstallercommon.extract_file
Change-Id: I852615101aafd1540e9f52dc924b693c1b4dae20
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
| |
This reverts commit af10f08f8d9ccc3a68b58aa701cb820c581c27c7.
Reason for revert: archive handling issue
Change-Id: Ia6bb03911502c84d7b52b1eaca18758f9f5f4cab
Reviewed-by: Antti Kokko <antti.kokko@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
-Remove archiveresolver.py in favor of simpler ArchiveResolver class
-Clean up sdkcomponent.py, use dataclasses, add better validations
-Use raw CI artifacts without extracting and repackaging when possible
(archive format supported by IFW, and patching not required)
-Add unit tests
-Adapt create_installer.py to required changes
Follow up: create_installer.get_component_data()
Change-Id: If4068dfd0842c5d5c343fa99ddabac86450adbb1
Reviewed-by: Antti Kokko <antti.kokko@qt.io>
|
|
|
|
|
|
|
|
|
|
| |
This reverts commit 64b709d4b422e6d59f9e703fbc1124dd3f77d556.
Reason for revert: required dependent changes are done, we can bring this change back
Change-Id: I1692b433eda3fd2a1e5aa541785dae942cbd1665
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
Reviewed-by: Akseli Salovaara <akseli.salovaara@qt.io>
|
|
|
|
|
|
|
|
|
| |
This reverts commit bd38f3c2897c3f82f8c10464f70ed301c618dae7.
Reason for revert: temporarily revert due to colorlog import issues
Change-Id: I8e47743780c8998580ca40de56cbe0d0b5f49504
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
-Remove inconsistent print function/logging module usage across scripts
-Initialize logger from logging_util.py
-Use CI friendly format in logging to stdout
-Log file format with extra debug information (script, line, process)
-Remove rainbow_logging_handler in favor of colorlog
-Remove unnecessary print/log functions from unit tests
-Remove unnecessary whitespace in logging messages
Change-Id: Ib913cf2f6a2b02e06628f94dceeeaf845191b96b
Reviewed-by: Antti Kokko <antti.kokko@qt.io>
|
|
|
|
|
|
|
| |
This feature is no longer used in packaging configurations.
Change-Id: Iaffbbca360ef7f494daac82c40fd267cf0fa5716
Reviewed-by: Antti Kokko <antti.kokko@qt.io>
|
|
|
|
|
|
|
|
| |
Enable --strict parameter in mypy.
Add missing type hints and resolve remaining type errors.
Change-Id: Id9547cc3351b88729930c577d4ed628780c1447b
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add comment about disabled checks in pylint hook and enable all default pylint checks except for those.
Address these miscellaneous pylint warnings:
-C0123: Use isinstance() rather than type() for a typecheck. (unidiomatic-typecheck)
-R1714: Consider merging these comparisons with "in" (consider-using-in)
-R1722: Consider using sys.exit() (consider-using-sys-exit)
-C0206: Consider iterating with .items() (consider-using-dict-items)
-R1710: All return statements in a function should return an expression. (inconsistent-return-statements)
-W0707: Consider explicitly re-raising using the 'from' keyword (raise-missing-from)
-W1510: Using subprocess.run without explicitly set `check` is not recommended. (subprocess-run-check)
-W0102: Dangerous default value as argument (dangerous-default-value)
-W1508: os.getenv default type is builtins.bool. Expected str or None. (invalid-envvar-default)
-W0212: Access to a protected member of a client class (protected-access)
-W0631: Using possibly undefined loop variable (undefined-loop-variable)
-W1113: Keyword argument before variable positional arguments list in the definition of __init__ function (keyword-arg-before-vararg)
-W0201: Attribute defined outside __init__ (attribute-defined-outside-init)
-W0221: Number of parameters in overridden method differ (arguments-differ)
-C0202: Class method should have 'cls' as first argument (bad-classmethod-argument)
-W1505: Using deprecated method currentThread() (deprecated-method)
-R1707: Disallow trailing comma tuple (trailing-comma-tuple)
-W0603: Using the global statement (global-statement)
-W0602: Using global for 'OUTPUT_STATES' but no assignment is done (global-variable-not-assigned)
-C2801: Unnecessarily calls dunder method __contains__. Use in keyword.
Change-Id: Ia1c6ff3f6c328a94b40a1c0f11c83151abc91423
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
| |
Enable this check and rename function arguments to be PEP8 compliant.
Change-Id: Ia75f66decdab75197335b1cff5f87c6f25b0d259
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
| |
Enable this check and rename functions.
Change-Id: Ifc6c3248f6c6b90a240d66bb2d20cd716d887729
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Enable pylint checks, and fix the following anti-patterns:
- superfluous-parens
- no-else-return, no-else-raise, no-else-break, no-else-continue
- unnecessary-comprehension
- pointless-statement, pointless-string-statement
- unnecessary-pass, useless-return
- unused-variable, unused-argument
- useless-object-inheritance
- condition-evals-to-constant
- simplifiable-if-expression
- redundant-unittest-assert
Requires to be submitted with the following change in tqtc-qtsdk:
251649c633eb7acc417cf88fd1c7f885e2bccd7b
Change-Id: I3dd50a7030e06565ae6c04d0a0e3aa07bc68218f
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
|
|
| |
Enable following checks in pylint:
- C0209: Formatting a regular string which could be a f-string
- W1202: Use lazy % formatting in logging functions
- W1203: Use lazy % formatting in logging functions
Use f-string for string formatting where recommended and lazy % formatting in logging.
Change-Id: I14924d27e0f800c47af1ec903be4c51f24191d26
Reviewed-by: Antti Kokko <antti.kokko@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
-Add isort to Pipfile, isort hook with blackformatter profile
-Fix imports in unittest hook
-Group and sort imports by their category (PEP8)
-Blank line between import groups (PEP8)
-Move all imports to toplevel (PEP8)
-Don't unnecessarily append to sys.path (PEP8)
-Import only required functions (this also helps reduce line length)
-Update deprecated imports such as builtins.WindowsError, ConfigParser.readfp
-Remove Python 2, cyclic and unused imports
-Fix platform dependent imports and remove unnecessaty try-excepts
-Remove unnecessary comments before imports, use in-line comments when needed
-Rename conflicting functions in imports, variables as absolute imports are recommended (PEP8)
Change-Id: I5ac7a3499ea47a6bdc9a9a2b5211841f65c011ea
Reviewed-by: Akseli Salovaara <akseli.salovaara@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fix whitespace, Enable following checks in flake8:
- E225 Missing whitespace around operator (E225)
- E226 Missing whitespace around arithmetic operator (E226)
- E231 Missing whitespace after ',', ';', or ':' (E231)
- E221 Multiple spaces before operator (E221)
- E201 Whitespace after '(' (E201)
- E252 Missing whitespace around parameter equals
- E251 Unexpected spaces around keyword / parameter equals
- E202 Whitespace before ')' (E202)
- E222 Multiple spaces after operator (E222)
- E211 Whitespace before '(' (E211)
Change-Id: Ib14a4a83dbc06cdfac511eb09912d7b9232df7e6
Reviewed-by: Akseli Salovaara <akseli.salovaara@qt.io>
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Correct the amount of lines in function and class definitions as well as
at the end of the file. Enable following flake8 checks:
- E301 Expected 1 blank line, found 0
- E302 Expected 2 blank lines, found 0
- E303 Too many blank lines
- E305 Expected 2 blank lines after end of function or class
- E306 Expected 1 blank line before a nested definition
- W391 Blank line at end of file
Change-Id: Ia9bb511ded47df1f43bfffa4be00abfdbda60cae
Reviewed-by: Akseli Salovaara <akseli.salovaara@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Enable following checks in flake8, and fix the resulting errors:
-F401 Module imported but unused
-F841 Local variable name is assigned to but never used
-F811 redefinition of unused module
-F541 (f-string is missing placeholders)
-F523 .format(...) unused positional arguments
Change-Id: Iba0e9f0148054d86b39b31202aba81a7f5c03a93
Reviewed-by: Akseli Salovaara <akseli.salovaara@qt.io>
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
-Update year and license header
-Update header formatting to match others in send_header_diff_kmail.py
-Change "## " to "# " in license headers to make them PEP8 compliant.
-Reference: https://peps.python.org/pep-0008/#block-comments
-flake8 rule descriptions and codes:
-Block comment should start with '# ' (E265)
-Too many leading '#' for block comment (E266)
Change-Id: Ife039694462e9908bc22ab253ba3202c7245a7a3
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
| |
Change current year for copyright statement and use Python 3 shebang in
Python 3 scripts with consistent style.
Change-Id: I61983c80cd09d0a8a4b458ac1ae5a72ccd2d1ce2
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
| |
Initial step to cleanup the code and prepare for
porting the script for Python3
Change-Id: If06b639da039b222524fb51e315cfb4506d5f28c
Reviewed-by: Antti Kokko <antti.kokko@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add new substition (archives_extract_dir) that can be used
together with Extract operation.
archives_extract_dir: %QT_VERSION%/Boot2Qt/%MACHINE%/toolchain
<Operationname="Extract">
<Argument>@TargetDir@/%ARCHIVES_EXTRACT_DIR%</Argument>
</Operation>
Change-Id: If364ab48ec5c130e9d43ec218eac00804bebc2bd
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Support "archive_action" per archive in config files. The action
needs to point to existing script file that must be executable.
archive_action: content_cleaner.py, --arg1=abc --arg2=abc
A comma separates the script from arguments.
The '--input-dir, is mandatory option for the action script.
The rest of the options are passed as such to the action script.
A content_cleaner.py script is provided which can be used to clean
content from the extracted archive based on glob rules.
Change-Id: I90e57cdacb1e9140f2b31403c7ef0899b99bf023
Reviewed-by: Antti Kokko <antti.kokko@qt.io>
|
|
|
|
|
|
|
| |
Just ignore and print a warning message.
Change-Id: I89a9bc595408e1e5ed0d547da01c4eb72ca9865b
Reviewed-by: Antti Kokko <antti.kokko@qt.io>
|
|
|
|
|
|
|
|
| |
Because configuration files use Unix paths we can't use os.path.join
on Windows.
Change-Id: Icfca76d540498fbd2c13a735934695dc4bffefb3
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The scripts already support defining a "data/" under the package
template. Though there was a bug where you can define filename(s)
directly in config files under the "data/", it mistakenly tried first
to check the given filename in the script's execution directory.
It should never try to check installer content directly from the
script's execution folder itself.
To support installing content per archive outside the components
'target_install_base' allow defining the same for individual archives
as well.
Task-number: QTIFW-1556
Change-Id: I861e636870aa608ad2a11e806652d7ed5ac57dfd
Reviewed-by: Antti Kokko <antti.kokko@qt.io>
|
|
|
|
|
|
|
|
|
|
|
| |
New component configs 'component_sha1_uri' and 'component_sha1_file' can
define a URI to a downloadable file and file path inside a downloadable
archive, respectively. The file is used to read a sha1 for the component,
which can then be used with substitution '%COMPONENT_SHA1%'.
Task-number: AUTOSUITE-760
Change-Id: If0d2a30c6e07f61df89609194ce96077e81fa9a8
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
| |
Rename the package components and template files using
all substitutions, instead of just the predefined
QT_PKG_VERSION.
Change-Id: Iaedef6ac3bd7e4a316e445c6a05297d0ce4aa845
Reviewed-by: Antti Kokko <antti.kokko@qt.io>
|
|
|
|
|
|
|
|
|
| |
Generic Qt package templates have version substitutions
on package names. Add package name version substitution handling
for QT_PKG_VERSION and QT_PKG_VERSION_MINOR variables.
Change-Id: I2ccdd72da7a09b9b0215c0fbba03a8e80d7f38dd
Reviewed-by: Antti Kokko <antti.kokko@qt.io>
|
|
|
|
|
|
|
|
|
| |
Fixing issues with another patch set on top of this change.
This reverts commit ed017afe477b55bd01bd16185ac83d3558ac1c5b.
Change-Id: Ia0a3ce99b23c53082d4fcb2a37d4b36db1495ca9
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
| |
Causes multitude of problems, reverting and fixing.
This reverts commit 3055eab4c65a45d292c76335a965777a019942bf.
Change-Id: I5639fbf6f02d90312cf068720b9b9d29126b4b87
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add support to 'include_filter' parameter used in packaging
configuration to separate between commercial and lgpl build.
For example for license repository commercial and lgpl
licenses are marked 'include_filter: opensource' or
'include_filter: enterprise', components without the
include filter are always included.
Change-Id: I4ce81a79e7fdf2320eb4ae0f89dd950e84bd1f93
Reviewed-by: Iikka Eklund <iikka.eklund@qt.io>
|
|
|
|
|
|
| |
Change-Id: I8844ab39924208b438a77376e6d09bf46aca8672
Reviewed-by: Antti Kokko <antti.kokko@qt.io>
Reviewed-by: Akseli Salovaara <akseli.salovaara@qt.io>
|
|
|
|
|
|
|
|
|
| |
The sys.exit terminates just the current thread. This is an issue
if using the threadedwork.py with multiple threads as any failure
should terminate the whole script.
Change-Id: I9926c4cf9c6754a5182d331dae5a2e66c5ff95b6
Reviewed-by: Eike Ziller <eike.ziller@qt.io>
|
|
|
|
|
|
| |
Change-Id: Id7e88b72fbe2be5f197c21e0ef4efd1454d7f518
Reviewed-by: Iikka Eklund <iikka.eklund@theqtcompany.com>
Reviewed-by: Jani Heikkinen <jani.heikkinen@theqtcompany.com>
|
|
|
|
|
|
|
|
|
|
| |
sdkcomponent.py:47: [W0611(unused-import), ] Unused import ArchiveLocationResolver
sdkcomponent.py:139: [C0325(superfluous-parens), ] Unnecessary parens after 'not' keyword
sdkcomponent.py:183: [W0301(unnecessary-semicolon), ] Unnecessary semicolon
Change-Id: I343a3574bcb07e645f69f6dae9f25f8726134aa2
Reviewed-by: Kai Koehne <kai.koehne@theqtcompany.com>
Reviewed-by: Iikka Eklund <iikka.eklund@theqtcompany.com>
|
|
|
|
|
|
|
|
|
| |
If sdk component marked optional for offline installer creation
and any of the archives included for the component are missing then
just log the issue and ignore that component.
Change-Id: I98b062f603e3d24916f2bfc98ef835d4910bcac9
Reviewed-by: Antti Kokko <antti.kokko@digia.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Introduce new variable for installer component definition. E.g.
[qt.extra.somepackage]
optional_for_offline: true
archives: extra.somepackage
target_install_base: /5.3/gcc_64
...
...
When the "optional_for_offline" contains keyword "true", "yes" or
"1" it is marked optional for offline installer builds.
So if the package is invalid e.g. content is missing it will
not stop the offline installer creation.
Change-Id: Id782fc1bf251d29bdbd34430cfef708b26786154
Reviewed-by: Jani Heikkinen <jani.heikkinen@digia.com>
Reviewed-by: Matti Paaso <matti.paaso@digia.com>
|
|
|
|
|
|
|
| |
task-number: QTBUG-39219
Change-Id: If89f3af95d342591171bfeda7bbdf6604532c4f8
Reviewed-by: Iikka Eklund <iikka.eklund@digia.com>
|
|
|
|
|
|
|
|
| |
With Python 2.7 the "replace(os.linesep, '')" seems to be broken on
Windows. Replace with version that works on all platforms.
Change-Id: Iad9f7812a20c0cc1f8de4b688e4ea999373ebcb0
Reviewed-by: Sergio Ahumada <sahumada@blackberry.com>
|
|
|
|
|
|
| |
Change-Id: I0757e0fb32dc5595530e495cfbf2bfd3dafdab88
Reviewed-by: Sergio Ahumada <sahumada@blackberry.com>
Reviewed-by: Antti Kokko <antti.kokko@digia.com>
|