diff options
Diffstat (limited to 'sources/pyside6/doc/developer')
| -rw-r--r-- | sources/pyside6/doc/developer/adapt_qt.rst | 54 | ||||
| -rw-r--r-- | sources/pyside6/doc/developer/add_module.rst | 61 | ||||
| -rw-r--r-- | sources/pyside6/doc/developer/add_port_example.rst | 67 | ||||
| -rw-r--r-- | sources/pyside6/doc/developer/add_tool.rst | 51 | ||||
| -rw-r--r-- | sources/pyside6/doc/developer/documentation.rst | 73 | ||||
| -rw-r--r-- | sources/pyside6/doc/developer/enumfeatures_doc.rst | 160 | ||||
| -rw-r--r-- | sources/pyside6/doc/developer/extras.rst | 55 | ||||
| -rw-r--r-- | sources/pyside6/doc/developer/feature-motivation.rst | 303 | ||||
| -rw-r--r-- | sources/pyside6/doc/developer/index.rst | 35 | ||||
| -rw-r--r-- | sources/pyside6/doc/developer/limited_api.rst | 703 | ||||
| -rw-r--r-- | sources/pyside6/doc/developer/signature_doc.rst | 361 |
11 files changed, 1923 insertions, 0 deletions
diff --git a/sources/pyside6/doc/developer/adapt_qt.rst b/sources/pyside6/doc/developer/adapt_qt.rst new file mode 100644 index 000000000..aec81fb0f --- /dev/null +++ b/sources/pyside6/doc/developer/adapt_qt.rst @@ -0,0 +1,54 @@ +.. _developer-adapt-qt: + +Adapt to new Qt versions +======================== + +Adapting to source changes +-------------------------- + +The dev branch of PySide is switched to a new Qt minor version +after its API review is finished and the API is stable. + +Until that happens, a patch should be continuously developed +to adapt to this version. + +The `new classes page <https://doc-snapshots.qt.io/qt6-6.7/newclasses67.html>`_ +is a good source of information for new API. + +New classes and should be added to the type system file (using +a ``since`` attribute) and ``CMakeList.txt`` file of the respective module. + +Should the class not be available on all platforms, the respective +``QT_CONFIG`` macro needs to be specified in the type system file and +feature checks need to be added to ``CMakeList.txt`` (see for example +``QPermission``). + +The process consists of running a build and evaluating the log file. +The script +`shiboken2tasks.py <https://code.qt.io/cgit/qt-creator/qt-creator.git/tree/scripts/shiboken2tasks.py>`_ +from the *Qt Creator* repository can be used to convert the shiboken warnings +into a `task file <https://doc.qt.io/qtcreator/creator-task-lists.html>`_ +for display in the build issues pane of *Qt Creator*. + +Warnings about new enumerations will be shown there; they should be added +to type system file using a ``since`` attribute. + +Warnings about not finding a function signature for modification +also need to be handled; mostly this is a sign of a function parameter +being changed from ``int`` to ``qsizetype`` or similar. + +If the build succeeds, a test run should be done. + +The Qt source code should be checked for new overloads +(indicated by ``QT6_DECL_NEW_OVERLOAD_TAIL`` starting from 6.7). +The resolution needs to be decided for each individual case, +mostly by removing old functions and using ``<declare-function>`` +to declare new API. + +Bumping the version +------------------- + +To instruct ``COIN`` to use the next version of Qt, adapt the files +``coin/dependencies.yaml`` and/or ``product_dependencies.yaml`` accordingly. +Next, the wheel names should be changed by adapting +``sources/shiboken6/.cmake.conf`` and ``sources/pyside6/.cmake.conf``. diff --git a/sources/pyside6/doc/developer/add_module.rst b/sources/pyside6/doc/developer/add_module.rst new file mode 100644 index 000000000..2eb962207 --- /dev/null +++ b/sources/pyside6/doc/developer/add_module.rst @@ -0,0 +1,61 @@ +.. _developer-add-module: + +Add a new module +================ + +New modules can be added for many reasons, the most important +one is when Qt enables or includes a new one for a new release. + +Adding the bindings, and documentation are the essentials +to include new modules, but adding tests and examples is ideal. + +Add bindings +------------ + +- Find the correct name (look at the include path of Qt). +- Add the module to the ``coin/dependencies.yaml`` file. +- Add it to ``sources/pyside6/cmake/PySideHelpers.cmake``. +- Add it to ``build_scripts/wheel_files.py`` (plugins, translations). +- Copy an existing module to ``sources/pyside6/PySide6/<name>``. +- Adapt the ``typesystem.xml`` and ``CMakeList.txt`` (using for example + *Qt Creator*'s case-preserving replace function). +- Make sure the dependencies are correct. +- Find the exported public classes, add them to the ``typesystem.xml`` file, + checking whether they are ``value-type`` or ``object-type``. Add their enums + and flags. +- Add the wrapper files to ``CMakeList.txt``. +- Create a test dir under ``sources/pyside6/tests`` with an empty + ``CMakeList.txt``. +- Try to build with the module added to the ``--module-subset`` option of + ``setup.py``. +- Watch out for shiboken warnings in the log. +- Be aware that ``ninja`` mixes stdout and stderr, so, the first warning is + typically hidden behind a progress message. +- A convenient way of doing this is using + ``qt-creator/scripts/shiboken2tasks.py`` from the + `*Qt Creator* repository <https://code.qt.io/cgit/qt-creator/qt-creator.git>`_ + converting them to a ``.tasks`` file which can be loaded into *Qt Creator*'s + issue pane. +- Link errors may manifest when ``generate_pyi`` imports the module trying + to create signatures. They indicate a missing source file entry + or a bug in the module itself. + +.. note:: For the build to succeed, the module must follow the Qt convention + of using ``#include <QtModule/header.h>`` since module include paths + are not passed in PySide. + +Distribution +------------ + +- Determine to which wheel the module belongs according to + `Qt Modules <https://doc.qt.io/qt-6/qtmodules.html>`_. +- Add the module to ``build_scripts/wheel_files.py`` for use by + ``create_wheels.py``. +- Add the module to one of the ``README.pyside6_*.md`` files. + +Add documentation +----------------- + +- Add entry to ``sources/pyside6/doc/modules.rst``. +- Add a .qdocconf.in file in ``sources/pyside6/doc/qtmodules``. +- Add module description ``.rst`` file in ``sources/pyside6/doc/extras``. diff --git a/sources/pyside6/doc/developer/add_port_example.rst b/sources/pyside6/doc/developer/add_port_example.rst new file mode 100644 index 000000000..b99641f45 --- /dev/null +++ b/sources/pyside6/doc/developer/add_port_example.rst @@ -0,0 +1,67 @@ +.. _developer-add-port-example: + +Add a new example or port one +============================= + +Adding examples is a good exercise for people wanting to become familiar with +the modules and its functionality. + +You can either design an example from scratch or inspired in another +application, or simply you can port an existing Qt example that does not have +a Python counterpart. + +Example code should be free of `flake8 <https://pypi.org/project/flake8/>`_ +warnings; this is enforced by a bot. A configuration file is provided +at the root of the repository. Offending lines can be excluded by a +``noqa`` directive if there is a good reason to do so. + +Keep in mind we do allow 100 columns for line length. + +Additionally, please use `isort <https://pypi.org/project/isort/>`_ to keep the +imports ordered and consistent with other examples. + +For example: + +.. code-block:: bash + + $ flake8 --config pyside-setup/.flake8 your_file.py + $ isort your_file.py + + + +Add a new example +----------------- + +- Check if the topic your example covers is not in an existing example already. +- Create a new directory inside the ``examples/<module>`` you think + is more relevant. +- Inside, place the code of the example, and also a ``.pyproject`` + file listing the files the example needs. +- If you want the example to be automatically displayed on the + example gallery, include a ``doc`` directory that contains a ``rst`` + file and a screenshot. Check other examples for formatting questions. +- When writing the ``rst`` file, you can include code snippets using + the ``literalinclude`` directive specifying the relative path + as listed in the ``.pyproject`` file. The `example_gallery` tool will + expand this (see the `pointconfiguration` example). +- For the code displayed in the tabs, you can create ``rstinc`` files + in the ``doc`` directory containing some description explaining them + (see the `samplebinding` example). + +Port a Qt example +----------------- + +- Quickly check the C++ example, fix outdated code. +- Port the sources using ``tools/tools/qtcpp2py.py`` (front-end for + ``snippets-translate``). +- Note that our examples need to have unique names due to the doc build. +- Verify that all slots are decorated using ``@Slot``. +- Add a ``.pyproject`` file (verify later on that docs build). +- Add a ``doc`` directory and descriptive ``.rst`` file, + and a screenshot if suitable (use ``optipng`` to reduce file size). +- Add the ``"""Port of the ... example from Qt 6"""`` doc string. +- Try to port variable and function names to snake case convention. +- Remove C++ documentation from ``sources/pyside6/doc/additionaldocs.lst``. + +.. note:: Example screenshots in ``.png`` should be optimized by + running ``optipng -o 7 -strip all``. Alternatively, the ``.webp`` format can be used. diff --git a/sources/pyside6/doc/developer/add_tool.rst b/sources/pyside6/doc/developer/add_tool.rst new file mode 100644 index 000000000..732e6b915 --- /dev/null +++ b/sources/pyside6/doc/developer/add_tool.rst @@ -0,0 +1,51 @@ +.. _developer-add-tool: + +Add a new tool or a Qt tool wrapper +=================================== + +Tooling is essential to |project|, for that reason you can find many ad-hoc +tools in the repository, which include wrappers of Qt tools or newly developed +tools to solve issues, or improve some project workflows. + +Add a new tool +-------------- + +Tools not available to end users +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This depicts the tools that are not shipped with Qt for Python wheels and are used to aid +Qt for Python development + +- Place your tool in the ``tools`` directory. +- If your project has more than one file, create a directory. +- Create a ``.pyproject`` file including all the relevant files + for your tool. + +Tools available to end users +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- Place your tool in the ``sources/pyside-tools`` directory. +- If your project has more than one file, create a directory. +- Create a ``.pyproject`` file including all the relevant files + for your tool. +- Add the relevant files in ``sources/pyside-tools/CMakeLists.txt``. +- Add the tool in ``sources/pyside-tools/pyside_tool.py``. +- Add the tool in ``build_scripts/__init__.py`` to create the setuptools entry points + i.e. this enable using the tool from the console as "pyside6-<tool_name>" +- Add an entry to ``sources/pyside6/doc/gettingstarted/package_details.rst``. +- Include the necessary Qt binaries explicitly on ``build_scripts/wheel_files.py`` +- Build with ``--standalone``, verify it is working. + + +Add a Qt tool wrapper +--------------------- + +- Add the relevant files in ``sources/pyside-tools/CMakeLists.txt``. +- Add the tool in ``sources/pyside-tools/pyside_tool.py``. +- Add the tool in ``build_scripts/__init__.py`` to create the setuptools entry points + i.e. this enable using the tool from the console as "pyside6-<tool_name>" +- Add an entry to ``sources/pyside6/doc/tools/index.rst`` and the detailed + documentation to ``sources/pyside6/doc/tools/<tool_name>.rst``. +- Include the necessary Qt binaries explicitly on ``build_scripts/wheel_files.py`` +- Add the necessary files to ``build_scripts/wheel_files.py``. +- Build with ``--standalone``, verify it is working. Also, check if the wheel bundles the tool. diff --git a/sources/pyside6/doc/developer/documentation.rst b/sources/pyside6/doc/developer/documentation.rst new file mode 100644 index 000000000..517bd46f1 --- /dev/null +++ b/sources/pyside6/doc/developer/documentation.rst @@ -0,0 +1,73 @@ +.. _developer-documentation: + +Fixing Documentation issues +=========================== + +Fixing texts +------------ + +Shiboken's ``<inject-documentation>`` element can be used to add texts. +It's ``"replace"`` mode is currently not implemented, though. + +Fixing snippets +--------------- + +Snippets can replaced by placing a Python equivalent under ``sources/pyside6/doc/snippets``. +The directory structure matches that of Qt. To replace a snippet with the id `0` in +``qtbase/examples/foo/snippet.cpp``, place a file ``qtbase/examples/foo/snippet_0.cpp.py`` +under that directory (one snippet per file with the snippet id appended to the base name). + +More complicated mappings can be added to ``tools/snippets_translate/override.py``. + +# Recreating the module descriptions after a Qt major version change + +The source tree contains .rst files containing the module description in +doc/extras (named for example "QtCore.rst"). They are extracted/adapted from +the C++ module descriptions. If there is no module description file, shiboken +will extract the module description from the webxml files generated by qdoc. +This ends up in the build directory under doc/rst/PySide6/<module>/index.rst. +It can be used as a starting point for a module description file. C++ +specific information like build instructions should be removed. + +The descriptions may link to tutorials which can be added to additionaldocs.lst +for webxml extraction. + +Maintaining additionaldocs.lst +------------------------------ + +The file is a list of additional documentation files. These are basically Qt +tutorials referenced by the documentation. They will receive some Python +adaption by shiboken/sphinx. + +The list can be created by the below script and some hand-editing. It will find +almost all documents. Quite a number of them might be unreferenced, but there +is no good way of filtering for this. +Pages of examples that exist in Python should be removed. + +.. code-block:: bash + + for F in *.webxml + do + echo "$F" | egrep '(-index)|(-module)|(-qmlmodule)\.webxml$' > /dev/null + if [ $? -ne 0 ] + then + if fgrep '<para>' "$F" > /dev/null # Exclude reference only + then + egrep "(<class )|(<namespace )" $F > /dev/null || echo $F + fi + fi + done + +Inheritance graphs +------------------ + +``inheritance_diagram.pyproject`` lists the script involved in inheritance +graph generation, ``inheritance_diagram.py`` being the main one used by sphinx. +The others have main-test drivers for checking. + +There are 2 scripts used for determining the inheritance: +* ``json_inheritance.py`` (env var ``INHERITANCE_FILE``) reads a + inheritance.json file containing the class hierarchy generated by + shiboken's doc generator. + +* ``import_inheritance.py`` actually tries to import the class (legacy) diff --git a/sources/pyside6/doc/developer/enumfeatures_doc.rst b/sources/pyside6/doc/developer/enumfeatures_doc.rst new file mode 100644 index 000000000..91b7b6346 --- /dev/null +++ b/sources/pyside6/doc/developer/enumfeatures_doc.rst @@ -0,0 +1,160 @@ +.. _enum-features: + +The Set of Enum Features +======================== + +The development of the new Python enums took the form of a series of patches. +While we put a lot of effort into supporting the old Enums (without promoting +them), it is still possible that someone has a case where they cannot use +the Python enums as they are now. To avoid people setting the environment +flag to disable this completely, we implemented a way to select each +combination of enum functions step by step with a specific set of flags. + + +The Possible Enum Flags +----------------------- + +This is the table of all flags used to control the creation of Python enums. + +====================== ===== ====================================================== +Flag Name Value +====================== ===== ====================================================== +ENOPT_OLD_ENUM 0x00 (False) No longer possible since PySide 6.6 +ENOPT_NEW_ENUM 0x01 (True) The default for PySide 6.4, full implementation +ENOPT_INHERIT_INT 0x02 Turn all Enum into IntEnum and Flag into IntFlag +ENOPT_GLOBAL_SHORTCUT 0x04 Re-add shortcuts for global enums +ENOPT_SCOPED_SHORTCUT 0x08 Re-add shortcuts for scoped enums +ENOPT_NO_FAKESHORTCUT 0x10 Don't fake rename (forgiveness mode) +ENOPT_NO_FAKERENAMES 0x20 Don't fake shortcuts (forgiveness mode) +ENOPT_NO_ZERODEFAULT 0x40 Don't use zero default (forgiveness mode) +ENOPT_NO_MISSING 0x80 Don't allow missing values in Enum +====================== ===== ====================================================== + +Such a set of flags can be defined either by the environment variable +``PYSIDE6_OPTION_PYTHON_ENUM`` or set by the Python variable +``sys.pyside6_option_python_enum`` before PySide6 is imported. +The environment variable also supports arbitrary integer expressions +by using ``ast.literal_eval``. + + +ENOPT_OLD_ENUM (0x00) +~~~~~~~~~~~~~~~~~~~~~ + +This option completely disables the new enum implementation. +Even though this is a valid option, we want to avoid it if possible. +The goal is to eventually remove the old implementation. To make this +possible, we have made the individual features of the enum implementation +accessible as flags. This way, if users report problems, we may be able +to provide a temporary solution before extending enum support accordingly. + + +ENOPT_NEW_ENUM (0x01) +~~~~~~~~~~~~~~~~~~~~~ + +In a perfect world, no one would choose anything other than this default +setting. Unfortunately, reality is not always like that. That is why +there are the following flags. + + +The most likely flags needed +---------------------------- + +If there are errors, they are likely to be the following: Either implicit +assumptions are there that require IntEnum, or global enums are used that +unfortunately cannot be replaced with tricks. + + +ENOPT_INHERIT_INT (0x02) +~~~~~~~~~~~~~~~~~~~~~~~~ + +When this flag is set, all ``enum.Enum/enum.Flag`` classes are converted to +``enum.IntEnum/enum.IntFlag``. This solves the most likely compatibility +problem when switching to Python enums. The old Shiboken enums always +inherit from int, but most Python enums do not. + +It was a decision of Python developers not to let enums inherit from int by +default, since no order should be implied. In most cases, inheritance from +int can be avoided, either by using the value property or better by +uplifting: instead of using ``AnEnum.AnInstance.value`` in a function that +expects an int argument, you can also convert the integer to an enumeration +instance after the call by ``AnEnum(int_arg)`` and use that in comparisons. + +However, there are cases where this is not possible, and explicit support in +PySide is simply not available. In those cases, you can use this flag as a +workaround until we have implemented alternatives. + + +ENOPT_GLOBAL_SHORTCUT (0x04) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +At the beginning of the Python enum implementation, we continued to support +the shortcut behavior of Shiboken enums: the enum constants were mirrored +into the enclosing scope. +This was later emulated in the course of forgiveness mode. For enum classes +in a PySide class this works fine, but for enum classes directly on the module +level there is no good way to implement forgiveness. + +It is unlikely that errors are hidden for global enums, because they should +already produce an error during import. But for cases without access to +the source code, you can help yourself with this flag. + +A flag value of 0x6 is likely to solve the majority of problems. + + +Flags for completeness +---------------------- + +The following flags complement the description of Python Enums. +They essentially serve the better understanding of the +implementation and make it fully transparent and customizable. + + +ENOPT_SCOPED_SHORTCUT (0x08) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For completeness, we also supported mirroring scoped enums, although this +has since been replaced by forgiveness mode. If you want to try this, +please also use the ENOPT_NO_FAKESHORTCUT flag (0x10), otherwise the +effect of this flag will remain invisible. + + +ENOPT_NO_FAKERENAMES (0x10) +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Forgiveness mode emulates renaming ``Enum.Flag`` classes back to Shiboken +QFlags structures, which have slightly different names. +So when such a defunct name is used, the system replaces it internally +with the new ``enum.Flag`` structure. Unless special boundary problems +are provoked, this replacement should work. + +To see the effect of this renaming, you can turn it off with this flag. + + +ENOPT_NO_ZERODEFAULT (0x40) +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As part of the forgiveness mode, Python enums can be created by a +parameterless call, although Python enums actually force a parameter +when called. + +The effect can be examined if this flag is set to disable it. + + +ENOPT_NO_MISSING (0x80) +~~~~~~~~~~~~~~~~~~~~~~~ + +There are a few cases where Shiboken enums use missing values. In +``enum.Flag`` structures, this is allowed anyway because we have set the +``FlagBoundary.KEEP`` flag (see ``enum.py``). + +Normal ``enum.Enum`` structures don't have this provision, but the +``enum`` module allows to pass a ``_missing_`` function for customization. + +Our way of dealing with this situation is to create a new fake +``enum.Enum`` class with the same name and a nameless instance, and +pretend with an attribute setting that it has the same type. +The additional instances created in this way are recorded in a class dict +``_sbk_missing_`` in order to preserve their identity. + +You will see the effect of not defining a ``_missing_`` function if you +set this flag. diff --git a/sources/pyside6/doc/developer/extras.rst b/sources/pyside6/doc/developer/extras.rst new file mode 100644 index 000000000..9788b539d --- /dev/null +++ b/sources/pyside6/doc/developer/extras.rst @@ -0,0 +1,55 @@ +Test a wheel +============ + +There is a tool that you can use to test a set of wheels called 'testwheel' but +it's currently in a different repository (``qt/qtqa``): + +- Use ``scripts/packagetesting/testwheel.py`` from the + `qtqa repository <https://code.qt.io/cgit/qt/qtqa.git>`_. + +To test the wheels: + +- Create a virtual environment and activate it. +- Install the dependencies listed on the ``requirements.txt`` file. +- Install all the wheels: ``shiboken6``, ``shiboken6-generator``, + and ``PySide6-Essentials``. +- Run the ``testwheel`` tool. +- Install ``PySide6-Addons`` wheels. +- Run again the ``testwheel`` tool. +- In case you have access to commercial wheels, don't forget the + ``PySide6-M2M`` as well, and re-run the ``testwheel`` tool. + +Build on the command line +========================= + +- Consider using ``build_scripts/qp5_tool.py``. + +Build with address sanitizer (Linux) +==================================== + +ASAN needs to be told to not exit on memory leaks and its library +needs to be pre-loaded. Assuming the library is found +at ``/usr/lib/gcc/x86_64-linux-gnu/11``: + +.. code-block:: bash + + export ASAN_OPTIONS=detect_leaks=0 + export LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/11/libasan.so + python setup.py build [...] --sanitize-address + +De-Virtualize the Python Files +============================== + +The Python files in the Shiboken module are completely virtual, i.E. +they are nowhere existent in the file system for security reasons. + +For debugging purposes or to change something, it might be desirable +to move these files into the normal file system, again. + +- Setting the environment variable "SBK_EMBED" once to false unpacks these + files when PySide6 or shiboken6 are imported. The files are written + into "side-packages/shiboken6/files.dir" and are used from then on. + +- Setting the variable to true removes "files.dir". + +- Without the "SBK_EMBED" variable, the embedding status remains sticky. diff --git a/sources/pyside6/doc/developer/feature-motivation.rst b/sources/pyside6/doc/developer/feature-motivation.rst new file mode 100644 index 000000000..1509ea724 --- /dev/null +++ b/sources/pyside6/doc/developer/feature-motivation.rst @@ -0,0 +1,303 @@ +.. _feature-why: + +Why do we have a __feature__? +============================= + + +History +------- + +In PySide user story PYSIDE-1019, we tested certain ways to +make PySide more pythonic. The first idea was to support some +way to allow for ``snake_case`` function names. + +This feature is possible with relatively low compatibility +problems, because having the same function with different names +would be not so nice, but a possible low-effort solution. + +When going to ``true_property``, things become different. When we +support properties as first class objects instead of getter +and setter functions, we get a conflict, because a function +cannot act as a property (no braces) and be a function at the +same time. + +This consideration led us to the idea: +Features must be selectable per-module. + + +Why are features selectable per-module? +--------------------------------------- + +Suppose you have some pre-existing code. Maybe you use some downloaded +code or you generated an interface file. When you now decide to +use a feature, you don't want all this existing stuff to become +incorrect. By using the statement + +.. code-block:: python + + from __feature__ import ... + +you declare that this module uses some feature. Other modules will not +be influenced by this decision and can stay unchanged. + + +Why dunder, and why not __future__? +----------------------------------- + +Especially in Python 2, but in a few cases also in Python 3, there is +the future statement + +.. code-block:: python + + from __future__ import ... + +That is a statement that can only appear at the beginning of a module, +and it switches how the Python parser works. + +Our first idea was to mimick this behavior for PySide, although we are +a bit cheating: The feature statement is not a syntactical construct, +and we cannot easily forbid that it is in the middle of a module. + +We then realized that the intention of Python's ``__future__`` import and +PySide's ``__feature__`` import are different: While Python implies by +``__future__`` some improvement, we do not want to associate with +``__feature__``. We simply think that some users who come from Python may +like our features, while others are used to the C++ convention and +consider something that deviates from the Qt documentation as drawback. + +The intention to use the ``from __feature__ import ...`` notation was the hope that +people see the similarity to Python's ``__future__`` statement and put that import +at the beginning of a module to make it very visible that this module +has some special global differences. + + +The snake_case feature +====================== + +By using the statement + +.. code-block:: python + + from __feature__ import snake_case + +all methods of all classes used in this module are changing their name. + +The algorithm to change names is this: + +* if the name has less than 3 chars, or +* if two upper chars are adjacent, or +* if the name starts with ``gl`` (which marks OpenGL), +* the name is returned unchanged. Otherwise +* a single upper char ``C`` is replaced by ``_c`` + + +The true_property feature +========================= + +By using the statement + +.. code-block:: python + + from __feature__ import true_property + +all methods of all classes used in this module which are declared in the Qt +documentation as property become real properties in Python. + +This feature is incompatible with the past and cannot coexist; it is +the reason why the feature idea was developed at all. + + +Normal Properties +----------------- + +Normal properties have the same name as before: + +.. code-block:: python + + QtWidgets.QLabel().color() + +becomes as property + +.. code-block:: python + + QtWidgets.QLabel().color + +When there is also a setter method, + +.. code-block:: python + + QtWidgets.QLabel().setColor(value) + +becomes as property + +.. code-block:: python + + QtWidgets.QLabel().color = value + +Normal properties swallow the getter and setter functions and replace +them by the property object. + + +Special Properties +------------------ + +Special properties are those with non-standard names. + +.. code-block:: python + + QtWidgets.QLabel().size() + +becomes as property + +.. code-block:: python + + QtWidgets.QLabel().size + +But here we have no setSize function, but + +.. code-block:: python + + QtWidgets.QLabel().resize(value) + +which becomes as property + +.. code-block:: python + + QtWidgets.QLabel().size = value + +In that case, the setter does not become swallowed, because so many +people are used to the ``resize`` function. + + +Class properties +---------------- + +It should be mentioned that we not only support regular properties +as they are known from Python. There is also the concept of class +properties which always call their getter and setter: + +A regular property like the aforementioned ``QtWidgets.QLabel`` has +this visibility: + +.. code-block:: python + + >>> QtWidgets.QLabel.size + <property object at 0x113a23540> + >>> QtWidgets.QLabel().size + PySide6.QtCore.QSize(640, 480) + +A class property instead is also evaluated without requiring an instance: + +.. code-block:: python + + >>> QtWidgets.QApplication.windowIcon + <PySide6.QtGui.QIcon(null) at 0x113a211c0> + +You can only inspect it if you go directly to the right class dict: + +.. code-block:: python + + >>> QtGui.QGuiApplication.__dict__["windowIcon"] + <PySide6.PyClassProperty object at 0x114fc5270> + + +About Property Completeness +--------------------------- + +There are many properties where the Python programmer agrees that these +functions should be properties, but a few are not properties, like + +.. code-block:: python + + >>> QtWidgets.QMainWindow.centralWidget + <method 'centralWidget' of 'PySide6.QtWidgets.QMainWindow' objects> + +We are currently discussing if we should correct these rare cases, as they +are probably only omissions. Having to memorize the missing properties +seems to be quite cumbersome, and instead of looking all properties up in +the Qt documentation, it would be easier to add all properties that +should be properties and are obviously missing. + + +Name Clashes and Solution +------------------------- + +There are some rare cases where a property already exists as a function, +either with multiple signatures or having parameters. +This is not very nice in C++ as well, but for Python this is forbidden. +Example: + +.. code-block:: python + + >>> from PySide6 import * + >>> from PySide6.support.signature import get_signature + >>> import pprint + >>> pprint.pprint(get_signature(QtCore.QTimer.singleShot)) + [<Signature (arg__1: int, arg__2: Callable) -> None>, + <Signature (msec: int, receiver: PySide6.QtCore.QObject, member: bytes) -> None>, + <Signature (msec: int, timerType: PySide6.QtCore.Qt.TimerType, + receiver: PySide6.QtCore.QObject, member: bytes) -> None>] + +When creating this property, we respect the existing function and use a slightly +different name for the property by appending an underscore. + +.. code-block:: python + + >>> from __feature__ import true_property + >>> QtCore.QTimer.singleShot_ + <property object at 0x118e5f8b0> + +We hope that these clashes can be removed in future Qt versions. + + +The __feature__ import +====================== + +The implementation of ``from __feature__ import ...`` is built by a slight +modification of the ``__import__`` builtin. We made that explicit by assigning +variables in the builtin module. This modification takes place at |project| +import time: + +* The original function in ``__import__`` is kept in ``__orig_import__``. +* The new function is in ``__feature_import__`` and assigned to ``__import__``. + +This function calls the Python function ``PySide6.support.__feature__.feature_import`` +first, and falls back to ``__orig_import__`` if feature import is not applicable. + + +Overriding __import__ +--------------------- + +This is not recommended. Import modifications should be done using import hooks, +see the Python documentation on `Import-Hooks`_. + +If you would like to modify ``__import__`` anyway without destroying the features, +please override just the ``__orig_import__`` function. + + +IDEs and Modifying Python stub files +------------------------------------ + +|project| comes with pre-generated ``.pyi`` stub files in the same location as +the binary module. For instance, in the site-packages directory, you can find +a ``QtCore.pyi`` file next to ``QtCore.abi3.so`` or ``QtCore.pyd`` on Windows. + +When using ``__feature__`` often with common IDEs, you may want to provide +a feature-aware version of ``.pyi`` files to get a correct display. The simplest +way to change them all in-place is the command: + +.. code-block:: bash + + pyside6-genpyi all --feature snake_case true_property + + +Using __feature__ with UIC files +-------------------------------- + +Features can be freely used together with generated UIC files. The UIC files +are _not_ converted, intentionally. Mixing them with feature selections in other +Python modules should always work, because switching will happen as needed, selected +by the currently active module. (Please report to us if this fails for an example) + + +.. _`Import-Hooks`: https://docs.python.org/3/reference/import.html#import-hooks diff --git a/sources/pyside6/doc/developer/index.rst b/sources/pyside6/doc/developer/index.rst new file mode 100644 index 000000000..92c84259d --- /dev/null +++ b/sources/pyside6/doc/developer/index.rst @@ -0,0 +1,35 @@ +.. _developer-notes: + +Developer Notes +=============== + +Developing |project| requires people to understand different processes +and steps that need to be taken into account when dealing with topics +related to modules, bindings, examples, and more. + +Development Topics +------------------ + +.. toctree:: + :maxdepth: 2 + + add_module.rst + add_port_example.rst + add_tool.rst + documentation.rst + adapt_qt.rst + extras.rst + +Implementation details +---------------------- + +Complementary, here you can find the reasoning and motivation for the +many features and implementation details that the project has: + +.. toctree:: + :maxdepth: 2 + + enumfeatures_doc.rst + limited_api.rst + signature_doc.rst + feature-motivation.rst diff --git a/sources/pyside6/doc/developer/limited_api.rst b/sources/pyside6/doc/developer/limited_api.rst new file mode 100644 index 000000000..44d3faad2 --- /dev/null +++ b/sources/pyside6/doc/developer/limited_api.rst @@ -0,0 +1,703 @@ +The Transition To The Limited Python API (PEP384) +================================================= + + +Foreword +-------- + +Python supports a limited API that restricts access to certain structures. +Besides eliminating whole modules and all functions and macros which names +start with an +underscore, the most drastic restriction is the removal of normal type object +declarations. + +For details about the eliminated modules and functions, please see the +`PEP 384`_ page for reference. + + +.. _`PEP 384`: https://www.python.org/dev/peps/pep-0384/ + + + +Changed Modules +--------------- + +All changed module's include files are listed with the changed functions here. +As a general rule, it was tried to keep the changes to a minimum diff. +Macros which are not available were changed to functions with the same name +if possible. Completely removed names ``Py{name}`` were re-implemented as ``Pep{name}``. + + +memoryobject.h +~~~~~~~~~~~~~~ + +The buffer protocol was completely removed. We redefined all the structures +and methods, because PySide uses that. This is an exception to the limited API +that we have to check ourselves. The code is extracted in bufferprocs_py37.h . +This is related to the following: + + +abstract.h +~~~~~~~~~~ + +This belongs to the buffer protocol like memoryobject.h . +As replacement for ``Py_buffer`` we defined ``Pep_buffer`` and several other +internal macros. + +The version is checked by hand, and the version number must be updated only +if the implementation does not change. Otherwise, we need to write version +dependent code paths. + +It is questionable if it is worthwhile to continue using the buffer protocol +or if we should try to get rid of ``Pep_buffer``, completely. + + +pydebug.h +~~~~~~~~~ + +We have no direct access to ``Py_VerboseFlag`` because debugging is not +supported. We redefined it as macro ``Py_VerboseFlag`` which calls ``Pep_VerboseFlag``. + + +unicodeobject.h +~~~~~~~~~~~~~~~ + +The macro ``PyUnicode_GET_SIZE`` was removed and replaced by ``PepUnicode_GetLength`` +which evaluates to ``PyUnicode_GetSize`` for Python 2 and ``PyUnicode_GetLength`` for Python 3. +Since Python 3.3, ``PyUnicode_GetSize`` would have the bad side effect of requiring the GIL! + +Function ``_PyUnicode_AsString`` is unavailable and was replaced by a macro +that calls ``_PepUnicode_AsString``. The implementation was a bit involved, +and it would be better to change the code and replace this function. + + +bytesobject.h +~~~~~~~~~~~~~ + +The macros ``PyBytes_AS_STRING`` and ``PyBytes_GET_SIZE`` were redefined to call +the according functions. + + +floatobject.h +~~~~~~~~~~~~~ + +``PyFloat_AS_DOUBLE`` now calls ``PyFloat_AsDouble``. + + +tupleobject.h +~~~~~~~~~~~~~ + +``PyTuple_GET_ITEM``, ``PyTuple_SET_ITEM`` and ``PyTuple_GET_SIZE`` were redefined as +function calls. + + +listobject.h +~~~~~~~~~~~~ + +``PyList_GET_ITEM``, ``PyList_SET_ITEM`` and ``PyList_GET_SIZE`` were redefined as +function calls. + + +dictobject.h +~~~~~~~~~~~~ + +``PyDict_GetItem`` also exists in a ``PyDict_GetItemWithError`` version that does +not suppress errors. This suppression has the side effect of touching global +structures. This function exists in Python 2 only since Python 2.7.12 and has +a different name. We simply implemented the function. +Needed to avoid the GIL when accessing dictionaries. + + +methodobject.h +~~~~~~~~~~~~~~ + +``PyCFunction_GET_FUNCTION``, ``PyCFunction_GET_SELF`` and ``PyCFunction_GET_FLAGS`` +were redefined as function calls. + +Direct access to the methoddef structure is not available, and we defined +``PepCFunction_GET_NAMESTR`` as accessor for name strings. + + +pythonrun.h +~~~~~~~~~~~ + +The simple function ``PyRun_String`` is not available. It was re-implemented +in a simplified version for the signature module. + + +funcobject.h +~~~~~~~~~~~~ + +The definitions of funcobject.h are completely missing, although there +are extra ``#ifdef`` conditional defines inside, too. This suggests that the exclusion +was unintended. + +We therefore redefined ``PyFunctionObject`` as an opaque type. + +The missing macro ``PyFunction_Check`` was defined, and the macro +``PyFunction_GET_CODE`` calls the according function. + +There is no equivalent for function name access, therefore we introduced +``PepFunction_GetName`` either as a function or as a macro. + +*TODO: We should fix funcobject.h* + + +classobject.h +~~~~~~~~~~~~~ + +Classobject is also completely not imported, instead of defining an opaque type. + +We defined the missing functions ``PyMethod_New``, ``PyMethod_Function`` and +``PyMethod_Self`` and also redefined ``PyMethod_GET_SELF`` and +``PyMethod_GET_FUNCTION`` as calls to these functions. + +*TODO: We should fix classobject.h* + + +code.h +~~~~~~ + +The whole code.c code is gone, although it may make sense to +define some minimum accessibility. This will be clarified on +`Python-Dev`_. We needed access to code objects and defined the missing +PepCode_GET_FLAGS and PepCode_GET_ARGCOUNT either as function or macro. +We further added the missing flags, although few are used: + +``CO_OPTIMIZED`` ``CO_NEWLOCALS`` ``CO_VARARGS`` ``CO_VARKEYWORDS`` ``CO_NESTED`` +``CO_GENERATOR`` + +*TODO: We should maybe fix code.h* + +.. _`Python-Dev`: https://mail.python.org/mailman/listinfo/python-dev + +datetime.h +~~~~~~~~~~ + +The DateTime module is explicitly not included in the limited API. +We defined all the needed functions but called them via Python instead +of direct call macros. This has a slight performance impact. + +The performance could be easily improved by providing an interface +that fetches all attributes at once, instead of going through the object +protocol every time. + +The re-defined macros and methods are:: + + PyDateTime_GET_YEAR + PyDateTime_GET_MONTH + PyDateTime_GET_DAY + PyDateTime_DATE_GET_HOUR + PyDateTime_DATE_GET_MINUTE + PyDateTime_DATE_GET_SECOND + PyDateTime_DATE_GET_MICROSECOND + PyDateTime_DATE_GET_FOLD + PyDateTime_TIME_GET_HOUR + PyDateTime_TIME_GET_MINUTE + PyDateTime_TIME_GET_SECOND + PyDateTime_TIME_GET_MICROSECOND + PyDateTime_TIME_GET_FOLD + + PyDate_Check + PyDateTime_Check + PyTime_Check + + PyDate_FromDate + PyDateTime_FromDateAndTime + PyTime_FromTime + +*XXX: We should maybe provide an optimized interface to datetime* + + +object.h +~~~~~~~~ + +The file object.h contains the ``PyTypeObject`` structure, which is supposed +to be completely opaque. All access to types should be done through +``PyType_GetSlot`` calls. Due to bugs and deficiencies in the limited API +implementation, it was not possible to do that. Instead, we have defined +a simplified structure for ``PyTypeObject`` that has only the fields that +are used in PySide. + +We will explain later why and how this was done. Here is the reduced +structure:: + + typedef struct _typeobject { + PyVarObject ob_base; + const char *tp_name; + Py_ssize_t tp_basicsize; + void *X03; // Py_ssize_t tp_itemsize; + void *X04; // destructor tp_dealloc; + void *X05; // printfunc tp_print; + void *X06; // getattrfunc tp_getattr; + void *X07; // setattrfunc tp_setattr; + void *X08; // PyAsyncMethods *tp_as_async; + void *X09; // reprfunc tp_repr; + void *X10; // PyNumberMethods *tp_as_number; + void *X11; // PySequenceMethods *tp_as_sequence; + void *X12; // PyMappingMethods *tp_as_mapping; + void *X13; // hashfunc tp_hash; + ternaryfunc tp_call; + reprfunc tp_str; + void *X16; // getattrofunc tp_getattro; + void *X17; // setattrofunc tp_setattro; + void *X18; // PyBufferProcs *tp_as_buffer; + void *X19; // unsigned long tp_flags; + void *X20; // const char *tp_doc; + traverseproc tp_traverse; + inquiry tp_clear; + void *X23; // richcmpfunc tp_richcompare; + Py_ssize_t tp_weaklistoffset; + void *X25; // getiterfunc tp_iter; + void *X26; // iternextfunc tp_iternext; + struct PyMethodDef *tp_methods; + void *X28; // struct PyMemberDef *tp_members; + void *X29; // struct PyGetSetDef *tp_getset; + struct _typeobject *tp_base; + PyObject *tp_dict; + descrgetfunc tp_descr_get; + void *X33; // descrsetfunc tp_descr_set; + Py_ssize_t tp_dictoffset; + initproc tp_init; + allocfunc tp_alloc; + newfunc tp_new; + freefunc tp_free; + inquiry tp_is_gc; /* For PyObject_IS_GC */ + PyObject *tp_bases; + PyObject *tp_mro; /* method resolution order */ + } PyTypeObject; + +Function ``PyIndex_Check`` had to be defined in an unwanted way due to +a Python issue. See file pep384_issue33738.cpp . + +There are extension structures which have been isolated as special macros that +dynamically compute the right offsets of the extended type structures: + +* ``PepType_SOTP`` for ``SbkObjectTypePrivate`` +* ``PepType_SETP`` for ``SbkEnumTypePrivate`` +* ``PepType_PFTP`` for ``PySideQFlagsTypePrivate`` + +How these extension structures are used can best be seen by searching +``PepType_{four}`` in the source. + +Due to the new heaptype interface, the names of certain types contain +now the module name in the ``tp_name`` field. To have a compatible way +to access simple type names as C string, ``PepType_GetNameStr`` has been +written that skips over dotted name parts. + +Finally, the function ``_PyObject_Dump`` was excluded from the limited API. +This is a useful debugging aid that we always want to have available, +so it is added back, again. Anyway, we did not reimplement it, and so +Windows is not supported. +Therefore, a forgotten debugging call of this functions will break COIN. :-) + + +Using The New Type API +---------------------- + +After converting everything but the object.h file, we were a little +bit shocked: it suddenly was clear that we would have no more +access to type objects, and even more scary that all types which we +use have to be heap types, only! + +For PySide with its intense use of heap type extensions in various +flavors, the situation looked quite unsolvable. In the end, it was +nicely solved, but it took almost 3.5 months to get that right. + +Before we see how this is done, we will explain the differences +between the APIs and their consequences. + + +The Interface +~~~~~~~~~~~~~ + +The old type API of Python knows static types and heap types. +Static types are written down as a declaration of a ``PyTypeObject`` +structure with all its fields filled in. Here is for example +the definition of the Python type ``object`` (Python 3.6):: + + PyTypeObject PyBaseObject_Type = { + PyVarObject_HEAD_INIT(&PyType_Type, 0) + "object", /* tp_name */ + sizeof(PyObject), /* tp_basicsize */ + 0, /* tp_itemsize */ + object_dealloc, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_reserved */ + object_repr, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + (hashfunc)_Py_HashPointer, /* tp_hash */ + 0, /* tp_call */ + object_str, /* tp_str */ + PyObject_GenericGetAttr, /* tp_getattro */ + PyObject_GenericSetAttr, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ + PyDoc_STR("object()\n--\n\nThe most base type"), /* tp_doc */ + 0, /* tp_traverse */ + 0, /* tp_clear */ + object_richcompare, /* tp_richcompare */ + 0, /* tp_weaklistoffset */ + 0, /* tp_iter */ + 0, /* tp_iternext */ + object_methods, /* tp_methods */ + 0, /* tp_members */ + object_getsets, /* tp_getset */ + 0, /* tp_base */ + 0, /* tp_dict */ + 0, /* tp_descr_get */ + 0, /* tp_descr_set */ + 0, /* tp_dictoffset */ + object_init, /* tp_init */ + PyType_GenericAlloc, /* tp_alloc */ + object_new, /* tp_new */ + PyObject_Del, /* tp_free */ + }; + +We can write the same structure in form of a ``PyType_Spec`` structure, +and there is even an incomplete tool *abitype.py* that does this conversion +for us. With a few corrections, the result looks like this:: + + static PyType_Slot PyBaseObject_Type_slots[] = { + {Py_tp_dealloc, (void *)object_dealloc}, + {Py_tp_repr, (void *)object_repr}, + {Py_tp_hash, (void *)_Py_HashPointer}, + {Py_tp_str, (void *)object_str}, + {Py_tp_getattro, (void *)PyObject_GenericGetAttr}, + {Py_tp_setattro, (void *)PyObject_GenericSetAttr}, + {Py_tp_richcompare, (void *)object_richcompare}, + {Py_tp_methods, (void *)object_methods}, + {Py_tp_getset, (void *)object_getsets}, + {Py_tp_init, (void *)object_init}, + {Py_tp_alloc, (void *)PyType_GenericAlloc}, + {Py_tp_new, (void *)object_new}, + {Py_tp_free, (void *)PyObject_Del}, + {0, 0}, + }; + static PyType_Spec PyBaseObject_Type_spec = { + "object", + sizeof(PyObject), + 0, + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, + PyBaseObject_Type_slots, + }; + +This new structure is almost compatible with the old one, but there +are some subtle differences. + +* The new types are generated in one step + +This seems to be no problem, but it was very much, due to the way the +types were built in PySide. Types were assembled piece by piece, and +finally the ``PyType_Ready`` function was called. + +With the new API, ``PyType_Ready`` is called already at the end of +``PyType_FromSpec``, and that meant that the logic of type creation became +completely turned upside down. + +* The new types are always heaptypes + +With the new type creation functions, it is no longer possible to +create "normal" types. Instead, they all have to be allocated on the +heap and garbage collected. The user should normally not recognize this. +But type creation is more constrained, and you cannot create a subtype +if the ``Py_TPFLAGS_BASETYPE`` is not set. This constraint was already +violated by PySide and needed a quite profound fix. + +* The new types always need a module + +While this is not a problem per se, the above new type spec will not create +a usable new type, but complain with:: + + DeprecationWarning: builtin type object has no __module__ attribute + +But there are more problems: + +* The new types have unexpected defaults + +When fields are empty, you would usually assume that they stay empty. +There are just a few corrections that ``PyType_Ready`` will do to a type. + +But there is the following clause in ``PyType_FromSpec`` that can give you +many headaches:: + + if (type->tp_dealloc == NULL) { + /* It's a heap type, so needs the heap types' dealloc. + subtype_dealloc will call the base type's tp_dealloc, if + necessary. */ + type->tp_dealloc = subtype_dealloc; + } + +In fact, before the move to the new API, the ``PyType_Ready`` function +filled empty ``tp_dealloc`` fields with ``object_dealloc``. And the code +that has been written with that in mind now becomes pretty wrong if suddenly +``subtype_dealloc`` is used. + +The way out was to explicitly provide an ``object_dealloc`` function. +This would then again impose a problem, because ``object_dealloc`` is not +public. Writing our own version is easy, but it again needs access to +type objects. But fortunately, we have broken this rule, already... + + +* The new types are only partially allocated + +The structures used in ``PyType_FromSpec`` are almost all allocated, +only the name field is static. This is no problem for types which are +statically created once. But if you want to parameterize things and +create multiple types with a single slots and spec definition, the name +field that is used for tp_name must be allocated dynamically. +This is misleading, since all the slots already are copies. + +* The new types don't support special offsets + +The special fields ``tp_weaklistoffset`` and ``tp_dictoffset`` are not supported +by ``PyType_FromSpec``. Unfortunately the documentation does not tell you +if you are allowed to set these fields manually after creating the type or not. +We finally did it and it worked, but we are not sure about correctness. + +See basewrapper.cpp function ``SbkObject_TypeF()`` as the only reference to +these fields in PySide. This single reference is absolutely necessary and +very important, since all derived types invisibly inherit these two fields. + + +Future Versions Of The Limited API +---------------------------------- + +As we have seen, the current version of the limited API does a bit of +cheating, because it uses parts of the data structure that should be +an opaque type. At the moment, this works fine because the data is +still way more compatible as it could be. + +But what if this is changed in the future? + +We know that the data structures are stable until Python 3.8 comes out. +Until then, the small bugs and omissions will hopefully all be solved. +Then it will be possible to replace the current small tricks by calls +to ``PyType_GetSlot`` in the way things should be. + +At the very moment when the current assumptions about the data structure +are no longer true, we will rewrite the direct attribute access with +calls to ``PyType_GetSlot``. After that, no more changes will be necessary. + + +Appendix A: The Transition To Simpler Types +------------------------------------------- + +After all code had been converted to the limited API, there was a +remaining problem with the ``PyHeapTypeObject``. + +Why a problem? Well, all the type structures in shiboken use +special extra fields at the end of the heap type object. This +currently enforces extra knowledge at compile time about how large the +heap type object is. In a clean implementation, we would only use +the ``PyTypeObject`` itself and access the fields *behind* the type +by a pointer that is computed at runtime. + + +Restricted PyTypeObject +~~~~~~~~~~~~~~~~~~~~~~~ + +Before we are going into details, let us motivate the existence of +the restricted ``PyTypeObject``: + +Originally, we wanted to use ``PyTypeObject`` as an opaque type and +restrict ourselves to only use the access function ``PyType_GetSlot``. +This function allows access to all fields which are supported by +the limited API. + +But this is a restriction, because we get no access to ``tp_dict``, +which we need to support the signature extension. But we can work +around that. + +The real restriction is that ``PyType_GetSlot`` only works for heap +types. This makes the function quite useless, because we have +no access to ``PyType_Type``, which is the most important type ``type`` +in Python. We need that for instance to compute the size of +``PyHeapTypeObject`` dynamically. + +With much effort, it is possible to clone ``PyType_Type`` as a heap +type. But due to a bug in the Pep 384 support, we need +access to the ``nb_index`` field of a normal type. Cloning does not +help because ``PyNumberMethods`` fields are *not* inherited. + +After we realized this dead end, we changed concept and did not +use ``PyType_GetSlot`` at all (except in function ``copyNumberMethods``), +but created a restricted ``PyTypeObject`` with only those fields +defined that are needed in PySide. + +Is this breakage of the limited API? I don't think so. A special +function runs on program startup that checks the correct position +of the fields of ``PyTypeObject``, although a change in those fields is +more than unlikely. +The really crucial thing is to no longer use ``PyHeapTypeObject`` +explicitly because that *does* change its layout over time. + + +Diversification +~~~~~~~~~~~~~~~ + +There were multiple ``Sbk{something}`` structures which all used a "d" field +for their private data. This made it not easy to find the right +fields when switching between objects and types:: + + struct LIBSHIBOKEN_API SbkObject + { + PyObject_HEAD + PyObject *ob_dict; + PyObject *weakreflist; + SbkObjectPrivate *d; + }; + + struct LIBSHIBOKEN_API SbkObjectType + { + PyHeapTypeObject super; + SbkObjectTypePrivate *d; + }; + +The first step was to rename the SbkObjectTypePrivate part from "d" to +"sotp". It was chosen to be short but easy to remember as abbreviation +of "SbkObjectTypePrivate", leading to:: + + struct LIBSHIBOKEN_API SbkObjectType + { + PyHeapTypeObject super; + SbkObjectTypePrivate *sotp; + }; + +After renaming, it was easier to do the following transformations. + + +Abstraction +~~~~~~~~~~~ + +After renaming the type extension pointers to ``sotp``, I replaced +them by function-like macros which did the special access *behind* +the types, instead of those explicit fields. For instance, the +expression:: + + type->sotp->converter + +became:: + + PepType_SOTP(type)->converter + +The macro expansion can be seen here:: + + #define PepHeapType_SIZE \ + (reinterpret_cast<PyTypeObject *>(&PyType_Type)->tp_basicsize) + + #define _genericTypeExtender(etype) \ + (reinterpret_cast<char *>(etype) + PepHeapType_SIZE) + + #define PepType_SOTP(etype) \ + (*reinterpret_cast<SbkObjectTypePrivate **>(_genericTypeExtender(etype))) + +This looks complicated, but in the end there is only a single new +indirection via ``PyType_Type``, which happens at runtime. This is the +key to fulfil what Pep 384 wants to achieve: *No more version-dependent fields*. + + +Simplification +~~~~~~~~~~~~~~ + +After all type extension fields were replaced by macro calls, we +could remove the following version dependent re-definition of ``PyHeapTypeObject`` +:: + + typedef struct _pyheaptypeobject { + union { + PyTypeObject ht_type; + void *opaque[PY_HEAPTYPE_SIZE]; + }; + } PyHeapTypeObject; + +, and the version dependent structure:: + + struct LIBSHIBOKEN_API SbkObjectType + { + PyHeapTypeObject super; + SbkObjectTypePrivate *sotp; + }; + +could be removed. SbkObjectType remains as a (deprecated) +type alias to PyTypeObject. + + +Appendix B: Verification Of PyTypeObject +---------------------------------------- + +We have introduced a limited PyTypeObject in the same place +as the original PyTypeObject, and now we need to prove that +we are allowed to do so. + +When using the limited API as intended, then types are completely +opaque, and access is only through ``PyType_FromSpec`` and (from +version 3.5 upwards) through ``PyType_GetSlot``. + +Python then uses all the slot definitions in the type description +and produces a regular heap type object. + + +Unused Information +~~~~~~~~~~~~~~~~~~ + +We know many things about types that are not explicitly said, +but they are inherently clear: + +(a) The basic structure of a type is always the same, regardless + if it is a static type or a heap type. + +(b) types are evolving very slowly, and a field is never replaced + by another field with different semantics. + +Inherent rule (a) gives us the following information: If we calculate +the offsets of the basic fields, then this info is also usable for non-heap +types. + +The validation checks if rule (b) is still valid. + + +How it Works +~~~~~~~~~~~~ + +The basic idea of the validation is to produce a new type using +``PyType_FromSpec`` and to see where in the type structure these fields +show up. So we build a ``PyType_Slot`` structure with all the fields we +are using and make sure that these values are all unique in the +type. + +Most fields are not interrogated by ``PyType_FromSpec``, and so we +simply used some numeric value. Some fields are interpreted, like +``tp_members``. This field must really be a ``PyMemberDef``. And there are +``tp_base`` and ``tp_bases`` which have to be type objects and lists +thereof. It was easiest to not produce these fields from scratch +but use them from the ``type`` object ``PyType_Type``. + +Then one would think to write a function that searches the known +values in the opaque type structure. + +But we can do better and use optimistically the observation (b): +We simply use the restricted ``PyTypeObject`` structure and assume that +every field lands exactly where we are awaiting it. + +And that is the whole proof: If we find all the disjoint values at +the places where we expect them, then verification is done. + + +About ``tp_dict`` +~~~~~~~~~~~~~~~~~ + +One word about the ``tp_dict`` field: This field is a bit special in +the proof, since it does not appear in the spec and cannot easily +be checked by ``type.__dict__`` because that creates a *dictproxy* +object. So how do we prove that is really the right dict? + +We have to create that ``PyMethodDef`` structure anyway, and instead of +leaving it empty, we insert a dummy function. Then we ask the +``tp_dict`` field if it has the awaited object in it, and that's it! + +#EOT diff --git a/sources/pyside6/doc/developer/signature_doc.rst b/sources/pyside6/doc/developer/signature_doc.rst new file mode 100644 index 000000000..a6c703dab --- /dev/null +++ b/sources/pyside6/doc/developer/signature_doc.rst @@ -0,0 +1,361 @@ +.. _signature-extension: + +The signature C extension +========================= + +This module is a C extension for CPython 3.5 and up, and CPython 2.7. +Its purpose is to provide support for the ``__signature__`` attribute +of builtin PyCFunction objects. + + +Short Introduction to the Topic +------------------------------- + +Beginning with CPython 3.5, Python functions began to grow a ``__signature__`` +attribute for normal Python functions. This is totally optional and just +a nice-to-have feature in Python. + +PySide, on the other hand, could use ``__signature__`` very much, because the +typing info for the 15000+ PySide functions is really missing, and it +would be nice to have this info directly available. + + +The Idea to Support Signatures +------------------------------ + +We want to have an additional ``__signature__`` attribute in all PySide +methods, without changing lots of generated code. +Therefore, we did not change any of the existing data structures, +but supported the new attribute by a global dictionary. + +When the ``__signature__`` property is requested, a method is called that +does a lookup in the global dict. This is a flexible approach with little impact +to the rest of the project. It has very limited overhead compared to direct +attribute access, but for the need of a signature access from time to time, +this is an adequate compromise. + + +How this Code Works +~~~~~~~~~~~~~~~~~~~ + +Signatures are supported for regular Python functions, only. Creating signatures +for ``PyCFunction`` objects would require quite some extra effort in Python. + +Fortunately, we found this special *stealth* technique, that saves us most of the +needed effort: + +The basic idea is to create a dummy Python function with **varnames**, **defaults** +and **annotations** properties, and then to use the inspect +module to create a signature object. This object is returned as the computed +result of the ``__signature__`` attribute of the real ``PyCFunction`` object. + +There is one thing that really changes Python a bit: + +* We added the ``__signature__`` attribute to every function. + +That is a little change to Python that does not harm, but it saves us +tons of code, that was needed in the early versions of the module. + +The internal work is done in two steps: + +* All functions of a class get the *signature text* when the module is imported. + This is only a very small overhead added to the startup time. It is a single + string for each whole class. +* The actual signature object is created later, when the attribute is really + requested. Signatures are cached and only created on first access. + +Example: + +The ``PyCFunction`` ``QtWidgets.QApplication.palette`` is interrogated for its +signature. That means ``pyside_sm_get___signature__()`` is called. +It calls ``GetSignature_Function`` which returns the signature if it is found. + + +Why this Code is Fast +~~~~~~~~~~~~~~~~~~~~~ + +It costs a little time (maybe 6 seconds) to run through every single signature +object, since these are more than 25000 Python objects. But all the signature +objects will be rarely accessed but in special applications. +The normal case are only a few accesses, and these are working pretty fast. + +The key to make this signature module fast is to avoid computation as much as +possible. When no signature objects are used, then almost no time is lost in +initialization. Only the above mentioned strings and some support modules are +additionally loaded on ``import PySide6``. +When it comes to signature usage, then late initialization is used and cached. +This technique is also known as *full laziness* in haskell. + +There are actually two locations where late initialization occurs: + +* ``dict`` can be no dict but a tuple. That is the initial argument tuple that + was saved by ``PySide_BuildSignatureArgs`` at module load time. + If so, then ``pyside_type_init`` in parser.py will be called, + which parses the string and creates the dict. +* ``props`` can be empty. Then ``create_signature`` in loader.py + is called, which uses a dummy function to produce a signature instance + with the inspect module. + +The initialization that is always done is just two dictionary writes +per class, and we have about 1000 classes. +To measure the additional overhead, we have simulated what happens +when ``from PySide6 import *`` is performed. +It turned out that the overhead is below 0.5 ms. + + +The Signature Package Structure +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The C++ code involved with the signature module is completely in the file +shiboken6/libshiboken/signature.cpp . All other functionality is implemented in +the ``signature`` Python package. It has the following structure:: + + sources/shiboken6/shibokenmodule/files.dir/shibokensupport + ├── __init__.py + ├── feature.py + ├── fix-complaints.py + ├── shibokensupport.pyproject + └── signature + ├── PSF-3.7.0.txt + ├── __init__.py + ├── errorhandler.py + ├── importhandler.py + ├── layout.py + ├── lib + │ ├── __init__.py + │ ├── enum_sig.py + │ ├── pyi_generator.py + │ └── tool.py + ├── loader.py + ├── mapping.py + ├── parser.py + └── qt_attribution.json + +Really important are the **parser**, **mapping**, **errorhandler**, **enum_sig**, +**layout** and **loader** modules. The rest is needed to create Python 2 compatibility +or be compatible with embedding and installers. + + +**loader.py** + This module assembles and imports the ``inspect`` module, and then exports the + ``create_signature`` function. This function takes a fake function and some + attributes and builds a ``__signature__`` object with the inspect module. + + +**parser.py** + This module takes a class signatures string from C++ and parses it into the + needed properties for the ``create_signature`` function. Its entry point is the + ``pyside_type_init`` function, which is called from the C module via ``loader.py``. + + +**mapping.py** + The purpose of the mapping module is maintaining a list of replacement strings + that map from the *signature text* in C to the property strings that Python + needs. A lot of mappings are resolved by rather complex expressions in ``parser.py``, + but a few hundred cases are better to spell explicitly, here. + +**errorhandler.py** + Since ``Qt For Python 5.12``, we no longer use the builtin type error messages from C++. + Instead, we get much better results with the signature module. At the same time, + this enforced supporting shiboken as well, and the signature module was no longer + optional. + +**enum_sig.py** + The diverse applications of the signature module all needed to iterate over modules, + classes and functions. In order to centralize this enumeration, the process has + been factored out as a context manager. The user has only to supply functions + that do the actual formatting. + + See for example the .pyi generator ``pyside6/PySide6/support/generate_pyi.py``. + +**layout.py** + As more applications used the signature module, different formatting of signatures + was needed. To support that, we created the function ``create_signature``, which + has a parameter to choose from some predefined layouts. + + +**typing27.py** + Python 2 has no typing module at all. This is a backport of the minimum that is needed. + + +**backport_inspect.py** + Python 2 has an inspect module, but lacks the signature functions, completely. + This module adds the missing functionality, which is merged at runtime into + the inspect module. + + +Multiple Arities +~~~~~~~~~~~~~~~~ + +One aspect that was ignored so far was *multiple arities*: How to handle it when +a function has more than one signature? + +I did not find any note on how multiple signatures should be treated in Python, +but this simple rules seem to work well: + +* If there is a list, then it is a multi-signature. +* Otherwise, it is a simple signature. + + +Impacts of The Signature Module +------------------------------- + +The signature module has a number of impacts to other PySide modules, which were +created as a consequence of its existence, and there will be a few more in the +future: + + +existence_test.py +~~~~~~~~~~~~~~~~~ + +The file ``pyside6/tests/registry/existence_test.py`` was written using the +signatures from the signatures module. The idea is that there are some 15000 +functions with a certain signature. + +These functions should not get lost by some bad check-in. Therefore, a list +of all existing signatures is kept as a module that assembles a +dictionary. The function existence is checked, and also the exact arity. + +This module exists for every PySide release and every platform. The initial +module is generated once and saved as ``exists_{plat}_{version}.py``. + +An error is normally only reported as a warning, but: + + +Interaction With The Coin Module +++++++++++++++++++++++++++++++++ + +When this test program is run in COIN, then the warnings are turned into +errors. The reason is that only in COIN, we have a stable configuration +of PySide modules that can reliably be compared. + +These modules have the name ``exists_{platf}_{version}_ci.py``, and as a big +exception for generated code, these files are *intentionally* checked in. + + +What Happens When a List is Missing? +++++++++++++++++++++++++++++++++++++ + +When a new version of PySide gets created, then the existence test files +initially do not exist. + +When a COIN test is run, then it will complain about the error and create +the missing module on standard output. +But since COIN tests are run multiple times, the output that was generated +by the first test will still exist at the subsequent runs. +(If COIN was properly implemented, we could not take that advantage and +would need to implement that as an extra exception.) + +As a result, a missing module will be reported as a test which partially +succeeded (called "FLAKY"). To avoid further flaky tests and to activate as a real test, +we can now capture the error output of COIN and check the generated module +in. + + +Explicitly Enforcing Recreation ++++++++++++++++++++++++++++++++ + +The former way to regenerate the registry files was to remove the files +and check that in. This has the desired effect, but creates huge deltas. +As a more efficient way, we have prepared a comment in the first line +that contains the word "recreate". +By uncommenting this line, a NameError is triggered, which has the same +effect. + + +init_platform.py +++++++++++++++++ + +For generating the ``exists_{platf}_{version}`` modules, the module +``pyside6/tests/registry/init_platform.py`` was written. It can be used +standalone from the commandline, to check the compatibility of some +changes, directly. + + +scrape_testresults.py +~~~~~~~~~~~~~~~~~~~~~ + +To simplify and automate the process of extracting the ``exists_{platf}_{version}_ci.py`` +files, the script ``pyside6/tests/registry/scrape_testresults.py`` has been written. + +This script scans the whole testresults website for PySide, that is:: + + https://testresults.qt.io/coin/api/results/pyside/pyside-setup/ + +On the first scan, the script runs less than 30 minutes. After that, a cache +is generated and the scan works *much* faster. The test results are placed +into the folder ``pyside6/tests/registry/testresults/embedded/`` with a +unique name that allows for easy sorting. Example:: + + testresults/embedded/2018_09_10_10_40_34-test_1536891759-exists_linux_5_11_2_ci.py + +These files are created only once. If they already exist, they are not touched, again. +The file `pyside6/tests/registry/known_urls.json`` holds all scanned URLs after +a successful scan. The ``testresults/embedded`` folder can be kept for reference +or can be removed. Important is only the json file. + +The result of a scan is then directly placed into the ``pyside6/tests/registry/`` +folder. It should be reviewed and then eventually checked in. + + +generate_pyi.py +~~~~~~~~~~~~~~~ + +``pyside6/PySide6/support/generate_pyi.py`` is still under development. +This module generates so-called hinting stubs for integration of PySide +with diverse *Python IDEs*. + +Although this module creates the stubs as an add-on, the +impact on the quality of the signature module is considerable: + +The module must create syntactically correct ``.pyi`` files which contain +not only signatures but also constants and enums of all PySide modules. +This serves as an extra challenge that has a very positive effect on +the completeness and correctness of signatures. + +The module has a ``--feature`` option to generate modified .pyi files. +A shortcut for this command is ``pyside6-genpyi``. + +A useful command to change all .pyi files to use all features is + +.. code-block:: bash + + pyside6-genpyi all --feature snake_case true_property + + +pyi_generator.py +~~~~~~~~~~~~~~~~ + +``shiboken6/shibokenmodule/files.dir/shibokensupport/signature/lib/pyi_generator.py`` +has been extracted from ``generate_pyi.py``. It allows the generation of ``.pyi`` +files from arbitrary extension modules created with shiboken. + +A shortcut for this command is ``shiboken6-genpyi``. + + +Current Extensions +~~~~~~~~~~~~~~~~~~ + +Before the signature module was written, there already existed the concept of +signatures, but in a more C++ - centric way. From that time, there existed +the error messages, which are created when a function gets wrong argument types. + +These error messages were replaced by text generated on demand by +the signature module, in order to be more consistent and correct. +This was implemented in ``Qt For Python 5.12.0``. + +Additionally, the ``__doc__`` attribute of PySide methods was not set. +It was easy to get a nice ``help()`` feature by creating signatures +as default content for docstrings. +This was implemented in ``Qt For Python 5.12.1``. + + +Literature +---------- + +* `PEP 362 – Function Signature Object <https://www.python.org/dev/peps/pep-0362/>`__ +* `PEP 484 – Type Hints <https://www.python.org/dev/peps/pep-0484/>`__ +* `PEP 3107 – Function Annotations <https://www.python.org/dev/peps/pep-3107/>`__ + + +*Personal Remark: This module is dedicated to our lovebird "Püppi", who died on 2017-09-15.* |