path: root/sources/shiboken2/libshiboken/pep384impl_doc.rst

****************************************
The Transition To The Limited Python API
****************************************


Foreword
========

Python supports a limited API that restricts access to certain structures.
Besides eliminating whole modules and all functions and macros which names
start with an
underscore, the most drastic restriction is the removal of normal type object
declarations.

For details about the eliminated modules and functions, please see the
`PEP 384`_ page for reference.


.. _`PEP 384`: https://www.python.org/dev/peps/pep-0384/



Changed Modules
===============

All changed module's include files are listed with the changed functions here.
As a general rule, it was tried to keep the changes to a minimum diff.
Macros which are not available were changed to functions with the same name
if possible. Completely removed names ``Py{name}`` were re-implemented as ``Pep{name}``.


memoryobject.h
--------------

The buffer protocol was completely removed. We redefined all the structures
and methods, because PySide uses that. This is an exception to the limited API
that we have to check ourselves. The code is extracted in bufferprocs_py37.h .
This is related to the following:


abstract.h
----------

This belongs to the buffer protocol like memoryobject.h .
As replacement for ``Py_buffer`` we defined ``Pep_buffer`` and several other
internal macros.

The version is checked by hand, and the version number must be updated only
if the implementation does not change. Otherwise, we need to write version
dependent code paths.

It is questionable if it is worthwhile to continue using the buffer protocol
or if we should try to get rid of ``Pep_buffer``, completely.


longobject.h
------------

``_PyLong_AsInt`` is not available. We defined a ``_PepLong_AsInt`` function, instead.
Maybe this should be replaced by ``PyLong_AsLong``.


pydebug.h
---------

We have no direct access to ``Py_VerboseFlag`` because debugging is not
supported. We redefined it as macro ``Py_VerboseFlag`` which calls ``Pep_VerboseFlag``.


unicodeobject.h
---------------

The macro ``PyUnicode_GET_SIZE`` was redefined to call into ``PyUnicode_GetSize``.
Function ``_PyUnicode_AsString`` is unavailable and was replaced by a macro
that calls ``_PepUnicode_AsString``. The implementation was a bit involved,
and it would be better to change the code and replace this function.


bytesobject.h
-------------

The macros ``PyBytes_AS_STRING`` and ``PyBytes_GET_SIZE`` were redefined to call
the according functions.


floatobject.h
-------------

``PyFloat_AS_DOUBLE`` now calls ``PyFloat_AsDouble``.


tupleobject.h
-------------

``PyTuple_GET_ITEM``, ``PyTuple_SET_ITEM`` and ``PyTuple_GET_SIZE`` were redefined as
function calls.


listobject.h
------------

``PyList_GET_ITEM``, ``PyList_SET_ITEM`` and ``PyList_GET_SIZE`` were redefined as
function calls.


methodobject.h
--------------

``PyCFunction_GET_FUNCTION``, ``PyCFunction_GET_SELF`` and ``PyCFunction_GET_FLAGS``
were redefined as function calls.

Direct access to the methoddef structure is not available, and we defined
``PepCFunction_GET_NAMESTR`` as accessor for name strings.


pythonrun.h
-----------

The simple function ``PyRun_String`` is not available. It was re-implemented
in a simplified version for the signature module.


funcobject.h
------------

The definitions of funcobject.h are completely missing, although there
are extra ``#ifdef`` conditional defines inside, too. This suggests that the exclusion
was unintended.

We therefore redefined ``PyFunctionObject`` as an opaque type.

The missing macro ``PyFunction_Check`` was defined, and the macro
``PyFunction_GET_CODE`` calls the according function.

There is no equivalent for function name access, therefore we introduced
``PepFunction_GetName`` either as a function or as a macro.

*TODO: We should fix funcobject.h*


classobject.h
-------------

Classobject is also completely not imported, instead of defining an opaque type.

We defined the missing functions ``PyMethod_New``, ``PyMethod_Function`` and
``PyMethod_Self`` and also redefined ``PyMethod_GET_SELF`` and
``PyMethod_GET_FUNCTION`` as calls to these functions.

*TODO: We should fix classobject.h*


code.h
------

The whole code.c code is gone, although it may make sense to
define some minimum accessibility. This will be clarified on
`Python-Dev`_. We needed access to code objects and defined the missing
PepCode_GET_FLAGS and PepCode_GET_ARGCOUNT either as function or macro.
We further added the missing flags, although few are used:

``CO_OPTIMIZED`` ``CO_NEWLOCALS`` ``CO_VARARGS`` ``CO_VARKEYWORDS`` ``CO_NESTED``
``CO_GENERATOR``

*TODO: We should maybe fix code.h*

.. _`Python-Dev`: https://mail.python.org/mailman/listinfo/python-dev

datetime.h
----------

The DateTime module is explicitly not included in the limited API.
We defined all the needed functions but called them via Python instead
of direct call macros. This has a slight performance impact.

The performance could be easily improved by providing an interface
that fetches all attributes at once, instead of going through the object
protocol every time.

The re-defined macros and methods are::

    PyDateTime_GET_YEAR
    PyDateTime_GET_MONTH
    PyDateTime_GET_DAY
    PyDateTime_DATE_GET_HOUR
    PyDateTime_DATE_GET_MINUTE
    PyDateTime_DATE_GET_SECOND
    PyDateTime_DATE_GET_MICROSECOND
    PyDateTime_DATE_GET_FOLD
    PyDateTime_TIME_GET_HOUR
    PyDateTime_TIME_GET_MINUTE
    PyDateTime_TIME_GET_SECOND
    PyDateTime_TIME_GET_MICROSECOND
    PyDateTime_TIME_GET_FOLD

    PyDate_Check
    PyDateTime_Check
    PyTime_Check

    PyDate_FromDate
    PyDateTime_FromDateAndTime
    PyTime_FromTime

*XXX: We should maybe provide an optimized interface to datetime*


object.h
--------

The file object.h contains the ``PyTypeObject`` structure, which is supposed
to be completely opaque. All access to types should be done through
``PyType_GetSlot`` calls. Due to bugs and deficiencies in the limited API
implementation, it was not possible to do that. Instead, we have defined
a simplified structure for ``PyTypeObject`` that has only the fields that
are used in PySide.

We will explain later why and how this was done. Here is the reduced
structure::

    typedef struct _typeobject {
        PyVarObject ob_base;
        const char *tp_name;
        Py_ssize_t tp_basicsize;
        void *X03; // Py_ssize_t tp_itemsize;
        void *X04; // destructor tp_dealloc;
        void *X05; // printfunc tp_print;
        void *X06; // getattrfunc tp_getattr;
        void *X07; // setattrfunc tp_setattr;
        void *X08; // PyAsyncMethods *tp_as_async;
        void *X09; // reprfunc tp_repr;
        void *X10; // PyNumberMethods *tp_as_number;
        void *X11; // PySequenceMethods *tp_as_sequence;
        void *X12; // PyMappingMethods *tp_as_mapping;
        void *X13; // hashfunc tp_hash;
        ternaryfunc tp_call;
        reprfunc tp_str;
        void *X16; // getattrofunc tp_getattro;
        void *X17; // setattrofunc tp_setattro;
        void *X18; // PyBufferProcs *tp_as_buffer;
        void *X19; // unsigned long tp_flags;
        void *X20; // const char *tp_doc;
        traverseproc tp_traverse;
        inquiry tp_clear;
        void *X23; // richcmpfunc tp_richcompare;
        Py_ssize_t tp_weaklistoffset;
        void *X25; // getiterfunc tp_iter;
        void *X26; // iternextfunc tp_iternext;
        struct PyMethodDef *tp_methods;
        void *X28; // struct PyMemberDef *tp_members;
        void *X29; // struct PyGetSetDef *tp_getset;
        struct _typeobject *tp_base;
        PyObject *tp_dict;
        descrgetfunc tp_descr_get;
        void *X33; // descrsetfunc tp_descr_set;
        Py_ssize_t tp_dictoffset;
        initproc tp_init;
        allocfunc tp_alloc;
        newfunc tp_new;
        freefunc tp_free;
        inquiry tp_is_gc; /* For PyObject_IS_GC */
        PyObject *tp_bases;
        PyObject *tp_mro; /* method resolution order */
    } PyTypeObject;

Function ``PyIndex_Check`` had to be defined in an unwanted way due to
a Python issue. See file pep384_issue33738.cpp .

There are extension structures which have been isolated as special macros that
dynamically compute the right offsets of the extended type structures:

*   ``PepType_SOTP`` for ``SbkObjectTypePrivate``
*   ``PepType_SETP`` for ``SbkEnumTypePrivate``
*   ``PepType_PFTP`` for ``PySideQFlagsTypePrivate``
*   ``PepType_SGTP`` for ``_SbkGenericTypePrivate``

How these extension structures are used can best be seen by searching
``PepType_{four}`` in the source.

Due to the new heaptype interface, the names of certain types contain
now the module name in the ``tp_name`` field. To have a compatible way
to access simple type names as C string, ``PepType_GetNameStr`` has been
written that skips over dotted name parts.

Finally, the function ``_PyObject_Dump`` was excluded from the limited API.
This is a useful debugging aid that we always want to have available,
so it is added back, again.


Using The New Type API
======================

After converting everything but the object.h file, we were a little
bit shocked: it suddenly was clear that we would have no more
access to type objects, and even more scary that all types which we
use have to be heap types, only!

For PySide with its intense use of heap type extensions in various
flavors, the situation looked quite unsolvable. In the end, it was
nicely solved, but it took almost 3.5 months to get that right.

Before we see how this is done, we will explain the differences
between the APIs and their consequences.


The Interface
-------------

The old type API of Python knows static types and heap types.
Static types are written down as a declaration of a ``PyTypeObject``
structure with all its fields filled in. Here is for example
the definition of the Python type ``object`` (Python 3.6)::

    PyTypeObject PyBaseObject_Type = {
        PyVarObject_HEAD_INIT(&PyType_Type, 0)
        "object",                                   /* tp_name */
        sizeof(PyObject),                           /* tp_basicsize */
        0,                                          /* tp_itemsize */
        object_dealloc,                             /* tp_dealloc */
        0,                                          /* tp_print */
        0,                                          /* tp_getattr */
        0,                                          /* tp_setattr */
        0,                                          /* tp_reserved */
        object_repr,                                /* tp_repr */
        0,                                          /* tp_as_number */
        0,                                          /* tp_as_sequence */
        0,                                          /* tp_as_mapping */
        (hashfunc)_Py_HashPointer,                  /* tp_hash */
        0,                                          /* tp_call */
        object_str,                                 /* tp_str */
        PyObject_GenericGetAttr,                    /* tp_getattro */
        PyObject_GenericSetAttr,                    /* tp_setattro */
        0,                                          /* tp_as_buffer */
        Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,   /* tp_flags */
        PyDoc_STR("object()\n--\n\nThe most base type"),  /* tp_doc */
        0,                                          /* tp_traverse */
        0,                                          /* tp_clear */
        object_richcompare,                         /* tp_richcompare */
        0,                                          /* tp_weaklistoffset */
        0,                                          /* tp_iter */
        0,                                          /* tp_iternext */
        object_methods,                             /* tp_methods */
        0,                                          /* tp_members */
        object_getsets,                             /* tp_getset */
        0,                                          /* tp_base */
        0,                                          /* tp_dict */
        0,                                          /* tp_descr_get */
        0,                                          /* tp_descr_set */
        0,                                          /* tp_dictoffset */
        object_init,                                /* tp_init */
        PyType_GenericAlloc,                        /* tp_alloc */
        object_new,                                 /* tp_new */
        PyObject_Del,                               /* tp_free */
    };

We can write the same structure in form of a ``PyType_Spec`` structure,
and there is even an incomplete tool *abitype.py* that does this conversion
for us. With a few corrections, the result looks like this::

    static PyType_Slot PyBaseObject_Type_slots[] = {
        {Py_tp_dealloc,     (void *)object_dealloc},
        {Py_tp_repr,        (void *)object_repr},
        {Py_tp_hash,        (void *)_Py_HashPointer},
        {Py_tp_str,         (void *)object_str},
        {Py_tp_getattro,    (void *)PyObject_GenericGetAttr},
        {Py_tp_setattro,    (void *)PyObject_GenericSetAttr},
        {Py_tp_richcompare, (void *)object_richcompare},
        {Py_tp_methods,     (void *)object_methods},
        {Py_tp_getset,      (void *)object_getsets},
        {Py_tp_init,        (void *)object_init},
        {Py_tp_alloc,       (void *)PyType_GenericAlloc},
        {Py_tp_new,         (void *)object_new},
        {Py_tp_free,        (void *)PyObject_Del},
        {0, 0},
    };
    static PyType_Spec PyBaseObject_Type_spec = {
        "object",
        sizeof(PyObject),
        0,
        Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
        PyBaseObject_Type_slots,
    };

This new structure is almost compatible with the old one, but there
are some subtle differences.

* The new types are generated in one step

This seems to be no problem, but it was very much, due to the way the
types were built in PySide. Types were assembled piece by piece, and
finally the ``PyType_Ready`` function was called.

With the new API, ``PyType_Ready`` is called already at the end of
``PyType_FromSpec``, and that meant that the logic of type creation became
completely turned upside down.

* The new types are always heaptypes

With the new type creation functions, it is no longer possible to
create "normal" types. Instead, they all have to be allocated on the
heap and garbage collected. The user should normally not recognize this.
But type creation is more constrained, and you cannot create a subtype
if the ``Py_TPFLAGS_BASETYPE`` is not set. This constraint was already
violated by PySide and needed a quite profound fix.

* The new types always need a module

While this is not a problem per se, the above new type spec will not create
a usable new type, but complain with::

    DeprecationWarning: builtin type object has no __module__ attribute

But there are more problems:

* The new types have unexpected defaults

When fields are empty, you would usually assume that they stay empty.
There are just a few corrections that ``PyType_Ready`` will do to a type.

But there is the following clause in ``PyType_FromSpec`` that can give you
many headaches::

    if (type->tp_dealloc == NULL) {
        /* It's a heap type, so needs the heap types' dealloc.
           subtype_dealloc will call the base type's tp_dealloc, if
           necessary. */
        type->tp_dealloc = subtype_dealloc;
    }

In fact, before the move to the new API, the ``PyType_Ready`` function
filled empty ``tp_dealloc`` fields with ``object_dealloc``. And the code
that has been written with that in mind now becomes pretty wrong if suddenly
``subtype_dealloc`` is used.

The way out was to explicitly provide an ``object_dealloc`` function.
This would then again impose a problem, because ``object_dealloc`` is not
public. Writing our own version is easy, but it again needs access to
type objects. But fortunately, we have broken this rule, already...


* The new types are only partially allocated

The structures used in ``PyType_FromSpec`` are almost all allocated,
only the name field is static. This is no problem for types which are
statically created once. But if you want to parameterize things and
create multiple types with a single slots and spec definition, the name
field that is used for tp_name must be allocated dynamically.
This is misleading, since all the slots already are copies.

* The new types don't support special offsets

The special fields ``tp_weaklistoffset`` and ``tp_dictoffset`` are not supported
by ``PyType_FromSpec``. Unfortunately the documentation does not tell you
if you are allowed to set these fields manually after creating the type or not.
We finally did it and it worked, but we are not sure about correctness.

See basewrapper.cpp function ``SbkObject_TypeF()`` as the only reference to
these fields in PySide. This single reference is absolutely necessary and
very important, since all derived types invisibly inherit these two fields.


Future Versions Of The Limited API
==================================

As we have seen, the current version of the limited API does a bit of
cheating, because it uses parts of the data structure that should be
an opaque type. At the moment, this works fine because the data is
still way more compatible as it could be.

But what if this is changed in the future?

We know that the data structures are stable until Python 3.8 comes out.
Until then, the small bugs and omissions will hopefully all be solved.
Then it will be possible to replace the current small tricks by calls
to ``PyType_GetSlot`` in the way things should be.

At the very moment when the current assumptions about the data structure
are no longer true, we will rewrite the direct attribute access with
calls to ``PyType_GetSlot``. After that, no more changes will be necessary.


Appendix A: The Transition To Simpler Types
===========================================

After all code had been converted to the limited API, there was a
remaining problem with the ``PyHeapTypeObject``.

Why a problem? Well, all the type structures in shiboken use
special extra fields at the end of the heap type object. This
currently enforces extra knowledge at compile time about how large the
heap type object is. In a clean implementation, we would only use
the ``PyTypeObject`` itself and access the fields *behind* the type
by a pointer that is computed at runtime.


Restricted PyTypeObject
-----------------------

Before we are going into details, let us motivate the existence of
the restricted ``PyTypeObject``:

Originally, we wanted to use ``PyTypeObject`` as an opaque type and
restrict ourselves to only use the access function ``PyType_GetSlot``.
This function allows access to all fields which are supported by
the limited API.

But this is a restriction, because we get no access to ``tp_dict``,
which we need to support the signature extension. But we can work
around that.

The real restriction is that ``PyType_GetSlot`` only works for heap
types. This makes the function quite useless, because we have
no access to ``PyType_Type``, which is the most important type ``type``
in Python. We need that for instance to compute the size of
``PyHeapTypeObject`` dynamically.

With much effort, it is possible to clone ``PyType_Type`` as a heap
type. But due to a bug in the Pep 384 support, we need
access to the ``nb_index`` field of a normal type. Cloning does not
help because ``PyNumberMethods`` fields are *not* inherited.

After we realized this dead end, we changed concept and did not
use ``PyType_GetSlot`` at all (except in function ``copyNumberMethods``),
but created a restricted ``PyTypeObject`` with only those fields
defined that are needed in PySide.

Is this breakage of the limited API? I don't think so. A special
function runs on program startup that checks the correct position
of the fields of ``PyTypeObject``, although a change in those fields is
more than unlikely.
The really crucial thing is to no longer use ``PyHeapTypeObject``
explicitly because that *does* change its layout over time.


Diversification
---------------

There were multiple ``Sbk{something}`` structures which all used a "d" field
for their private data. This made it not easy to find the right
fields when switching between objects and types::

    struct LIBSHIBOKEN_API SbkObject
    {
        PyObject_HEAD
        PyObject *ob_dict;
        PyObject *weakreflist;
        SbkObjectPrivate *d;
    };

    struct LIBSHIBOKEN_API SbkObjectType
    {
        PyHeapTypeObject super;
        SbkObjectTypePrivate *d;
    };

The first step was to rename the SbkObjectTypePrivate part from "d" to
"sotp". It was chosen to be short but easy to remember as abbreviation
of "SbkObjectTypePrivate", leading to::

    struct LIBSHIBOKEN_API SbkObjectType
    {
        PyHeapTypeObject super;
        SbkObjectTypePrivate *sotp;
    };

After renaming, it was easier to do the following transformations.


Abstraction
-----------

After renaming the type extension pointers to ``sotp``, I replaced
them by function-like macros which did the special access *behind*
the types, instead of those explicit fields. For instance, the
expression::

    type->sotp->converter

became::

    PepType_SOTP(type)->converter

The macro expansion can be seen here::

    #define PepHeapType_SIZE \
        (reinterpret_cast<PyTypeObject *>(&PyType_Type)->tp_basicsize)

    #define _genericTypeExtender(etype) \
        (reinterpret_cast<char *>(etype) + PepHeapType_SIZE)

    #define PepType_SOTP(etype) \
        (*reinterpret_cast<SbkObjectTypePrivate **>(_genericTypeExtender(etype)))

This looks complicated, but in the end there is only a single new
indirection via ``PyType_Type``, which happens at runtime. This is the
key to fulfil what Pep 384 wants to achieve: *No more version-dependent fields*.


Simplification
--------------

After all type extension fields were replaced by macro calls, we
could remove the following version dependent re-definition of ``PyHeapTypeObject``
::

    typedef struct _pyheaptypeobject {
        union {
            PyTypeObject ht_type;
            void *opaque[PY_HEAPTYPE_SIZE];
        };
    } PyHeapTypeObject;

, and the version dependent structure::

    struct LIBSHIBOKEN_API SbkObjectType
    {
        PyHeapTypeObject super;
        SbkObjectTypePrivate *sotp;
    };

could be replaced by the simplified::

    struct LIBSHIBOKEN_API SbkObjectType
    {
        PyTypeObject type;
    };

which is no longer version-dependent.
Note that we tried to replace the above struct directly by ``PyTypeObject``,
but that was too much. The distinction between ``SbkObjectType`` and
``PyTypeObject`` is still needed.


Appendix B: Verification Of PyTypeObject
========================================

We have introduced a limited PyTypeObject in the same place
as the original PyTypeObject, and now we need to prove that
we are allowed to do so.

When using the limited API as intended, then types are completely
opaque, and access is only through ``PyType_FromSpec`` and (from
version 3.5 upwards) through ``PyType_GetSlot``.

Python then uses all the slot definitions in the type description
and produces a regular heap type object.


Unused Information
------------------

We know many things about types that are not explicitly said,
but they are inherently clear:

(a) The basic structure of a type is always the same, regardless
    if it is a static type or a heap type.

(b) types are evolving very slowly, and a field is never replaced
    by another field with different semantics.

Inherent rule (a) gives us the following information: If we calculate
the offsets of the basic fields, then this info is also usable for non-heap
types.

The validation checks if rule (b) is still valid.


How it Works
------------

The basic idea of the validation is to produce a new type using
``PyType_FromSpec`` and to see where in the type structure these fields
show up. So we build a ``PyType_Slot`` structure with all the fields we
are using and make sure that these values are all unique in the
type.

Most fields are not interrogated by ``PyType_FromSpec``, and so we
simply used some numeric value. Some fields are interpreted, like
``tp_members``. This field must really be a ``PyMemberDef``. And there are
``tp_base`` and ``tp_bases`` which have to be type objects and lists
thereof. It was easiest to not produce these fields from scratch
but use them from the ``type`` object ``PyType_Type``.

Then one would think to write a function that searches the known
values in the opaque type structure.

But we can do better and use optimistically the observation (b):
We simply use the restricted ``PyTypeObject`` structure and assume that
every field lands exactly where we are awaiting it.

And that is the whole proof: If we find all the disjoint values at
the places where we expect them, then verification is done.


About ``tp_dict``
-----------------

One word about the ``tp_dict`` field: This field is a bit special in
the proof, since it does not appear in the spec and cannot easily
be checked by ``type.__dict__`` because that creates a *dictproxy*
object. So how do we prove that is really the right dict?

We have to create that ``PyMethodDef`` structure anyway, and instead of
leaving it empty, we insert a dummy function. Then we ask the
``tp_dict`` field if it has the awaited object in it, and that's it!

#EOT