Limited_API: Fix PyIndex_Check once and for all

PyIndex_Check was left as a macro for Python <= 3.7 . This was
fixed for Python 3.8 (I failed to submit the patch in time :( )

The problem is a bit weird, because we cannot do a compile time
decision which Python version it is, because exactly that is
only known at runtime. Therefore:

- we cannot use a builtin version of PyIndex_Check, because
  this would create a link error with Python < 3.8

- PyType_GetSlot would help with this, but unfortunately this
  worked only with heap types, and the use case is on normal
  integers.

The solution is quite ok:
-------------------------
The structure of the type objects from Python 3.6 on is compatible
enough for the field offset that we need here, so on old Python
versions, the old type structure can be used.

From Python 3.10 on, PyType_GetSlot is extended to non-heap types,
and we can simply use that.

This patch can be removed completely when we drop Python 3.7 .
An automated warning that suggests this removal was added.

[ChangeLog][shiboken6] The handling of a complex Limited API bug
was fixed for different combinations of PySide/Python versions.

Change-Id: I945aa5ae1ea5cd9de7c6e140c32a1e9467735a8e
Fixes: PYSIDE-1797
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>
(cherry picked from commit bcd1ac22f8e4e804b4082e311d4a8c43f3f3d4d6)
Reviewed-by: Qt Cherry-pick Bot <cherrypick_bot@qt-project.org>

7 files changed, 76 insertions, 17 deletions

diff --git a/build_scripts/main.py b/build_scripts/main.py
index b4576d0e8..e25f22bd0 100644
--- a/build_scripts/main.py
+++ b/build_scripts/main.py
@@ -151,11 +151,12 @@ def get_make(platform_arch, build_type):
     return (make_path, make_generator)
 
 
-def check_allowed_python_version():
-    """
-    Make sure that setup.py is run with an allowed python version.
-    """
+_allowed_versions_cache = None
 
+def get_allowed_python_versions():
+    global _allowed_versions_cache
+    if _allowed_versions_cache is not None:
+        return _allowed_versions_cache
     pattern = r'Programming Language :: Python :: (\d+)\.(\d+)'
     supported = []
 
@@ -165,6 +166,17 @@ def check_allowed_python_version():
             major = int(found.group(1))
             minor = int(found.group(2))
             supported.append((major, minor))
+
+    _allowed_versions_cache = sorted(supported)
+    return _allowed_versions_cache
+
+
+def check_allowed_python_version():
+    """
+    Make sure that setup.py is run with an allowed python version.
+    """
+
+    supported = get_allowed_python_versions()
     this_py = sys.version_info[:2]
     if this_py not in supported:
         log.error(f"Unsupported python version detected. Supported versions: {supported}")
@@ -589,6 +601,8 @@ class PysideBuild(_build, DistUtilsCommandMixin, BuildInfoCollectorMixin):
             f"-DQt5Help_DIR={self.qtinfo.docs_dir}",
             f"-DCMAKE_BUILD_TYPE={self.build_type}",
             f"-DCMAKE_INSTALL_PREFIX={self.install_dir}",
+            # Record the minimum Python version for later use in Shiboken.__init__
+            f"-DMINIMUM_PYTHON_VERSION={get_allowed_python_versions()[0]}",
             module_src_dir
         ]
 
diff --git a/sources/shiboken6/libshiboken/pep384_issue33738.cpp b/sources/shiboken6/libshiboken/pep384_issue33738.cpp
index c20edeefa..59713f2f5 100644
--- a/sources/shiboken6/libshiboken/pep384_issue33738.cpp
+++ b/sources/shiboken6/libshiboken/pep384_issue33738.cpp
@@ -47,7 +47,8 @@
 // Simple solution: Create the structure and write such a function.
 // Long term: Submit a patch to python.org .
 
-// Update: I did the long-term solution for python 3.7 in issue 33738.
+// This structure comes from Python 3.7, but we have checked that
+// it also works for Python 3.8 and 3.9.
 
 typedef struct {
     /* Number implementations must check *both*
@@ -112,10 +113,45 @@ typedef struct _oldtypeobject {
 
 } PyOldTypeObject;
 
-int PyIndex_Check(PyObject *obj)
+static bool is_compatible_version()
 {
-    PyOldTypeObject *type = reinterpret_cast<PyOldTypeObject *>(Py_TYPE(obj));
-    return type->tp_as_number != NULL &&
-           type->tp_as_number->nb_index != NULL;
+    auto *sysmodule = PyImport_AddModule("sys");
+    auto *dic = PyModule_GetDict(sysmodule);
+    auto *version = PyDict_GetItemString(dic, "version_info");
+    auto *major = PyTuple_GetItem(version, 0);
+    auto *minor = PyTuple_GetItem(version, 1);
+    auto number = PyLong_AsLong(major) * 1000 + PyLong_AsLong(minor);
+    return number < 3010;
+}
+
+///////////////////////////////////////////////////////////////////////
+//
+// PYSIE-1797: The Solution
+// ========================
+//
+// Inspecting the data structures of Python 3.6, 3.7, 3.8 and 3.9
+// shows that concerning the here needed offset of nb_index, they
+// are all compatible.
+// That means: We can use the above definition for all these versions.
+//
+// From Python 3.10 on, the `PyType_GetSlot` function also works with
+// non-heap types. That means this solution will always work.
+//
+// Note: When we have moved to Python 3.8 as the minimum version,
+// this whole nonsense can be trashed.
+// There is an automatic warning about this in parser.py .
+//
+
+LIBSHIBOKEN_API int PyIndex_Check(PyObject *obj)
+{
+    static bool old_python_version = is_compatible_version();
+    if (old_python_version) {
+        auto *type = reinterpret_cast<PyOldTypeObject *>(Py_TYPE(obj));
+        return type->tp_as_number != nullptr &&
+               type->tp_as_number->nb_index != nullptr;
+    }
+    // From Python 3.10 on, we can use PyType_GetSlot also with normal types!
+    unaryfunc nb_index = reinterpret_cast<unaryfunc>(PyType_GetSlot(Py_TYPE(obj), Py_nb_index));
+    return nb_index != nullptr;
 }
 
diff --git a/sources/shiboken6/libshiboken/pep384impl.cpp b/sources/shiboken6/libshiboken/pep384impl.cpp
index b7a39adcb..1ddbe3ba1 100644
--- a/sources/shiboken6/libshiboken/pep384impl.cpp
+++ b/sources/shiboken6/libshiboken/pep384impl.cpp
@@ -191,9 +191,8 @@ check_PyTypeObject_valid()
     Py_DECREF(probe_tp_mro);
 }
 
-#if PY_VERSION_HEX < PY_ISSUE33738_SOLVED
+// PYSIDE-1797: This must be a runtime decision.
 #include "pep384_issue33738.cpp"
-#endif
 
 #endif // Py_LIMITED_API
 
diff --git a/sources/shiboken6/libshiboken/pep384impl.h b/sources/shiboken6/libshiboken/pep384impl.h
index 62a66e37e..0198ff3f1 100644
--- a/sources/shiboken6/libshiboken/pep384impl.h
+++ b/sources/shiboken6/libshiboken/pep384impl.h
@@ -140,12 +140,14 @@ typedef struct _typeobject {
 #endif
 
 // This was a macro error in the limited API from the beginning.
-// It was fixed in Python master, but did make it only in Python 3.8 .
-#define PY_ISSUE33738_SOLVED 0x03080000
-#if PY_VERSION_HEX < PY_ISSUE33738_SOLVED
+// It was fixed in Python master, but did make it only into Python 3.8 .
+
+// PYSIDE-1797: This must be a runtime decision.
+//              Remove that when the minimum Python version is 3.8,
+//              because the macro PyIndex_Check bug was fixed then.
+/// FIXME: Remove PyIndex_Check and pep384_issue33738.cpp when Python 3.7 is gone.
 #undef PyIndex_Check
 LIBSHIBOKEN_API int PyIndex_Check(PyObject *obj);
-#endif
 
 LIBSHIBOKEN_API PyObject *_PepType_Lookup(PyTypeObject *type, PyObject *name);
 
diff --git a/sources/shiboken6/shibokenmodule/CMakeLists.txt b/sources/shiboken6/shibokenmodule/CMakeLists.txt
index da534ba75..2d5276652 100644
--- a/sources/shiboken6/shibokenmodule/CMakeLists.txt
+++ b/sources/shiboken6/shibokenmodule/CMakeLists.txt
@@ -43,6 +43,10 @@ configure_file("${CMAKE_CURRENT_SOURCE_DIR}/_config.py.in"
 install(FILES "${CMAKE_CURRENT_BINARY_DIR}/_config.py"
         DESTINATION "${PYTHON_SITE_PACKAGES}/shiboken6")
 
+if ("${MINIMUM_PYTHON_VERSION}" STREQUAL "")
+    set(MINIMUM_PYTHON_VERSION None)
+endif()
+
 # PYSIDE-1497: This `..` is the crucial trick to unify the path location of `Shiboken`.
 configure_file("${CMAKE_CURRENT_SOURCE_DIR}/__init__.py.in"
                "${CMAKE_CURRENT_BINARY_DIR}/../__init__.py" @ONLY)
diff --git a/sources/shiboken6/shibokenmodule/__init__.py.in b/sources/shiboken6/shibokenmodule/__init__.py.in
index 3dd59b024..ab95667da 100644
--- a/sources/shiboken6/shibokenmodule/__init__.py.in
+++ b/sources/shiboken6/shibokenmodule/__init__.py.in
@@ -1,5 +1,6 @@
 __version__ = "@FINAL_PACKAGE_VERSION@"
 __version_info__ = (@shiboken_MAJOR_VERSION@, @shiboken_MINOR_VERSION@, @shiboken_MICRO_VERSION@, "@shiboken_PRE_RELEASE_VERSION_TYPE@", "@shiboken_PRE_RELEASE_VERSION@")
+__minimum_python_version__ = @MINIMUM_PYTHON_VERSION@
 
 # PYSIDE-932: Python 2 cannot import 'zipfile' for embedding while being imported, itself.
 # We simply pre-load all imports for the signature extension.
diff --git a/sources/shiboken6/shibokenmodule/files.dir/shibokensupport/signature/parser.py b/sources/shiboken6/shibokenmodule/files.dir/shibokensupport/signature/parser.py
index 68d991049..86451f68f 100644
--- a/sources/shiboken6/shibokenmodule/files.dir/shibokensupport/signature/parser.py
+++ b/sources/shiboken6/shibokenmodule/files.dir/shibokensupport/signature/parser.py
@@ -77,7 +77,7 @@ In effect, 'type_map' maps text to real Python objects.
 """
 
 def _get_flag_enum_option():
-    from shiboken6 import __version_info__ as ver
+    from shiboken6 import __version_info__ as ver, __minimum_python_version__ as pyminver
     # PYSIDE-1735: Use the new Enums per default if version is >= 6.4
     #              This decides between delivered vs. dev versions.
     #              When 6.4 is out, the switching mode will be gone.
@@ -97,10 +97,13 @@ def _get_flag_enum_option():
         flag = bool(getattr(sys, sysname))
     sysver = sys.version_info[:2]
     if flag and sysver < (3, 7):
-        import warnings
         warnings.warn(f"Enums with functional API are not supported in "
                       f"Python {'.'.join(map(str, sysver))}")
         flag = False
+    # PYSIDE-1797: Emit a warning when we may remove pep384_issue33738.cpp
+    if pyminver and pyminver >= (3, 8):
+        warnings.warn(f"\n    *** Python is at version {'.'.join(map(str, pyminver))} now. "
+                      f"The file pep384_issue33738.cpp should be removed ASAP! ***")
     # modify the sys attribute to bool
     setattr(sys, sysname, flag)
     # modify the env attribute to "0" or "1"