Fix encapsulation of plugins

Fixes that triggering "About Python Extensions" action resulted in
python runtime error "TypeError: must be type, not None" when
referencing PySide2.QtWidgets.

To improve encapsulation of plugins, all modules that they loaded
were deleted after their main.py finished running. This breaks
if these modules were accessed later e.g. triggered by actions.

Instead of this hack, ensure encapsulation of plugins by making them
actual Python packages.
Qt Creator's python extension path is added to python's module search
path, and extensions are simply imported as packages.
The python extension's main.py simply becomes a standard python
module __init__.py.

This also means that python extensions can depend on, and use other
python extensions' functionality by importing them with "import".

Change-Id: Ibe74c24e337b321007f5fa19c97bd35a5c1f0375
Reviewed-by: Friedemann Kleint <Friedemann.Kleint@qt.io>

11 files changed, 23 insertions, 37 deletions

diff --git a/README.md b/README.md
index fd2252c..2cc6722 100644
--- a/README.md
+++ b/README.md
@@ -1,11 +1,10 @@
 # Python Extensions for QtCreator
 
 This plugin for QtCreator makes it possible to extend the QtCreator by
-writing Python extensions. These extensions consist of a directory
-containing a `main.py` and any other Python files necessary. They can
-be shared as zip archives, installed and managed through the included
-extension manager (which is a Python extension itself) and thus allow
-the QtCreator to be easily extended with custom functionality.
+writing Python extensions. These extensions are simply Python packages,
+so they are directories containing a `__init__.py` and any other Python
+files necessary. They can be shared as zip archives, installed and managed
+through the included extension manager (which is a Python extension itself) and thus allow the QtCreator to be easily extended with custom functionality.
 
 **WARNING:** This is a first draft / proof of concept and only offers
 very limited functionality. There will still be many bugs and so far
@@ -129,21 +128,20 @@ The following process allows the plugin to work:
  2. When QtCreator is executed, the C++ plugin searches the standard
     QtCreator plugin directories for a directory named `python`, the
     first directory found is the Python extension directory.
-    - Now, each subdirectory represents it's own Python extension. For each
-      subdirectory the C++ plugin checks whether it contains a `setup.py`.
-      If it does, this setup script is executed.
-    - After all the setup scripts have been executed, each subdirectory is
-      checked for a file named `main.py`. This file is the extensions entry
-      point and is executed by the C++ plugin.
+    - Now, each package in this directory represents it's own Python extension.
+      For each package the C++ plugin checks whether it
+      contains a `setup.py`. If it does, this setup script is executed.
+    - After all the setup scripts have been executed, each package is loaded
+      with `import`, which executes the initialization code in its
+      `__init__.py`.
  3. Now all Python extensions have registered their actions / menus / etc.,
     which can be triggered from the QtCreator interface.
 
 When executed, the Python extensions can import any modules / packages
-installed for the system Python or found in their own directory. While
-the C++ plugin takes some precautions to isolate the Python extensions
-when executed, they are still all run in the same Python instance,
-which means that they are not completely isolated. However, so far
-this did not turn out to be a problem.
+installed for the system Python, in Qt Creator's extensions directory, or
+found in their own directory.
+Python extensions are all run in the same Python instance,
+which means that they are not completely isolated.
 
 For a more detailed description, please refer to the documentation in
 `docs`.
diff --git a/docs/extensions.md b/docs/extensions.md
index eda9c4c..6d4998d 100644
--- a/docs/extensions.md
+++ b/docs/extensions.md
@@ -102,14 +102,12 @@ When importing modules, the following important locations will be checked (in th
 
  1. The folder of the extension itself (files and folders in your extension)
  2. The system Python path entries (anything you `pip install`ed globally)
- 3. The QtCreator specific Python module directory
+ 3. Qt Creator's python extension directory
+ 4. The Qt Creator specific Python module directory
     - Note: This is where you should install any dependencies missing
       if you want to use non-standard Python packages / modules
     - This last path is accessible with `PluginInstance.pythonPackagePath()`
 
-Any changes you make to sys.path and any modules you import, will be cleared after your script
-finished executing.
-
 ## Reserved variable names
 Names that look like
 ```Python
diff --git a/examples/macroexpander/main.py b/examples/macroexpander/__init__.py
index bd64028..bd64028 100644
--- a/examples/macroexpander/main.py
+++ b/examples/macroexpander/__init__.py
diff --git a/examples/numpysetup/main.py b/examples/numpysetup/__init__.py
index cf67157..cf67157 100644
--- a/examples/numpysetup/main.py
+++ b/examples/numpysetup/__init__.py
diff --git a/examples/projects/main.py b/examples/projects/__init__.py
index daac366..daac366 100644
--- a/examples/projects/main.py
+++ b/examples/projects/__init__.py
diff --git a/examples/requirerequests/main.py b/examples/requirerequests/__init__.py
index 9ca4ee9..9ca4ee9 100644
--- a/examples/requirerequests/main.py
+++ b/examples/requirerequests/__init__.py
diff --git a/examples/smallmenu/main.py b/examples/smallmenu/__init__.py
index b2753c1..b2753c1 100644
--- a/examples/smallmenu/main.py
+++ b/examples/smallmenu/__init__.py
diff --git a/examples/transform/main.py b/examples/transform/__init__.py
index b0d47f1..b0d47f1 100644
--- a/examples/transform/main.py
+++ b/examples/transform/__init__.py
diff --git a/plugins/pythonextensions/pythonextensionsplugin.cpp b/plugins/pythonextensions/pythonextensionsplugin.cpp
index 078a78c..f0ddcc5 100644
--- a/plugins/pythonextensions/pythonextensionsplugin.cpp
+++ b/plugins/pythonextensions/pythonextensionsplugin.cpp
@@ -164,8 +164,10 @@ QString PythonExtensionsPlugin::pythonPackagePath()
 void PythonExtensionsPlugin::initializePythonBindings()
 {
     // Add our custom module directory
-    if (extensionDir().exists())
+    if (extensionDir().exists()) {
+        PyUtil::addToSysPath(extensionDir().path().toStdString());
         PyUtil::addToSysPath(pythonPackagePath().toStdString());
+    }
     // Initialize the Python context and register global QtCreator variable
     if (!PyUtil::bindShibokenModuleObject("PythonExtension", "QtCreator")) {
         qWarning() << "Python bindings could not be initialized";
@@ -268,23 +270,11 @@ void PythonExtensionsPlugin::initializePythonExtensions()
     // Run the extension initialization code
     for (const QString &extension : extension_list) {
         qDebug() << "Trying to initialize extension" << extension;
-
-        QFile extension_main(extension_dir.absolutePath() + "/" + extension + "/main.py");
-        if (extension_main.open(QIODevice::ReadOnly)) {
-            QTextStream in(&extension_main);
-            QString extension_code = in.readAll();
-            if (!PyUtil::runScriptWithPath(
-                extension_code.toStdString(),
-                QString(extension_dir.absolutePath() + "/" + extension).toStdString()
-            )) {
-                qWarning() << "Failed to initialize extension" << extension;
-                Core::MessageManager::write(Constants::MESSAGE_MANAGER_PREFIX + tr("Failed to initialize extension ") + extension);
-            } else {
-                m_loadedExtensions << extension;
-            }
+        if (!PyUtil::runScript("import " + extension.toStdString())) {
+            qWarning() << "Failed to initialize extension" << extension;
+            Core::MessageManager::write(Constants::MESSAGE_MANAGER_PREFIX + tr("Failed to initialize extension ") + extension);
         } else {
-            qWarning() << "Failed to load main.py for extension" << extension;
-            Core::MessageManager::write(Constants::MESSAGE_MANAGER_PREFIX + tr("Failed to load main.py for extension ") + extension);
+            m_loadedExtensions << extension;
         }
     }
 
diff --git a/python/extensionmanager/main.py b/python/extensionmanager/__init__.py
index 048423b..048423b 100644
--- a/python/extensionmanager/main.py
+++ b/python/extensionmanager/__init__.py
diff --git a/tests/testextension/main.py b/tests/testextension/__init__.py
index 0c3f988..0c3f988 100644
--- a/tests/testextension/main.py
+++ b/tests/testextension/__init__.py