Initial commit

This is a quite large commit containing:

 * The main extension that runs and initializes Python
 * Some (example) bindings
 * An initial build script for the main extension
 * Optional binding and examples of how to create them
 * An initial build script for the optional bindings
 * A simple extension manager written in Python
 * A few example Python extensions
 * Some documentation (both in the code and as markdown files)
 * A collection of helpful python scripts
 * A small collection of unit tests
 * A TODO list

For any additional details the code / docs should be consulted.

Change-Id: I3937886cfefa2f64d5a78013889a8e097eec8261
Reviewed-by: Eike Ziller <eike.ziller@qt.io>

4 files changed, 355 insertions, 0 deletions

diff --git a/docs/README.md b/docs/README.md
new file mode 100644
index 0000000..22fbe0e
--- /dev/null
+++ b/docs/README.md
@@ -0,0 +1,17 @@
+# Documentation for Python Extensions
+
+This is the documentation for the QtCreator Python Extensions plugin.
+
+## Contents
+ 1. [Writing Python Extensions](./extensions.md)
+ 2. [The C++ PythonExtensionsPlugin](./plugin.md)
+ 3. [Build and Development Tools](./tools.md)
+
+## General overview
+
+Please refer to the section \`How it works' in the main [README.md](../README.md) file.
+
+## A Note regarding Python versions
+In principle, the C++ plugin supports both Python 2 and 3 (at least for the versions tested, i.e.
+2.7 and 3.5). However, the extensions themselves might only support one version of Python. This is
+currently the case for the extension manager.
diff --git a/docs/extensions.md b/docs/extensions.md
new file mode 100644
index 0000000..eda9c4c
--- /dev/null
+++ b/docs/extensions.md
@@ -0,0 +1,131 @@
+# Documentation for Python Extension Authors
+
+This document is intended to provide an introduction to writing Python extensions
+for the QtCreator.
+
+## Importing QtCreator specific modules
+Currently, the bindings are available from Python under the following names:
+```Python
+from PythonExtension import QtCreator # Imports the generated module for the typesystem
+from PythonExtension import PluginInstance # Imports the plugin instance
+```
+
+## `PythonExtension.PluginInstance`
+This is the Python binding for the extension manager that works on the C++ side. It provides the
+following functions, that can (and should) be used from Python:
+
+```Python
+from PythonExtension import PluginInstance as inst
+
+# Returns a PySide QDir which points to the extension directory
+inst.extensionDir()
+
+# Returns a list with the names of all Python extensions
+# If loaded only is supplied as True, only extensions that
+# where loaded successfully are returned
+inst.extensionList(loadedOnly = False)
+
+# Mark an extension as loaded
+# This should normally not be used and exists mainly
+# for use by the extension manager
+inst.flagAsLoaded("name_of_extension")
+
+# Returns the path of the custom location to
+# where missing dependencies should be pip installed
+inst.pythonPackagePath()
+```
+
+
+# QtCreator bindings
+Generally, the parts of QtCreator that are exposed have an interface that is nearly
+identical to the C++ interface.
+
+## Working with Menus
+You can add new items to the menus of QtCreator. You can either add an action directly, or
+add a new action container, that holds a sub-menu.
+
+### Adding a sub menu
+The following code snippet illustrates how to add a new menu.
+
+```Python
+from PythonExtension import QtCreator
+
+def hello():
+    print("Hello World.")
+
+# By convention, the menuId starts with "Python"
+menuId = "Python.SmallMenu.Menu"
+
+menu = QtCreator.Core.ActionManager.createMenu(menuId)
+menu.menu().setTitle("My menu")
+menu.menu().addAction("My action", hello)
+# Add our new menu to the "Tools" menu in QtCreator
+QtCreator.Core.ActionManager.actionContainer("QtCreator.Menu.Tools").addMenu(menu)
+```
+
+### Adding a new action directly
+The following code snippet illustrates how to add a new action to an existing action container.
+
+```Python
+from PythonExtension import QtCreator
+
+def hello():
+    print("Hello World.")
+
+# Add a new action to the "Tools" menu
+menu = QtCreator.Core.ActionManager.actionContainer("QtCreator.Menu.Tools")
+menu.menu().addAction("My action", hello)
+```
+
+### Menu Id list
+Currently, the following menu ids are available in QtCreator.
+
+```Python
+"QtCreator.Menu.File"
+"QtCreator.Menu.File.RecentFiles"
+"QtCreator.Menu.Edit"
+"QtCreator.Menu.Edit.Advanced"
+"QtCreator.Menu.Tools"
+"QtCreator.Menu.Tools.External"
+"QtCreator.Menu.Window"
+"QtCreator.Menu.Window.Panes"
+"QtCreator.Menu.Window.ModeStyles"
+"QtCreator.Menu.Window.Views"
+"QtCreator.Menu.Help"
+```
+
+
+# The embedded Python interpreter
+
+## Python modules and `sys.path`
+When importing modules, the following important locations will be checked (in this order):
+
+ 1. The folder of the extension itself (files and folders in your extension)
+ 2. The system Python path entries (anything you `pip install`ed globally)
+ 3. The QtCreator specific Python module directory
+    - Note: This is where you should install any dependencies missing
+      if you want to use non-standard Python packages / modules
+    - This last path is accessible with `PluginInstance.pythonPackagePath()`
+
+Any changes you make to sys.path and any modules you import, will be cleared after your script
+finished executing.
+
+## Reserved variable names
+Names that look like
+```Python
+qt_creator_{SOME_NAME}_symbol_mchawrioklpilnjajqkfl
+```
+are reserved for the ExtensionManager. Unless you know exactly what you are doing, you should never
+touch any of these variables.
+
+## Installing dependencies
+There are two ways you can install dependencies. If you need very fine-grained control over how
+extension looks like before it can be run, you can supply a `setup.py` script, which is run
+separately before your extension is started. An example of such a script can be found in the example
+extension `examples/numpysetup`.
+
+If you only want to install dependencies, you can also include a standard Python `requirements.txt`
+file in your extension folder. **Note that this file is pip installed _once_.** After the initial
+pip install, a new file called `requirements.txt.installed` is created in the extensions folder. To
+run the install again, this file simply has to be deleted. **Be careful to remove this file, when
+distributing your extension in a .zip file.**
diff --git a/docs/plugin.md b/docs/plugin.md
new file mode 100644
index 0000000..e94633b
--- /dev/null
+++ b/docs/plugin.md
@@ -0,0 +1,164 @@
+# C++ Plugin Documentation
+
+This document is intended for people who want to understand, maintain or
+extend the C++ part of this plugin.
+
+**NOTE:** The plugins C++ sources are located at `plugins/pythonextensions`.
+
+## Files
+The following files exist as part of the C++ plugin:
+
+ * `pyutil.[h|cpp]`
+   - Contains utilities for working with the embedded Python interpreter
+   - All the direct interaction with Shiboken / Python happens
+     in the .cpp file to prevent a known error
+   - These files were originally based of the ['scriptableapplication'](http://code.qt.io/cgit/pyside/pyside-setup.git/tree/examples/scriptableapplication)
+     example from the PySide project
+ * `pythonextensionsplugin.[h|cpp]`
+   - Contains the IPlugin class and manages the life-cycle of Python
+     extensions
+   - This name is the standard for QtCreator plugins
+ * `pythonextensions_global.h`
+   - standard plugin file
+ * `typesystem_qtcreator.xml`
+   - Typesystem for Shiboken
+   - This file describes the classes and types that are bound to
+     Python
+ * `wrappedclasses.h`
+   - This file contains the headers that Shiboken will parse when generating
+     the bindings
+ * `glue` (directory)
+   - This directory contains any glue code needed for the bindings
+   - Currently it looks like this will mainly be needed to overcome the missing
+     support for C++ function pointers in Shiboken
+
+## Buildsystem
+The following is part of the buildsystem. The buildsystem has so far only been
+tested on Linux.
+
+ * `pythonextensions.pro`
+   - QMake project file
+   - The projects dependencies on the QtCreator side are handled in
+     the file `pythonextensions_dependencies.pri`
+   - The current build system emerged in part from the standard plugin build
+     setup, combined with some .pro code from the 'scriptableapplication' and
+     a lot of bits and pices added during the initial development phase (mainly
+     through trial and error)
+ * `pyside2.pri`
+   - This file is originally based of the 'scriptableapplication'
+     example from the PySide project
+   - It is responsible for detecting the PySide2 configuration
+ * `tools/pyside2_config.py`
+   - This script is a copy of a script found in the PySide2 examples
+   - It detects the PySide2 configuration and is called by the .pri file
+   - **Note:** This script will be officially included as a PySide util in the future.
+
+### Goals of the buildsystem
+The buildsystem aims to achieve two things
+
+ 1. Build the C++ plugin, linking against any relevant libraries
+ 2. Install the generated shared object, as well as copy the
+    `/python` directory into the QtCreator plugin directory. This
+    directory contains the included extension manager, which is implemented
+    as a Python extension
+
+## General architecture of the plugin
+During initialization, the plugin does several things:
+
+ 1. Initialize the Python bindings (this happens during the plugin's `initialize`
+    call)
+    1. initialize the embedded Python
+    2. bind the module object generated by Shiboken
+       - Note that this requires a patch for Shiboken, that
+         exposes this generated object in a way that allows
+         the `PyUtils` functions to access it
+    3. bind the PythonExtensionsPlugin instance itself
+
+ 2. Run the `setup.py` script for every extension that provides it (this happens
+    during the `delayedInitialize` call)
+    - This is run here to not block the QtCreator ui while loading the extensions
+    - It is separated from the main extension runtime, to provide a facility to
+      extensions that depend on Python modules that are not part of the standard
+      Python library (for an example, see `examples/numpysetup` or the extension
+      loader)
+    - This script fails without an explicit warning in the plugin, however, all
+      Python errors are printed to stdout, so they can be inspected there during
+      extension development
+    - Note that the extension manager attempts to run an extensions `main.py`
+      script after installing it, but does not consider the `setup.py` script.
+      If `main.py` than fails to run, the extension manager advises the user to
+      restart QtCreator
+    - There is currently no facility for scripts to detect if they have loaded
+      before their setup was called
+    - Because the setup is run every time, the setup script needs to detect if
+      e.g. attempting to install dependencies is necessary (e.g. trying to import
+      them)
+
+ 3. Attempt to run the `main.py` script for every extension (this happens
+    during the `delayedInitialize` call)
+    - An extension that does not provide a `main.py` is considered to have failed
+      loading
+    - An extension for which the `main.py` fails to run is considered to have not
+      loaded
+    - Provided `main.py` succeeds, the plugin is
+      considered to have loaded (even if the setup script failed)
+
+The extension manager provides functions to find the extension directory, list
+all installed extensions and list all loaded extensions. (The loaded extensions
+are tracked by the PythonExtensionPlugin instance.) For details on the signatures
+of these functions, see `pythonextensionsplugin.h`.
+
+The plugin directories that are searched for the extension directory (currently
+`/python`) are determined using `ExtensionSystem::PluginManager::pluginPaths()`.
+
+## Important points / workarounds for problems regarding the various dependencies
+
+When executing Python code, be sure that Python was initialized (this is typically
+done by the provided helper functions). Usually, any extension code should be
+executed using `runScriptWithPath()` (never use `runScript()`, as it does not
+provide any facilities for separating extensions).
+
+The util function `runScriptWithPath()` runs a supplied script in a way that
+allows to import Python packages / modules from the provided path. It also
+isolates the script by cleaning the module namespace (and removing all
+modifications to sys.path) after the script finishes execution. Currently this is
+achieved by wrapping the supplied script with some special Python code, that
+implements these precautions. Because of this, no variables which have
+`mchawrioklpilnjajqkfl` as part of their name should be used in extension
+scripts. This function is used for both executing `setup.py` and `main.py` and
+should be used whenever executing user supplied scripts.
+
+The Python initialization needs to manually link the Python shared library due
+to a bug in CPython. This is done in `init()` (`PyUtils.cpp`) and has some
+facilities for detecting the required version and linking against the correct
+shared object. However, this needs to be more extensively tested and will probably
+have to be amended with several special cases for different platforms / versions.
+
+In the `PyUtil.cpp` file, the macros `signals` and `slots` are undefined due to
+compatibility reasons with the CPython headers. Due to this incompatibility, any
+QtCreator code and Shiboken code needs to be well separated. (These symbols need
+to be undefined **BEFORE** including anything Python related.)
+
+The glue code for the MacroExpander functions `registerVariable()` and `registerPrefix()`
+stores a reference to a Python object that can be called from other functions of the
+MacroExpander. For this to work, there may not be an unmatched call to `PyEval_SaveThread()`
+in any of the bindings for these functions. To achieve this, all those functions are modified
+using a code injection in the `typesystem_qtcreator.xml`.
+
+### My build suddenly starts seg-faulting for no apparent reason
+**DON'T PANIC**, did you change the bindings? Check if the value of
+```
+SBK_PYTHONEXTENSIONS_INTERNAL_PYTHONEXTENSIONSPLUGIN_IDX
+```
+changed. (See pyutil.h and the `..._python.h` in the generated bindings.)
+
+## Thinks that could be broken on other platforms
+
+This is _not_ an exhaustive list, _nor_ is it backed by any actual experience with
+building this project on non-linux operating systems. However, the following things
+are quite hacky and might not work on other systems:
+
+ * Different Python versions might not work
+ * Wired Python setups might trip it up (e.g. on macOS)
+   - Especially watch out for the manually linked Python in pyutils.cpp
+ * The pip install stuff might not select the correct version of Python on every machine
diff --git a/docs/tools.md b/docs/tools.md
new file mode 100644
index 0000000..e7e6612
--- /dev/null
+++ b/docs/tools.md
@@ -0,0 +1,43 @@
+# Documentation regarding included tools / utilities
+
+This project includes a few Python scripts, included as utilities for convenience. All of these are
+relatively small and self-contained. They should also all include usage instructions in comments at
+the beginning of the scripts. Some of them are scattered around the project. These include:
+
+ * [The optional bindings build script](../optional/setup.py)
+   - This script is intended to help building the optional bindings, by bundling files needed by
+     all optional binding libraries in a `template` directory.
+   - **Available options are:**
+     - --qmake=/path/to/your/qmake *required*
+     - --only=binding_dir *optional, only build binding_dir*
+     - clean *optional, ignores all options and removes the build directories*
+
+ * [The example packaging script](../examples/package.py)
+   - This script creates .zip files for use with the included extension manager form the example
+     extension sources.
+   - **Available options are:**
+     - *run with no arguments in `examples` directory to create extension .zip files*
+     - clean *remove generated .zip files*
+   - **Note:** There is a symbolic link to this script in the `tests` directory
+
+The other scripts are in the [tools folder](../tools/), they are:
+
+ * [The license script](../tools/license.py)
+   - This script adds the Qt license to all relevant source files.
+   - Takes no options and should be run in root directory with `$ python ./tools/license.py`.
+
+ * [PySide build helper script](../tools/pyside2_config.py)
+   - This is taken verbatim from the pyside-setup project and is used to discover the pyside
+     installation / setup during builds. Should not be used manually.
+
+ * [Sanity helper script](../tools/sanity.py)
+   - This script automatically corrects some annoying nags for the Qt Gerrit sanity check.
+   - Takes no options and should be run in root directory with `$ python ./tools/sanity.py`.
+
+ * [Build helper script](../tools/build.py)
+   - This script executes all the build scripts in this project in the correct order
+   - Mainly for lazy people and only usable if everything is setup correctly
+   - **Available options are:**
+     - --qmake=/path/to/your/qmake *required*
+     - --skip *optional, only build main c++ plugin*
+     - --user *optional, build to user directory*