The Python interpreter is able to load extensions from dynamic/shared libraries such as Python modules if they provide an applicable interface using Python/C API. This API must be incorporated in a source code of extension using a Python.h C header file that is distributed with Python sources. In many distributions of Linux, this header file is contained in a separate package (for example, python-dev in Debian/Ubuntu) but under Windows, it is distributed by default with the interpreter. On POSIX systems (for example, Linux and macOS), it can be found in the include/ directory of your Python installation. On Windows, it can be found in the Include/ directory of your Python installation.
Python/C API traditionally changes with every release of Python. In most cases, these are only additions of new features to the API so are generally source-compatible. Anyway, in most cases, they are not binary compatible due to changes in the Application Binary Interface (ABI). This means that extensions must be compiled separately for every version of Python. Also, different operating systems have incompatible ABIs, so this makes it practically impossible to create a binary distribution for every possible environment. This is the reason why most Python extensions are distributed in source form.
Since Python 3.2, a subset of Python/C API has been defined to have stable ABIs. Thanks to this, it is possible to build extensions using this limited API (with a stable ABI), so extensions can be compiled only once for a given operating system and it will work with any version of Python higher or equal to 3.2 without the need for recompilation. Anyway, this limits the number of API features and does not solve the problems of older Python versions. It also does not allow you to create a single binary distribution that would work on multiple operating systems. So this is a trade-off and the price of the stable ABI seems to be a bit high for a very low gain.
It is important to know that Python/C API is a feature that is limited only to CPython implementations. Some efforts were made to bring extension support to alternative implementations such as PyPI, Jython, or IronPython, but it seems that there is no stable and complete solution for them at the moment. The only alternative Python implementation that should deal easily with extensions is Stackless Python because it is in fact only a modified version of CPython.
C extensions for Python need to be compiled into shared/dynamic libraries before they can be imported because there is no native way to import C/C++ code in Python directly from sources. Fortunately, distutils and setuptools provide helpers to define compiled extensions as modules, so compilation and distribution can be handled using the setup.py script as if they were ordinary Python packages. The following is an example of the setup.py script from the official documentation that handles the preparation of simple package distribution that has some extension written in C:
from distutils.core import setup, Extension
module1 = Extension(
'demo',
sources=['demo.c']
)
setup(
name='PackageName',
version='1.0',
description='This is a demo package',
ext_modules=[module1]
)
Once prepared this way, the following additional step is required in your distribution flow:
python setup.py build
This step will compile all your extensions defined as the ext_modules argument according to all additional compiler settings provided with the Extension() constructor. The compiler that will be used is the one that is a default for your environment. This compilation step is not required if the package is going to be distributed as a source distribution. In that case, you need to be sure that the target environment has all the compilation prerequisites such as the compiler, header files, and additional libraries that are going to be linked to your binary (if your extension needs any). More details of packaging the Python extensions will be explained later in the Challenges with using extensions section.
In the next section, we will discuss why we need to use extensions.