Python 3.12 Preview: Subinterpreters

With the upcoming release of Python 3.12 this fall and Python 3.13 following a year later, you might have heard about how Python subinterpreters are changing. The upcoming changes will first give extension module developers better control of the GIL and parallelism, potentially speeding up your programs.

The following release may take this even further and allow you to use subinterpreters from Python directly, making it easier for you to incorporate them into your programs!

In this tutorial, you’ll:

• Get a high-level view of what Python subinterpreters are
• Learn how changes to CPython’s global state in Python 3.12 may change things for you
• Get a glimpse of what changes might be coming for subinterpreters in Python 3.13

To get the most out of this tutorial, you should be familiar with the basics of Python, as well as with the global interpreter lock (GIL) and concurrency. You’ll encounter some C code, but only a little.

You’ll find many other new features, improvements, and optimizations in Python 3.12. The most relevant ones include the following:

• Ever better error messages
• Support for the Linux perf profiler
• More powerful f-strings
• Improved static typing features

Go ahead and check out what’s new in the changelog for more details on these and other features. It’s definitely worth your time to explore what’s coming!

What Are Python Subinterpreters?

Before you start thinking about subinterpreters, recall that an interpreter is a program that executes a script directly in a high-level language instead of translating it to machine code. In the case of most Python users, CPython is the interpreter you’re running. A subinterpreter is a copy of the full CPython interpreter that can run independently from the main interpreter that started alongside your program.

Most of the state of the subinterpreter is separate from the main interpreter. This includes elements like the global scope name table and the modules that get imported. However, this doesn’t include some of the items that the operating system provides to the process, like file handles and memory.

This is different from threading or other forms of concurrency in that threads can share the same context and global state while allowing a separate flow of execution. For example, if you start a new thread, then it still has the same global scope name table.

A subinterpreter, however, can be described as a collection of cooperating threads that have some shared state. These threads will have the same set of imports, independent of other subinterpreters. Spawning new threads in a subinterpreter adds new threads to this collection, which won’t be visible from other interpreters.

Some of the upcoming changes that you’ll see will also allow subinterpreters to improve parallelism in Python programs.

Subinterpreters have been a part of the Python language since version 1.5, but they’ve only been available as part of the C-API, not from Python. But there are large changes coming that will make them more useful and interesting for everyday Python users.

What’s Changing in Python 3.12 (PEP 684)?

Now that you know what a Python subinterpreter is, you’ll take a look at what’s changing in the upcoming releases of CPython.

Most of the subinterpreter changes are described in two proposals, PEP 684 and PEP 554. Only PEP 684 will make it into the 3.12 release. PEP 554 is scheduled for the 3.13 release but hasn’t been officially approved yet.

Changes to the Global State and the GIL

The main focus of PEP 684 is refactoring the internals of the CPython source code so that each subinterpreter can have its own global interpreter lock (GIL). The GIL is a lock, or mutex, which allows only one thread to have control of the Python interpreter. Until this PEP, there was a single GIL for all subinterpreters, which meant that no matter how many subinterpreters you created, only one could run at a single time.

Moving the GIL so that each subinterpreter has a separate lock is a great idea. So, why hasn’t it been done already? The issue is that the GIL is preventing multiple threads from accessing some of the global state of CPython simultaneously, so it’s protecting your program from bugs that race conditions could cause.

The core developers needed to move much of the previously global state into per-interpreter storage, meaning each interpreter has its own independent version: