It’s a seemingly basic question that we did not have an answer to, up until very recently.

So, I did what any sane person would do! Naturally, I set up four different runners across different architectures and operating systems, and created a website to answer the most important question: does JIT go brrr?

Building a server farm (accidentally)

I didn’t really plan to scope creep things this much. I started with just a single Raspberry Pi running the benchmark suite nightly, around the same time I set up my RPi buildbot. But, one thing led to another, and all of a sudden I’m running a small server farm in my closet (which isn’t totally off brand for me as I have been known to roll my own infra from time to time!).

In any case, I was also motivated to set this up so that we could have a reliable pipeline set up to collect benchmark stats on consumer-grade hardware. We have access to beefy, server-grade machines via corporate sponsors, but those aren’t necessarily representative of a basic laptop or a Raspberry Pi. I figured it could be pretty interesting to see how things were going. So I took inventory of the random machines lying around the house, and ended up with this suite of machines:

…and before you ask, yeah, the majority of these machines have names related to the Alien franchise (I’m a big fan!). Blueberry is just blueberry, because Pi 🥧!

How the benchmarking stack works

Okay, so that’s roughly the hardware setup but before diving into the software side of things, it helps to understand how the pieces fit together. Thankfully, this is where I could lean on existing projects like bench_runner.

bench_runner is really the tooling or framework for how this all comes together. I’m not going to delve too deep into the setup here since it’s well documented in the repo’s README but really this is what handles actually running the pyperformance benchmark suite on self-hosted GitHub Actions runners. It calculates the improvements, handles all the options to enable optimization flags, generates comparison plots and result tables.

For context, pyperformance is the benchmark suite focused on real-world benchmarks rather than synthetic ones. This matters because we want to measure how actual Python code performs, not just micro-optimizations that don’t translate to real applications.

bench_runner also relies on your having configured a results repository, in my case: pyperf_bench. This is where all the results, plots, stats etc. get stored after each run. Workflows are triggered from this repo, a self-hosted runner picks it up and uses bench_runner to build CPython and run benchmarks, and then results are committed back here.

Setting up self-hosted runners

The actual runner setup was surprisingly straightforward — GitHub’s self-hosted runner documentation walks you through it. You’re basically downloading a runner application, configuring it with a token from your repo, and running it as a service. The runner then polls GitHub for jobs that match its labels.

The trickier part is getting the benchmarking environment right. Each machine needs:

• Build dependencies for compiling CPython from source (gcc/clang, make, libssl-dev, etc.)
• LLVM installed (the JIT needs it for compilation)
• bench_runner and its dependencies
• A system relatively free from background process noise…

That last point is where things get fun. Benchmarking is all about reducing variance, and consumer hardware loves variance. You really want stable, reproducible results, which means minimizing anything that could interfere with CPU cycles while benchmarks are running.

Each machine had its own quirks (non-exhaustive):

On the Pi, thermal throttling meant I needed a better cooler. The Pi 5 starts throttling at 80°C and gets more aggressive at 85°C—when that happens, the CPU drops from 2.4GHz down to 1.5GHz, which would absolutely tank benchmark consistency. I also had to fiddle with IRQ affinity permissions since I wanted to pin benchmark processes to specific CPU cores to reduce noise.

Windows had a parade of permission errors…and literally everything on Windows doesn’t work the way I’d expect so that debugging was pretty unfun.

And macOS? aargh, mediaanalysisd. If you ever decommission a MacBook and decide to use it as a GitHub Actions runner, make sure you’ve completely turned off iCloud syncing. Otherwise this friendly background daemon will come along scanning photos for AI features like face recognition and object detection, eating CPU cycles for hours and completely wrecking your benchmark stability.

All this to say, it’s not too tricky but expect some friction and debugging if you try to set one up yourself!

The dashboard pipeline

Here’s what happens every night:

• Each runner runs the benchmark suite twice: once for the interpreter, once with the JIT enabled, via a GitHub Action on a cron schedule
• The results are compared per machine to derive the speedup
• After all runs finish, I trigger a dashboard refresh which runs a script on another Raspberry Pi cluster (since I also self-host everything 🤠). The script pulls any new results, computes the geometric mean using pyperf, and loads everything into the database.
• Results appear on doesjitgobrrr.com shortly after!

So, what have we learned?

If you recall from various PyCon talks, Brandt has jokingly said the JIT was “0% faster/slower” for quite a while. However, things are actually coming together now! Results are quite dependent on operating system and hardware specs, but preliminary results show that the JIT is about 4-5% faster than the interpreter on x86-64 Linux, and 7-8% faster on AArch64 macOS over the tail-calling interpreter in 3.15, as evidenced by runners sponsored by Meta.

However, my server farm does show more interesting results! On our nightly runs funded by yours truly, we’re regularly seeing:

• ~2% speedups on my Raspberry Pi 5 (blueberry) — aarch64
• ~5% speedups on my i5 running Ubuntu 24.04 (ripley) — x86_64
• ~8% speedups on my M3 Pro (jones) — aarch64
• ~15% speedups on my AMD Ryzen 5 3600X running Windows 11 (prometheus) — x86_64

Beyond tracking nightly progress, the dashboard has also helped us catch performance regressions from PRs the same day they land, which feels awesome! The power of data!

…and to be honest, that’s pretty awesome if you ask me. Yes, we still have work ahead of us, but man, I’m super proud of all the work we’ve done over the past few months that have gotten us this far. The JIT has also become pretty community-driven at this point, and we’re actively building up new contributors! Super exciting stuff!

Reading materials and links

• The unofficial JIT performance dashboard — does JIT go brrr?
• pyperf_bench — where all my data lives from our runs
• doesjitgobrrr on GitHub
• bench_runner — for running pyperformance on GitHub Actions runners
• pyperformance — the benchmark suite
• pyperf — toolkit for Python benchmarking
• PEP 744 — JIT Compilation

One particular area where using containers or Kubernetes adds friction is in debugging workflows. That FastAPI app that you’re running with Docker Compose or Kubernetes? Yeah, the debugger isn’t going to just magically connect after configuring your launch.json file and running it in VS Code anymore. Maybe you’ve ended up setting up some kind of sidecar service to run your debugging in your cluster. Or, maybe you’ve added debugpy to your code, configured it to wait for connections, restarted your pod, set up port-forwarding, and hoped your app is still in the right state by the time you connect. Oh yeah, and throw a uvicorn --reload in there and…aaaaaaa. Setting all of that up is a true headache and honestly, pretty peripheral to the task at hand! You were just trying to debug that app you’re building anyway!

So, I built a tool that I’m calling debugwand - a zero-preparation remote debugger for Python applications running in Kubernetes clusters or Docker containers. With debugwand, there’s no sidecar pod, no application code changes, and virtually no setup required (okay, just a teeny tiny bit…it’s just configuration!).

Containers hate to see me comin'

How, you might ask? This brings me to perhaps the single coolest feature in Python 3.14 that makes this all possible (okay, okay…I’m being dramatic – this is like picking a favourite child!): sys.remote_exec(). With sys.remote_exec(), we can execute a Python script inside another running process all without restarting it. The script runs with full access to the target’s memory, modules, and state, which is exactly what you need to start a debug server on-demand.

So what does debugwand do?

When you run wand debug, the CLI goes out and:

1. Finds your target - For Kubernetes, it discovers pods for your service (and handles Knative, because of course you’re using Knative). For Docker, you just point it at a container.

2. Picks a process - It finds all the Python processes running in the pod/container with CPU and memory stats, so (tries to) choose the right one. Running uvicorn --reload? It’ll detect that automatically too (you can always override the choice if you want to debug a different process).

3. Injects debugpy - Here’s where sys.remote_exec() does its thing. debugwand writes a small script that starts a debugpy server and injects it into your running process. No restart. No code changes. Your app keeps serving requests like nothing happened.

4. Sets up the connection - For Kubernetes, it handles port-forwarding automatically. For Docker, you just need to expose the port when you start the container.

5. You connect your editor - Point VS Code (or nvim, or whatever DAP client you like) at localhost:5679 (you can choose whatever port you’d like as well) and you’re debugging. Set breakpoints, inspect variables, step through code - the works.

The whole flow to get everything set up should only take a few minutes to configure things and then you should be off to the races! Pretty neat, huh?

You can try it out

A quick note: This tool is experimental and intended for local development. Enabling SYS_PTRACE capabilities has security implications you’ll want to consider before using this anywhere near production (don’t do that!).

While I built this primarily out of curiosity and for our workflow at work, you’re welcome to give this a shot! If you’re running Python 3.14, give it a spin by uv tool install debugwand (see PyPI here).

After that, you’ll need to:

• Ensure your cluster/container has SYS_PTRACE kernel capabilities enabled. Remember! Local development only, please!
• Make sure you’ve port forwarded 5679 (or whatever port you want to use)
• Set up your launch.json if you’re using VS Code.

Then it’s just:

# Kubernetes
wand debug -n my-namespace -s my-service

# Docker 
wand debug --container my-container

And then you’re off to the races! Just start your debugger and set some breakpoints!

Reading materials and links

• Remote debugging attachment protocol
• debugwand on GitHub
• debugwand on PyPI

Python was my first programming language, learned because a mentor at my first internship suggested I could script away all the button clicks I was doing in ArcGIS.

Python was my first and second job out of college, where I automated drone flights to try to shoot tree seeds out of pneumatic tubes for reforestation (the startup has since pivoted!) and built a data pipeline for processing geospatial data in a mapping application used by millions on their phones.

Python was the epicenter of my career pivot to product management, where I learned that I love working on developer tools.

Python was where I met my now husband, the coolest and smartest person I know.

Python was where I found my confidence that it was not just okay that I don’t have a degree in engineering but actually welcomed and cool that I didn’t.

Python was where I found my community and made friends that live all over the world. Under any other circumstances, I may never have met these people but I’m so lucky that Python gave me this opportunity.

I guess I’m saying all this because maybe when people hear me say that Python is really important to me, they think I mean it purely from a technical perspective. While it’s true that I think Python is an amazing programming language, it’s really a lot more than that to me. There’s this swirly, amazing mess of social, technical and emotional factors that keep me coming back to Python.

Running for a seat on the Steering Council is honestly not something I even imagined myself doing. Earlier this year in my EuroPython keynote, I spent 45 minutes telling the audience that I had impostor syndrome. Okay, it wasn’t quite that simple but really the talk was about coming from a non-traditional background and figuring out how to contribute to CPython. During the talk, I had a couple calls to action for attendees. The first was to find your community and to surround yourself with people who encourage you to do things outside your comfort zone. In this case, I am forever grateful to Pablo for nudging me over text three days before the nomination deadline with a “hey, crazy idea, what if you ran for the steering council?”. The second call to action was something I try to apply to many things in my life, which is to “do it, scared”. Running for a seat on the Steering Council was definitely a “doing it scared” moment for me.

All this is to say that I’m really excited to have this opportunity over the next year or so. I had all of this in more detail in my nomination statement but thematically, my goals right now are: 1) ensure that all the right bits come together to make Python 3.15 the fastest Python ever (and lay the groundwork for future versions to be even faster), and 2) support making CPython and related projects under python/ the most welcoming and healthy projects in open source. That involves supporting today’s contributors by making things like the PEP process less arduous, and extends to new contributors by understanding and addressing barriers to entry. In my mind, both of these areas are absolutely table stakes for making sure Python’s future is as bright as can be.

Anyway, I just wanted to write this all down because I have feelings™️ but I’m sure many, many other things will pop up. I’m sure that I’ll learn a lot from my fellow Steering Council members, the core team, and the community.

Thanks for reading. I love Python! Python is forever!

This is a post in a series around making CPython internals more approachable. If I missed something or you’d like to request a topic, feel free to drop me a line via email. You can also read other posts in the series here.

So, this post is going to be potentially probably very niche, but I just spent a lot of time going down a rabbit hole working on CPython’s JIT LLVM 20 upgrade, and I learned a thing or two and figured maybe other folks might be interested as well. I am also a firm believer in teaching to solidify your learning, so I decided to write this post. It’ll be one part diary entry to commemorate a sort of wild debugging chain at the Paris airport (twice!!), and one part educational!

Cool, cool…alright, so I’ve worked on our LLVM upgrades for the JIT for our last three version bumps. When you think about updating a dependency, you might think, “go into some manifest file, change a 1 to a 2, and boom, you’re done.” However, it’s become somewhat of a running joke amongst the team that I take on this seemingly trivial task and then end up opening a can of worms because there is almost always some catch or some weird thing that breaks.

This time around…there was a new worm in the can…and it was called a trampoline. But before we get into that, I want to walk you through how I stumbled upon doing this work and why it was necessary for this version of LLVM.

But first, more about stencils

If you recall from this post, before Python even runs with the JIT, we generate stencil files at build time using LLVM. These stencil files are really just a very long list of C functions (one per bytecode) compiled into machine code templates. The stencil file contains the raw machine code bytes for each operation, relocation information (where we need to patch things at runtime), which symbols (external functions) each stencil needs, and which stencils need trampolines.

Now, you might be asking, “Savannah, I want to see a stencil file,” and then I’d tell you that you’ll have to build Python with the JIT enabled. Or, if you’re less inclined, you can see this stencil for the LOAD_FAST instruction. Gibberish? That’s okay. The important part is that at runtime, we compile traces for a sequence of instructions that represent your program. For each operation, we get its precompiled stencil, copy it into executable memory, patch in the actual addresses we need, and then combine everything together into runnable machine code.

Now, when we generate these stencils, we use a bunch of compiler flags to tell LLVM how we want the stencils generated. You don’t need to care too much about the specific flags or what they all mean at this point, but there are a lot of them. In a way, it’s kind of like passing some kind of special incantation to the compiler…or at least it feels like that when things are broken and you are hitting random assertion errors in our instructions like I was.

Alright, now let’s talk about upgrading to LLVM 20!

So this time, when I bumped the version, everything worked, except on x86_64 Darwin debug builds!

Curious, right…well, there are a couple of reasons for this. For one, we were previously using a compiler flag (-fno-plt), which I discovered through some trial and error (and compiler warnings) isn’t supported on macOS in LLVM 20, so that had to go. Second, our debug builds are not optimized, so code can be naturally further apart in memory than in release builds. Without optimizations, the compiler doesn’t pack things tightly, which means our generated machine code and the runtime symbols it needs to call can end up separated by more than 2GB in memory. So basically, when I hit the assertion error saying that patch_32r for x86_64 was more than 2GB away, I had two options: 1) the simpler option - find the right combination of compiler flags to make it work (-mcmodel=medium,large; -fno-pic etc.) or 2) implement a trampoline (we will get back to this in a second).

So naturally, I tried the simplest option first. I started down this rabbit hole in earnest during a layover at 8 am in the Paris airport on my way to Spain for a team offsite, delirious and running on about 45 minutes of sleep.

Unfortunately, that didn’t work!

Live footage of me passing -mcmodel=large into the compiler for the fifth time, hoping it fixes all my problems

So, I gave up…not really…But I did take a week break for the offsite and PyCon Spain.

However, on the way home from Spain, I had six whole hours in the Paris airport during my layover to get back at it. Thinking more about this problem, I remembered that Diego had implemented trampolines for aarch64 a while back. I decided to give up on compiler flags and really go down the rabbit hole. What ensued next was a long while of reading x86_64 instruction encoding to figure out how I could maybe handwrite x86-64 machine code byte-by-byte…a dark, dark place.

Okay, so what the heck is a trampoline…

Okay, she’s said the title of the blog post. We must be getting close! Yes, okay…so this is what a trampoline is - it’s a small piece of code that acts as a bridge to another place in memory, some place we cannot directly reach¹. In this case, we need to access some symbol that’s more than 2GB away from where we currently are in memory.

In the stencil snippet I linked to above, you’ll notice some lines that say patch_x86_64_trampoline. Let’s look at the actual patch instruction for the trampoline and walk through it.

// Generate and patch x86_64 trampolines.
void
patch_x86_64_trampoline(unsigned char *location, int ordinal, jit_state *state)
{
    uint64_t value = (uintptr_t)symbols_map[ordinal];
    int64_t range = (int64_t)value - 4 - (int64_t)location;

    // If we are in range of 32 signed bits, we can patch directly
    if (range >= -(1LL << 31) && range < (1LL << 31)) {
        patch_32r(location, value - 4);
        return;
    }

    // Out of range - need a trampoline
    unsigned char *trampoline = get_trampoline_slot(ordinal, state);

    /* Generate the trampoline (14 bytes, padded to 16):
       0: ff 25 00 00 00 00    jmp *(%rip)
       6: XX XX XX XX XX XX XX XX   (64-bit target address)

       Reference: https://wiki.osdev.org/X86-64_Instruction_Encoding#FF (JMP r/m64)
    */
    trampoline[0] = 0xFF;
    trampoline[1] = 0x25;
    memset(trampoline + 2, 0, 4);
    memcpy(trampoline + 6, &value, 8);

    // Patch the call site to call the trampoline instead
    patch_32r(location, (uintptr_t)trampoline - 4);
}

Here’s what’s happening:

First, we get the value - this is the memory address of the actual symbol we need to jump to. The ordinal is just an index that identifies which symbol we’re dealing with (like “symbol #5” or “symbol #37”). A symbol is an external function that the JIT-compiled code needs to call at runtime, things like PyObject_GetAttr (to get an attribute from a Python object) or PyDict_GetItem (to get an item from a dictionary). Every external function is assigned an ordinal number.

Then we calculate the range, which is the distance in memory between our current location (location) and where we want to reach (value). Next, we check if the symbol is reachable - in this case, within 2GB in memory (this is the assertion that failed in the first place!). If it’s reachable, we just use the standard patch function, patch_32r, and we’re done.

If not, we need a trampoline! We call get_trampoline_slot to get a memory location for our trampoline.

Then we generate the trampoline itself; this is where I had to read some wild x86-64 encoding instructions to figure out the machine code for a jump instruction. The trampoline is just 14 bytes (padded to 16): the first 6 bytes are the jump instruction (jmp *(%rip)), and the next 8 bytes are the 64-bit address we’re jumping to.

Finally, we patch the original call site to point to the trampoline instead of trying to reach the far-away symbol directly.

Let’s go down the rabbit hole a bit more and dissect the get_trampoline_slot function. This is sort of clever, so bear with me.

The problem we’re solving: we have a pool of trampolines (basically an array of memory slots), and we need to figure out “for symbol X, which slot in the pool should I use?” However, not every symbol needs a trampoline. Remember, we only need trampolines when the symbol is too far to reach. So we can’t just use the symbol’s ordinal as the array index, because that would waste a ton of memory. If we have 100 symbols but only 5 need trampolines, we don’t want to allocate 100 trampoline slots!

The solution is to use a bitmask to track which symbols need trampolines and calculate their slot positions on the fly. Bitmasks kind of break my brain a bit, so I’ll try to break this down for folks who feel similarly 😭

// Get the trampoline memory location for a given symbol ordinal.
static unsigned char *
get_trampoline_slot(int ordinal, jit_state *state)
{
    const uint32_t symbol_mask = 1 << (ordinal % 32);
    const uint32_t trampoline_mask = state->trampolines.mask[ordinal / 32];
    assert(symbol_mask & trampoline_mask);

    // Count the number of set bits in the trampoline mask lower than ordinal
    int index = _Py_popcount32(trampoline_mask & (symbol_mask - 1));
    for (int i = 0; i < ordinal / 32; i++) {
        index += _Py_popcount32(state->trampolines.mask[i]);
    }

    unsigned char *trampoline = state->trampolines.mem + index * TRAMPOLINE_SIZE;
    assert((size_t)(index + 1) * TRAMPOLINE_SIZE <= state->trampolines.size);
    return trampoline;
}

Here’s the key idea: we need to answer “how many trampolines come before this symbol?” The answer tells us which slot in our array to use. We store our bitmask as an array of 32-bit integers, where each bit represents whether a symbol needs a trampoline. Let’s walk through an example. Say we have symbols 3, 7, 15, 37, and 42 that need trampolines (the only bits set to 1 in our bitmask). If we’re looking for symbol 37’s slot:

Step 1: Find which bit represents our symbol
Symbol 37 is bit 5 in the second 32-bit chunk (37 % 32 = 5). We create a mask with just that bit set: 1 << 5.

Step 2: Count all trampolines in earlier chunks
Symbol 37 is in chunk 1 (37 / 32 = 1, integer division), so we need to count all set bits in chunk 0. That’s symbols 3, 7, and 15 - giving us 3 trampolines. We use _Py_popcount32, which counts set bits in a 32-bit integer.

Step 3: Count trampolines before us in our own chunk
We use (symbol_mask - 1) to create a mask of all bits lower than ours (for bit 5, this gives us bits 0-4), then AND it with the trampoline mask to see which of those lower bits are actually set, then count them. In our example, there are no symbols between 32 and 37 that need trampolines, so this adds 0.

Step 4: Calculate the final position
Symbol 37 gets index 3 (the 4th slot, since we’re 0-indexed). We take our base memory address (state->trampolines.mem) and add index * TRAMPOLINE_SIZE to get to the right slot.

Break your brain a bit? Same! However, the beauty of this approach is that our trampoline pool is densely packed. We only allocate as many slots as we actually need, but we can still quickly calculate where any symbol’s trampoline lives.

Putting it all together

So, TL;DR, this is what’s happening:

Original code:     [call instruction] --X--> [target] (too far, >2GB away)

With trampoline:   [call instruction] -----> [trampoline] -----> [target]
                                          |                   	 |
                                          | jmp *(%rip)          |
                                          | [64-bit address] ----+

The trampoline is just a tiny piece of code that lives close enough to our call site (within 2GB) that we can reach it with a normal 32-bit relative jump, and it contains an instruction that can jump to the full 64-bit address of our actual target.

And that’s that!

If you made it this far, congrats! You now know way more about trampolines than you probably ever wanted to, and hey! You didn’t need to be a compiler engineer to understand it (hopefully!)

If you enjoyed this post, please consider sharing it with anyone you think might find it interesting. If you have any questions or feedback, feel free to reach out to me via email.

TL;DR I’m joining the engineering team at FastAPI Labs ⚡ and closing this chapter of my career in product management. Read on for more!

Building for builders

A little over five years ago, I shared on social media that I was transitioning from engineering to product management. Since then, I’ve had the privilege of working on developer tools at Microsoft, Docker, and most recently at Snowflake.

What I have loved the most about working in developer tools product management is that I got to stay technical and in the weeds, all while deeply focusing on user experience. I was able to participate in engineering conversations about design and implementation, even though I wasn’t writing production code, because lower-level design choices directly impacted the end-user experience. Despite having a recurring existential crisis that I missed working as an engineer full-time, I continued to work as a product manager because I loved it and was good at it (and my open source work scratched the true engineering itch whenever I needed it).

I’ve learned a great deal by working as a product manager, and I genuinely believe that I’ve become a better engineer through this experience than I would have been if I had stayed an engineer in name only. There’s also a long series of personal and professional events that followed the choice to try product management, for which I am eternally grateful. If I could go back in time, I would choose this path a thousand times over.

However, as time has gone on, I’ve felt myself getting further from the parts of the role that gave me the most energy, which, as it turns out, are the engineering bits. I also always said that the title was irrelevant to me. Of course, levels (e.g., senior, staff, principal) do matter, as they help garner a certain level of respect, but titles like software engineer or product manager never held much significance for me.

What I know is that I want to build things - specifically, tools and experiences for builders. For me, that has always meant two things: a deep commitment to developer experience and a connection to Python and open source. The products and projects I am proudest of are those that have genuinely made developers’ lives easier, removed friction, and allowed people to spend more time creating (or, frankly, just giving them the agency to choose what they want to spend their time on). I think often when we’re thinking about scaling quickly or signing the biggest deal, it’s easy to lose sight of what delightful looks like. However, when we ignore too many papercuts, we create an open wound. We make our products unusable and fail to solve the problem in front of us.

What’s important to me is remembering and emphasizing the humanity of what we’re building. I want to focus on solving problems for people. I want to spend the time talking to developers about how their workflow sucks, I want to fix that tiny bug that will make them more productive, and I want to deeply understand user workflows and tooling to the point where I can anticipate what better looks like before they might be able to articulate it themselves.

I want to take the “less scalable” path if it means that I can solve the problem correctly the first time. I sincerely believe that if you build the right experience and truly solve the problem, the people (and the money) will come.

One key problem that still has not been truly solved for Python developers is the journey from local development environment to a deployed application (with all the bells and whistles). Developers will spend hours setting up everything locally, only to face infrastructure, cloud jargon, and difficult decisions. This is a problem I’m all too familiar with, as both an engineer and someone who worked on tooling for Azure. This problem persists, especially in an era where an increasing number of new developers are choosing Python as their language of choice (just look at the latest JetBrains Python Developer Survey). I believe there’s still a significant developer experience pain point here.

And so, this all leads me to the title of this post: in a couple days, I’m starting a new chapter.

On October 6th, I’m joining the engineering team at FastAPI Labs to help build the future of deploying and managing FastAPI applications 🚀.

So why FastAPI Labs? Why now?

FastAPI is now the most popular Python web framework, so fixing this problem is both timely and impactful. More than that, I believe that FastAPI Labs is set up to build this the right way from the start, both in terms of working toward the right solution to the problem and the company ethos. You can read more about the FastAPI Cloud announcement here, or better yet, join the waiting list.

I first met Sebastián Ramírez (aka tiangolo) on Twitter about five years ago when I was working on the Pylance language server. Even then, what stood out to me was his attention to detail, his relentless focus on developer experience, and his understanding of the importance of open source in our community. It was always clear in his work that he didn’t just care about making something work but about making it delightful. So when Sebastián approached me about joining FastAPI Labs, it was a no-brainer. The chance to work alongside someone whose values resonate so profoundly with my own, on a problem that matters, at the right time, was an easy yes. I am absolutely stoked to be joining the team at such an exciting time!

I’m also grateful that part of my role includes dedicated time to contribute to open source. I’ll finally have the space to focus more deeply on my work on CPython, preparing for my role as Release Manager for Python 3.16/3.17, advancing CPython performance work (e.g., ensuring FastAPI runs smoothly on JIT/free-threaded builds), and continuing to support FastAPI and other related projects in the ecosystem. I feel tremendously fortunate to have the opportunity to work at a company that not only understands my open source work but also celebrates it as core to its own mission. This is something I never really thought I’d have the opportunity to do, so I’m incredibly excited.

So, that’s it. With one chapter ending, another one begins.

Savannah 🤝🏻 FastAPI Labs

…and yes, in case you were wondering, the subheader of this blog post is indeed a Charli xcx reference. Always.

This is a post in a series around making CPython internals more approachable. If I missed something or you’d like to request a topic, feel free to drop me a line via email. You can also read other posts in the series here.

Ever wonder what really happens under the hood when you run your Python code? If you’re using a JIT build of CPython, the answer may involve a few more steps than you’d expect but thankfully, you don’t have to be a compiler engineer to understand it.

Before I get into it, I want to shamelessly plug that you can help us test JIT builds of CPython pretty easily as of Python 3.14! You can now get official Python builds from python.org for both Windows and macOS that include CPython’s experimental just-in-time (JIT) compiler built in but off by default. While the JIT builds are not (yet) recommended for production use, you can enable the JIT using the PYTHON_JIT=1 environment variable. We’d love to hear about your experience using Python with the JIT - the good, the bad, the ugly!

Alright, let’s get after it.

What happens when you execute your code: A brief overview of CPython’s interpreter

…Well, before we get into what happens in a JIT build of Python, we should probably briefly talk about what happens in a “regular” build for anyone that isn’t familiar with how the interpreter works, as this lays the foundation for the JIT builds later on. I think the best way to talk about this is by example. To cover this, let’s consider this very basic function:

def abs(a: int, b: int) -> int:
    if a > b:
        return a - b
    return b - a

So, let’s say you execute this with your local version of Python and boom, the code is running. But what actually happens here?

First, your code is broken down into tokens

When you run this code, the first thing that happens is that Python breaks it down into tokens. Tokens are the smallest units of meaning in your code, like keywords, identifiers, literals, and operators. For example, in our function, def, abs, (, a, b, if, >, return, and so on are all tokens. This process is known as lexical analysis or tokenization. You can see what the tokens for our function look like using the tokenize module in the standard library. Here’s how you can do that:

import tokenize
from io import BytesIO
source = b"""
def abs(a: int, b: int) -> int:
    if a > b:
        return a - b
    return b - a
"""
tokens = tokenize.tokenize(BytesIO(source).readline)
for token in tokens:
    print(token)

This will output a list of tokens that look something like this:

TokenInfo(type=65 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='')
TokenInfo(type=63 (NL), string='\n', start=(1, 0), end=(1, 1), line='\n')
TokenInfo(type=1 (NAME), string='def', start=(2, 0), end=(2, 3), line='def abs(a: int, b: int) -> int:\n')
TokenInfo(type=1 (NAME), string='abs', start=(2, 4), end=(2, 7), line='def abs(a: int, b: int) -> int:\n')
TokenInfo(type=55 (OP), string='(', start=(2, 7), end=(2, 8), line='def abs(a: int, b: int) -> int:\n')
TokenInfo(type=1 (NAME), string='a', start=(2, 8), end=(2, 9), line='def abs(a: int, b: int) -> int:\n')
TokenInfo(type=55 (OP), string=':', start=(2, 9), end=(2, 10), line='def abs(a: int, b: int) -> int:\n')
TokenInfo(type=1 (NAME), string='int', start=(2, 11), end=(2, 14), line='def abs(a: int, b: int) -> int:\n')
TokenInfo(type=55 (OP), string=',', start=(2, 14), end=(2, 15), line='def abs(a: int, b: int) -> int:\n')
TokenInfo(type=1 (NAME), string='b', start=(2, 16), end=(2, 17), line='def abs(a: int, b: int) -> int:\n')
TokenInfo(type=55 (OP), string=':', start=(2, 17), end=(2, 18), line='def abs(a: int, b: int) -> int:\n')
TokenInfo(type=1 (NAME), string='int', start=(2, 19), end=(2, 22), line='def abs(a: int, b: int) -> int:\n')
TokenInfo(type=55 (OP), string=')', start=(2, 22), end=(2, 23), line='def abs(a: int, b: int) -> int:\n')
TokenInfo(type=55 (OP), string='->', start=(2, 24), end=(2, 26), line='def abs(a: int, b: int) -> int:\n')
TokenInfo(type=1 (NAME), string='int', start=(2, 27), end=(2, 30), line='def abs(a: int, b: int) -> int:\n')
TokenInfo(type=55 (OP), string=':', start=(2, 30), end=(2, 31), line='def abs(a: int, b: int) -> int:\n')
TokenInfo(type=4 (NEWLINE), string='\n', start=(2, 31), end=(2, 32), line='def abs(a: int, b: int) -> int:\n')
TokenInfo(type=5 (INDENT), string='    ', start=(3, 0), end=(3, 4), line='    if a > b:\n')
TokenInfo(type=1 (NAME), string='if', start=(3, 4), end=(3, 6), line='    if a > b:\n')
TokenInfo(type=1 (NAME), string='a', start=(3, 7), end=(3, 8), line='    if a > b:\n')
TokenInfo(type=55 (OP), string='>', start=(3, 9), end=(3, 10), line='    if a > b:\n')
TokenInfo(type=1 (NAME), string='b', start=(3, 11), end=(3, 12), line='    if a > b:\n')
TokenInfo(type=55 (OP), string=':', start=(3, 12), end=(3, 13), line='    if a > b:\n')
TokenInfo(type=4 (NEWLINE), string='\n', start=(3, 13), end=(3, 14), line='    if a > b:\n')
TokenInfo(type=5 (INDENT), string='        ', start=(4, 0), end=(4, 8), line='        return a - b\n')
TokenInfo(type=1 (NAME), string='return', start=(4, 8), end=(4, 14), line='        return a - b\n')
TokenInfo(type=1 (NAME), string='a', start=(4, 15), end=(4, 16), line='        return a - b\n')
TokenInfo(type=55 (OP), string='-', start=(4, 17), end=(4, 18), line='        return a - b\n')
TokenInfo(type=1 (NAME), string='b', start=(4, 19), end=(4, 20), line='        return a - b\n')
TokenInfo(type=4 (NEWLINE), string='\n', start=(4, 20), end=(4, 21), line='        return a - b\n')
TokenInfo(type=6 (DEDENT), string='', start=(5, 4), end=(5, 4), line='    return b - a\n')
TokenInfo(type=1 (NAME), string='return', start=(5, 4), end=(5, 10), line='    return b - a\n')
TokenInfo(type=1 (NAME), string='b', start=(5, 11), end=(5, 12), line='    return b - a\n')
TokenInfo(type=55 (OP), string='-', start=(5, 13), end=(5, 14), line='    return b - a\n')
TokenInfo(type=1 (NAME), string='a', start=(5, 15), end=(5, 16), line='    return b - a\n')
TokenInfo(type=4 (NEWLINE), string='\n', start=(5, 16), end=(5, 17), line='    return b - a\n')
TokenInfo(type=6 (DEDENT), string='', start=(6, 0), end=(6, 0), line='')
TokenInfo(type=0 (ENDMARKER), string='', start=(6, 0), end=(6, 0), line='')

Yeah, that’s…a lot but don’t worry, you don’t have to memorize it or anything. The key takeaway here is that Python has broken down your code into its smallest meaningful parts, which will be used in the next steps of execution.

Next, your code is parsed

Next, these tokens are combined to form a structure called an abstract syntax tree (AST), which is a tree-based representation of the structure of your code. The AST captures the hierarchical structure of your code, showing how different parts relate to each other. For example, in our function, the AST would show that abs is a function definition, a and b are parameters, and the if statement is a conditional that leads to different return statements.

It’s also at this stage that Python checks for syntax errors. If there are any, it raises a SyntaxError and stops execution.

We can see what our simple function above’s AST would look like using the ast module in the standard library. The code looks something like this:

import ast

source = """
def abs(a: int, b: int) -> int:
    if a > b:
        return a - b
    return b - a

abs(5, 3)
"""

tree = ast.parse(source)
print(ast.dump(tree, indent=4))

…and this would return a tree like so:

Module(
    body=[
        FunctionDef(
            name='abs',
            args=arguments(
                args=[
                    arg(
                        arg='a',
                        annotation=Name(id='int')),
                    arg(
                        arg='b',
                        annotation=Name(id='int'))]),
            body=[
                If(
                    test=Compare(
                        left=Name(id='a'),
                        ops=[
                            Gt()],
                        comparators=[
                            Name(id='b')]),
                    body=[
                        Return(
                            value=BinOp(
                                left=Name(id='a'),
                                op=Sub(),
                                right=Name(id='b')))]),
                Return(
                    value=BinOp(
                        left=Name(id='b'),
                        op=Sub(),
                        right=Name(id='a')))],
            returns=Name(id='int')),
        Expr(
            value=Call(
                func=Name(id='abs'),
                args=[
                    Constant(value=5),
                    Constant(value=3)]))])

Again, this is a lot of information for such a short function but what you should really glean from this is that every variable, statement, function, constant, etc. along with its relationship is represented in this tree.

Then, we compile to bytecode

Next, Python compiles that AST down into bytecode, which is really a lower-level, platform-independent representation of your code. This is what the CPython interpreter actually executes.

Just like with the AST, you can see what the bytecode representation of this function would be. We can see what this would look like for the same function we looked at earlier using the dis module (aka the disassembly module) in the standard library.

import dis
def abs(a: int, b: int) -> int:
    if a > b:
        return a - b
    return b - a
dis.dis(abs)

  3           RESUME                   0

  4           LOAD_FAST_BORROW_LOAD_FAST_BORROW 1 (a, b)
              COMPARE_OP             148 (bool(>))
              POP_JUMP_IF_FALSE        9 (to L1)
              NOT_TAKEN

  5           LOAD_FAST_BORROW_LOAD_FAST_BORROW 1 (a, b)
              BINARY_OP               10 (-)
              RETURN_VALUE

  7   L1:     LOAD_FAST_BORROW_LOAD_FAST_BORROW 16 (b, a)
              BINARY_OP               10 (-)
              RETURN_VALUE

This might look intimidating, but it’s just a lower-level form of your original code. Here’s a quick mapping, removing some instructions for brevity:

This shows how Python breaks your logic into a series of simple instructions. Each instruction is a single operation that the interpreter can execute. For example, LOAD_FAST_BORROW_LOAD_FAST_BORROW loads the values of a and b, COMPARE_OP compares them, and BINARY_OP performs the subtraction.

CPython runs your code using a bytecode interpreter. It uses an internal evaluation loop, which executes one bytecode instruction at a time, dispatching to the appropriate C function that handles it. This loop also manages the Python Virtual Machine (PVM), which maintains the call stack, handles memory management, exception handling, and more.

There’s more to say here about the Global Interpreter Lock, garbage collection, etc. but I’m going to save that for another post. The key takeaway here is that the PVM executes these bytecode instructions in a loop, processing each instruction in sequence until it reaches the end of the function or encounters a return statement.

For all intents and purposes, your code is now running in the Python interpreter. This is how Python executes your code in a regular build of CPython.

But, wait, there’s more: The Specializing Adaptive Interpreter

Since Python 3.11, we’ve had something called the Specializing Adaptive Interpreter in CPython (a significant contributor to why Python 3.11 was about 25% faster than Python 3.10 for most workloads). We won’t get into this too deep in this blog post but in essence, the idea here is that once a bytecode instruction has been executed enough times in a code path, the interpreter can “specialize” it based on types and values seen at runtime.

For example, let’s consider the BINARY_OP instruction in our bytecode. If the interpreter sees you’re doing a lot of integer subtraction, it might optimize that instruction internally by installing a fast path for integers. This means that while the bytecode still says BINARY_OP, the interpreter skips type checks and uses a specialized implementation for integer subtraction behind the scenes, making it significantly faster, even without the JIT compiler.

Okay, so what happens in JIT builds?

Right, right. Okay, so now that we understand how the interpreter works, we can talk about what happens when you run your code in a JIT build of CPython.

Enter the micro-instruction (uops) interpreter

So, your code is running in a regular build of CPython is already doing some smart things to optimize your bytecode. But what if we could do even better? What if we could take those bytecode instructions and turn them into something even more efficient? This is where the micro-instruction interpreter comes in.

So once your code has “warmed up” or been executed enough times, we can start to optimize it even further. What’s really neat is that the specializing adaptive interpreter actually provides us with a lot of profiling information about the code being executed that helps with all of this. With the micro-op interpreter, we break each bytecode instruction in the code path down into even smaller, more specialized instructions called micro-operations, or uops. These uops are designed to be more efficient and can be executed much faster than the original bytecode instructions. The process of breaking down bytecode instructions into traces happens automatically thanks to some domain-specific language (DSL) infrastructure that was introduced in Python 3.12; it’s effectively a table look up to say that this bytecode instruction maps to these uops. Once we have these uops, we can even start to optimize them further by removing unnecessary checks and operations (…again, a topic for another post).

I’d be remiss if I didn’t mention that the micro-op interpreter is a separate interpreter from the regular bytecode one. In a JIT build of CPython, both interpreters are available, and once a function becomes “hot,” execution can switch from bytecode to uops. That might sound like a big performance win, but not quite yet. In fact, things often get slower at this stage. The micro-op interpreter introduces overhead by breaking each bytecode instruction into smaller, more granular uops and dispatching more instructions overall. It’s a trade-off: we’re doing extra work now to prepare for the real speedup that comes next, when the JIT compiler steps in to generate optimized machine code and (hopefully) recover that lost performance and then some.

When you build Python with --enable-experimental-jit or set PYTHON_JIT=1 in Python 3.14 builds, you’re not just enabling the JIT itself, but the micro-op interpreter as well.

JIT Compilation

Alright, we’ve finally made it! Let’s talk about the JIT.

First off, let’s talk about what a JIT compiler is, in case you’re not already familiar. A JIT (Just-In-Time) compiler is a type of compiler that translates code into machine code at runtime, rather than before execution.

In the context of CPython, our JIT compiler uses a technique called copy-and-patch. This technique is covered in this paper but don’t worry, we don’t need to get too academic here. Basically, what happens is as follows:

1. When CPython is built, we use LLVM to generate precompiled stencil files for your specific platform and architecture. These stencil files contain templates for how to translate the micro-ops we talked about earlier into machine code.
2. When your code is executed, the JIT compiler monitors the execution and identifies “hot” traces—sections of code that are executed frequently.
3. When a hot trace is detected, the JIT compiler takes the relevant micro-ops, which are the smaller, specialized instructions we covered earlier, and uses the precompiled stencil templates to generate native machine code.
   • The JIT compiler fills in the placeholders in the stencil templates with the actual values needed for your code, such as addresses of variables, constants, and cached results (“patching” up the code).
   • These stencil files are then linked together to form a trace, which is a sequence of micro-ops that can be executed as native machine code.
   • Finally, the JIT compiler executes this native machine code directly instead of interpreting.

Now, the elephant in the room here is that the JIT does not (yet!) make Python a whole lot faster. In most cases, the JIT builds range from slower to about the same performance as the non-JIT build of Python. As of 3.14, the JIT is faster in select benchmarks but we have a ways to go still. Ken Jin has a great blog post that goes into more detail about the performance of the JIT builds in Python 3.14 (among other reflections) if you’re interested.

Putting it all together

So, to summarize, when you run your code in a JIT build of CPython, the following happens:

1. Your code is tokenized, parsed, and compiled into bytecode as usual.
2. The bytecode is executed by the regular bytecode interpreter, which may specialize some instructions based on runtime profiling.
3. If the code is executed enough times, the micro-op interpreter kicks in, breaking down the bytecode instructions into smaller, more specialized uops.
4. The JIT compiler then compiles these uops into native machine code using precompiled stencil templates, optimizing the execution of your code.
5. The native machine code is executed directly by the CPU, bypassing the bytecode interpreter and micro-op interpreter.

…and that’s it! You now have an understanding of how your code runs in a JIT build of CPython and you didn’t have to be a compiler engineer to understand it!

Suggested readings & videos

Some other great talks, blog posts, etc. by other folks working on Python:

• Maybe watch one of Brandt’s talks on this topic:
  • Building a JIT compiler for CPython
  • What they don’t tell you about building a JIT compiler for CPython
• Diego’s EuroPython 2025 talk
• ICYMI earlier in the post, Ken Jin’s Reflections on 2 years of CPython’s JIT Compiler: The good, the bad, the ugly is also great if you want to learn more about the JIT builds in Python 3.14 and what it’s taken to get to this point.
• Check out PEP 744, it’s really not that scary!

If you enjoyed this post, please consider sharing it with anyone you think might find it interesting. If you have any questions or feedback, feel free to reach out to me via email.