PyTorch × TPU (TorchTPU & OpenXLA): Compiler Internals and Operator Fusion Optimization

pytorch-×-tpu-(torchtpu-&-openxla):-compiler-internals-and-operator-fusion-optimization

TL;DR — Real TPU throughput comes from minimizing memory round-trips: TorchTPU’s OpenXLA compiler captures the full graph and fuses operations directly inside fast on-chip memory. Keeping that speedup requires avoiding mid-pass CPU syncs that split the graph and bounding variable input shapes to stop recompilations.

1. Accelerator Memory Hierarchy and the Principle of Operator Fusion

1.1 The Speed Gap Between Compute Units and Memory

Deep learning accelerators such as GPUs and TPUs spend more time moving data into and out of memory than actually computing arithmetic. While arithmetic throughput has scaled dramatically generation over generation, memory bandwidth — the rate at which bytes can travel between memory and compute — has grown much more slowly, creating a bottleneck known as the Memory Wall. Understanding why this bottleneck arises starts with distinguishing the three physical regions inside an accelerator, shown from left to right below.

Figure 1. Three-Tier Accelerator Memory & Compute Hierarchy (HBM ↔ On-Chip Memory ↔ MXU/VPU)

The first region on the left is HBM (High-Bandwidth Memory). Because it provides tens to hundreds of gigabytes of capacity, model weights, activations, and input/output tensors reside in HBM by default. However, because HBM sits off the compute die, every read or write across the memory bus incurs substantial latency.

The second region in the center is On-Chip Memory (VMEM & Registers). Located right next to the execution pipelines, it acts as an ultra-fast scratchpad where values can be read and written without memory-bus stalls. Physical die area restricts its capacity to tens of megabytes, so only small tiles of data actively being computed can live there at any moment.

The third region on the right consists of the Compute Units ( MXU and VPU), which split responsibilities on a TPU into two specialized engines:

  • The MXU (Matrix Multiply Unit) handles dense matrix multiplications such as nn.Linear and torch.matmul. Built as a 2D systolic array of tens of thousands of multiply-accumulate cells through which activations and weights stream rhythmically, it completes massive matrix products in microseconds.
  • The VPU (Vector Processing Unit) handles pointwise operations that apply the same formula independently to every element (such as addition, multiplication, or the GELU activation function) as well as reduction operations that aggregate elements across a dimension to compute means or variances (such as LayerNorm).

Before an MXU or VPU can compute anything, its operands must be fetched from HBM into on-chip memory, and finished results must eventually be written back to HBM. Because moving data between HBM and on-chip memory takes an order of magnitude longer than performing the math itself, minimizing HBM round-trips is the single most decisive factor in accelerator utilization.

1.2 Why Unfused Kernel Execution Hits a Bottleneck

An independent unit of work dispatched to an accelerator is called a kernel, and the command issued by the CPU telling the accelerator to run that unit is called a kernel launch. Consider the first half of a standard Transformer feed-forward sub-block:

  1. h = Linear(x) (matrix multiplication followed by bias addition)
  2. h = GELU(h) (GELU(h) = 0.5 · h · (1 + tanh(√(2/π) · (h + 0.044715 · h³))), original GELU paper)

Under Unfused Kernel Execution, where each Python operation runs as its own separate kernel in sequence, the accelerator suffers a Repeated HBM Round-Trip Bottleneck:

  1. ① Read x and ② Write h: The first kernel fetches input x from HBM, computes Linear(x) on the MXU, writes the full intermediate tensor h back out to HBM, and terminates.
  2. ③ Re-read h and ④ Write Output: Immediately afterward, the second kernel must re-read that exact same intermediate tensor h from HBM into on-chip memory just to run GELU(h) on the VPU, before writing the output back to HBM.
  3. Worse still, if the elementary math operations inside GELU (h³, addition, tanh, scalar multiplications) also run as separate kernels, multi-megabyte intermediate tensors bounce back and forth to HBM at every tiny step while the compute units sit idle waiting on memory traffic.

1.3 How Compilers and Operator Fusion Eliminate Round-Trips

Deep learning compilers and Operator Fusion exist specifically to eliminate those redundant memory transfers. By default, PyTorch runs in Eager mode, executing each Python line immediately as an independent kernel on the accelerator. Eager mode is convenient for interactive debugging because intermediate tensors are materialized right away, but because the runtime cannot see future operations ahead of time, every operation is forced to flush its result to HBM.

By contrast, graph compilation via torch.compile scans the entire forward() method ahead of execution to capture the full dependency blueprint – the computation graph – and lowers it into a machine-analyzable IR (Intermediate Representation).

Inspecting this IR lets the compiler merge chains of consecutive operations into a single unified execution unit through Operator Fusion (denoted %fusion in XLA). As illustrated on the right side of the diagram below ( Operator Fusion (%fusion)), instead of flushing the output of Linear(x) back to HBM ( No Intermediate HBM Write), the compiler keeps those values pinned inside On-Chip Memory (VMEM & Registers), streams them directly through the entire GELU(h) formula on the VPU inside a %fusion Single Loop, and writes only the final result to HBM once. Modern LLM acceleration techniques such as FlashAttention and custom Pallas TPU kernels rely on this exact same principle – fusing tiled operations inside VMEM so large intermediate attention matrices never touch HBM.

Comparison of Repeated HBM Round-Trip Bottleneck in Unfused Kernel Execution vs. Operator Fusion (%fusion) Single Loop

2. Three Generations of the PyTorch × TPU Software Stack

PyTorch was designed around line-by-line Eager execution, whereas Google’s TPU architecture was co-designed from day one around OpenXLA, an ahead-of-time whole-graph compiler. Bridging that architectural gap has driven three distinct generations of the PyTorch × TPU stack.

Three-Generation Architecture Evolution of the PyTorch × TPU Stack (Gen 1 Legacy torch_xla → Gen 2 openxla Bridge → Gen 3 Native TorchTPU)

2.1 How Each Generation Works

Gen 1: Legacy torch_xla (Lazy Tensor Execution)

Early versions of PyTorch had no built-in mechanism like torch.compile to capture whole-model graphs ahead of time. To work around this, first-generation torch_xla ( GitHub) introduced Lazy Tensor execution. Operations invoked on a separate ‘xla:0’ device created via xm.xla_device() did not execute immediately; instead, they silently recorded nodes into a pending graph behind the scenes until the user hit a manual xm.mark_step() barrier, which finally handed the accumulated graph to XLA HLO for compilation and TPU execution. Re-tracing graphs across the Python/C++ boundary every step added substantial host overhead, and scattering manual xm.mark_step() calls throughout standard PyTorch training loops made codebases harder to maintain.

import torch_xla.core.xla_model as xm



device = xm.xla_device() # Creates 'xla:0' device

model = MyModel().to(device)

out = model(x)

xm.mark_step() # Graph compilation & execution happen only at this manual barrier

Gen 2: torch_xla 2.x (openxla Bridge)

PyTorch 2.x added TorchDynamo, which intercepts Python bytecode just before execution to capture FX graphs automatically via the torch.compile workflow. An FX graph ( torch.fx) is PyTorch’s native Python-level graph representation – it strips away surrounding Python classes and boilerplate, recording only pure tensor operations (inputs, operator calls, and outputs) as a clean, ordered list of nodes that serves as the common blueprint handed off to any compiler backend ( inductor, openxla, tpu, etc.). Building on this frontend, torch_xla 2.x introduced the torch.compile(model, backend=”openxla”) bridge – eliminating the manual step barrier. Once TorchDynamo captures the FX graph, it passes cleanly through three standardized OpenXLA layers:

  • StableHLO: A portable, version-stable graph format shared across PyTorch, JAX, and TensorFlow so different frontends can hand graphs to OpenXLA through a single contract.
  • XLA HLO (High-Level Operations): OpenXLA’s internal compiler IR where whole-graph optimizations — such as operator fusion (%fusion) and algebraic simplification – actually take place.
  • PJRT (Pretty much Just another RunTime): OpenXLA’s universal hardware plugin interface (effectively a standard device adapter) that loads and runs compiled executables on physical targets such as TPUs, GPUs, or CPUs. Because hardware drivers plug in beneath PJRT without changing the compiler above it, setting PJRT_DEVICE=CPU swaps in the local CPU plugin – running the exact same OpenXLA HLO compilation pipeline on any laptop or workstation without a physical TPU VM.
import torch, torch_xla



device = torch_xla.device() # Returns 'xla:0' (runs via local CPU plugin when PJRT_DEVICE=CPU)

model = MyModel().to(device)

compiled = torch.compile(model, backend="openxla")

out = compiled(x)

torch_xla.sync()

Gen 3: Native TorchTPU (torch_tpu)

Even in Gen 2, routing execution through the external torch_xla package and ‘xla:0’ bridge left subtle friction with mainstream PyTorch libraries. Redesigned natively by Google and Meta, TorchTPU ( pip install torch_tpu) removes that external bridge completely. Whenever you call a + b or torch.matmul in Python, PyTorch delegates the work to ATen ( A Tensor Library) – its foundational C++ operator engine – whose internal dispatcher inspects each tensor’s device tag (“cpu”, “cuda”, etc.) and routes the call to the matching hardware kernel. To let new accelerators plug into that dispatcher as first-class devices alongside “cuda” without patching PyTorch’s core source tree, ATen provides a dedicated C++ extension slot called PrivateUse1. Because torch_tpu registers torch.device(“tpu”) directly into PrivateUse1 and communicates straight with Google’s low-level libtpu hardware driver/compiler library underneath, existing GPU scripts run unmodified simply by changing “cuda” to “tpu”.

Furthermore, torch_tpu unifies whole-graph Compile Mode ( torch.compile(model, backend=”tpu”) via StableHLO → XLA HLO) alongside three native Eager execution modes inside a single stack:

  • Strict Eager: Dispatches each operator asynchronously line by line, matching standard PyTorch eager semantics 1:1.
  • DeferAndFuse: Automatically batches and fuses consecutive operators queued on the device stream, delivering fusion speedups even in dynamic code where full torch.compile is difficult to apply.
  • Debug: Synchronizes CPU and TPU after every single operation so runtime errors point to the exact offending line of Python code.
import torch



device = torch.device("tpu")

model = MyModel().to(device)



# 1) Strict Eager mode: asynchronous op-by-op execution matching standard PyTorch

out = model(x)



# 2) DeferAndFuse Eager mode: automatically fuses consecutive queued ops without torch.compile

torch.tpu.set_eager_mode(torch.tpu.EagerMode.DEFER_AND_FUSE)

out = model(x)



# 3) Compile mode: lowers full graph through StableHLO -> XLA HLO with full %fusion optimization

compiled = torch.compile(model, backend="tpu")

out = compiled(x)

Table 1. Feature Comparison Across Three Generations of the PyTorch × TPU Stack

Why Use backend=”openxla” When Verifying Locally Without a TPU? OpenXLA is not a simulator – it is the actual production compiler powering Cloud TPUs, and both Gen 2 backend=”openxla” and Gen 3 backend=”tpu” run the exact same FX Graph → StableHLO → XLA HLO (%fusion) optimization pipeline under the hood. The difference lies only at the bottom driver layer: Gen 3 torch_tpu links directly against the physical libtpu hardware driver and therefore cannot initialize on a machine without TPU chips, whereas Gen 2 torch_xla (backend=”openxla”) sits atop the pluggable PJRT runtime and can run on a standard CPU via PJRT_DEVICE=CPU. All HLO dumps and experiments in Sections 3-6 use this local CPU backend to inspect the compiler directly, and transfer to a Cloud TPU VM simply by switching to backend=”tpu”.

3. Compiler IR Deep Dive: TorchInductor vs. OpenXLA

To see concretely how PyTorch’s default CPU/GPU compiler TorchInductor ( backend=”inductor”, CPU IR debugging guide) and the TPU compiler OpenXLA ( backend=”openxla” / “tpu”) translate the exact same model into fundamentally different execution structures, we can extract and compare their actual generated IRs side by side.

Our target module is a Transformer feed-forward sub-block ( TransformerSubBlock) taking a 3D input tensor x of shape (4, 128, 512) – batch size 4, sequence length 128 tokens, and hidden dimension 512. When executing linear layers, the leading batch and sequence dimensions ( 4 × 128 = 512 token vectors) flatten into a 2D matrix of shape (512, 512). The circled stage numbers ①-④ in the code below correspond 1:1 to every diagram, slide, and IR walkthrough throughout the rest of this post.

import torch

import torch.nn as nn



class TransformerSubBlock(nn.Module):

def __init__(self, d_model: int = 512, d_ff: int = 2048):

super().__init__()

self.fc1 = nn.Linear(d_model, d_ff)

self.act = nn.GELU(approximate="tanh") # Polynomial + tanh approximation

self.fc2 = nn.Linear(d_ff, d_model)

self.norm = nn.LayerNorm(d_model)



def forward(self, x: torch.Tensor) -> torch.Tensor:

residual = x

h = self.fc1(x) # ① First Linear (fc1): (512×512) @ (512×2048) matmul + bias(2048) add

h = self.act(h) # ② GELU Activation: 0.5 * h * (1 + tanh(0.79788 * (h + 0.044715 * h^3)))

h = self.fc2(h) # ③ Second Linear (fc2): (512×2048) @ (2048×512) matmul + bias(512) add

out = self.norm(residual + h) # ④ Residual Add (x + h) + LayerNorm

return out

Stage ④ combines a Residual Add (skip connection), which adds the original input x directly to the transformed output h, with LayerNorm, which normalizes each token vector to zero mean and unit variance before applying learnable scale ( weight, gamma) and shift ( bias, beta) parameters.

With just two environment variables, we can dump the generated code from both compilers on any local CPU machine without a TPU:

# 1) Dump FX graph and generated C++ / Python wrapper code from TorchInductor

TORCH_COMPILE_DEBUG=1 python -c "

import torch; m = torch.compile(TransformerSubBlock().eval(), backend='inductor')

m(torch.randn(4, 128, 512))"



# 2) Dump pre-optimization (before_optimizations) and post-optimization (after_optimizations) HLO from OpenXLA

PJRT_DEVICE=CPU XLA_FLAGS="--xla_dump_to=./xla_dump --xla_dump_hlo_as_text" python -c "

import torch, torch_xla; dev = torch_xla.device()

m = torch.compile(TransformerSubBlock().eval().to(dev), backend='openxla')

out = m(torch.randn(4, 128, 512).to(dev)); torch_xla.sync()"

3.1 High-Level Comparison of Compiled Outputs

Table 2. Compiled IR Comparison: TorchInductor vs. OpenXLA

3.2 Visual Graph Comparison Across Compilation Stages

Raw HLO Graphviz dumps emitted directly by XLA inline dozens of scalar constants and layout metadata nodes at once, producing sprawling canvases over 4,000 pixels wide. To make the structural differences immediately legible without losing fidelity, the three slides below preserve the exact function names, HLO op names, and buffer identifiers from the compiler dumps in a unified layout. Each color and shape carries a consistent meaning across all three slides:

  • Blue boxes: Dense matrix multiplications (Inductor’s External Matmul extern_kernels.addmm or OpenXLA’s Pure Matmul dot.36, dot.59)
  • Yellow boxes: C++ Fused Kernels generated by TorchInductor for non-matmul stages (cpp_fused_gelu_0, cpp_fused_add_native_layer_norm_1)
  • Red cylinders: Intermediate buffers (buf0 ~ buf4) allocated in main memory between split kernels, representing forced memory writes and re-reads
  • Gray boxes: Unfused individual HLO operations prior to OpenXLA’s optimization passes
  • Green boxes: Post-optimization OpenXLA %fusion clusters (%fusion #1 ~ #5) that merge chains of operations into single on-chip memory / register loops with no intermediate HBM writes

Slide 1. TorchInductor Execution Structure (4 Kernel Launches · 4 Intermediate Buffer Allocations)

Slide 2. OpenXLA Pre-Optimization HLO Graph (86 Unfused Individual Ops)

Slide 3. OpenXLA Post-Optimization HLO Graph (2 dot Matmuls + 5 %fusion Clusters)

Flipping through the three slides highlights the architectural progression at a glance:

  • In Slide 1 (TorchInductor Execution Structure), red cylinders (buf0, buf1, and buf2·buf3·buf4) sit between every blue External Matmul (extern_kernels.addmm) and yellow C++ Fused Kernel, forcing each stage to write its output to main memory and the next kernel to read it back in.
  • In Slide 2 (OpenXLA Pre-Optimization HLO Graph), stages ①-④ are initially decomposed into 86 fine-grained blue and gray unfused operations.
  • In Slide 3 (OpenXLA Post-Optimization HLO Graph), every gray operation has been absorbed into green %fusion clusters (%fusion Cluster #1 and %fusion Clusters #2 ~ #5), eliminating intermediate red memory buffers so the two dot matmuls (dot.36, dot.59) and fused regions connect cleanly.

3.3 1:1 Code-to-Slide Walkthrough

(1) Inside Slide 1: TorchInductor’s Generated Wrapper Code

The 4 Kernel Launches and buffers buf0 ~ buf6 in Slide 1 map 1:1 to the call(args) wrapper emitted by TorchInductor under TORCH_COMPILE_DEBUG=1. Three helper primitives appear in the generated code:

  • empty_strided_cpu: Allocates a fresh buffer in main memory (buf0, buf2, buf3, buf4) to hold intermediate results. For instance, buf0 is a (512, 2048) matrix in float32 (4 bytes per element), occupying 512 × 2048 × 4 = 4,194,304 bytes (4.19MB). Because we dumped this on a local CPU, the allocator ends in _cpu; on a GPU, it becomes empty_strided_cuda and allocates the exact same 4.19MB buffer in GPU HBM.
  • reinterpret_tensor: Changes how tensor dimensions and strides are viewed without copying underlying bytes – for example, viewing the 2D (512, 2048) memory of buf0 as a 3D (4, 128, 2048) tensor named buf1.
  • extern_kernels.addmm: Calls a pre-compiled external BLAS routine (torch.addmm) that computes matrix multiplication plus bias addition.
def call(args):

# arg0_1: Input Tensor x (4, 128, 512)

# arg1_1, arg2_1: ① fc1 weight (2048, 512) & bias (2048)

# arg3_1, arg4_1: ③ fc2 weight (512, 2048) & bias (512)

# arg5_1, arg6_1: ④ LayerNorm gamma (512) & beta (512)

arg0_1, arg1_1, arg2_1, arg3_1, arg4_1, arg5_1, arg6_1 = args



# [Intermediate Buffer #1] Allocate buf0 (512×2048, 4.19MB) in Main Memory for ① fc1 output

buf0 = empty_strided_cpu((512, 2048), (2048, 1), torch.float32)



# [Kernel Launch #1] External Matmul (extern_kernels.addmm) computes ① First Linear (fc1) -> writes buf0

extern_kernels.addmm(arg2_1, reinterpret_tensor(arg0_1, (512, 512), ...), ..., out=buf0)



# [Kernel Launch #2 & Main Memory Overwrite] Re-read buf0 as 3D view buf1,

# compute ② GELU Activation in C++ Fused Kernel (cpp_fused_gelu_0), and overwrite buf1

buf1 = reinterpret_tensor(buf0, (4, 128, 2048), (262144, 2048, 1), 0); del buf0

cpp_fused_gelu_0(buf1)



# [Intermediate Buffer #2] Allocate buf2 (512×512, 1.05MB) in Main Memory for ③ fc2 output

buf2 = empty_strided_cpu((512, 512), (512, 1), torch.float32)



# [Kernel Launch #3] External Matmul (extern_kernels.addmm) computes ③ Second Linear (fc2) -> writes buf2

extern_kernels.addmm(arg4_1, reinterpret_tensor(buf1, (512, 2048), ...), ..., out=buf2)



# [Intermediate Buffers #3·#4] Allocate per-token Mean (buf3) & Var (buf4) buffers for ④ LayerNorm

buf3 = empty_strided_cpu((4, 128, 1), (128, 1, 512), torch.float32)

buf4 = empty_strided_cpu((4, 128, 1), (128, 1, 512), torch.float32)



# [Kernel Launch #4] C++ Fused Kernel (cpp_fused_add_native_layer_norm_1) computes

# ④ Residual Add (x + h) + LayerNorm and returns Final Output Tensor buf6

buf6 = reinterpret_tensor(buf2, (4, 128, 512), (65536, 512, 1), 0); del buf2

cpp_fused_add_native_layer_norm_1(buf6, arg0_1, arg5_1, arg6_1, buf3, buf4)

return (buf6, )

Because TorchInductor’s default mode ( mode=”default”) delegates dense matrix multiplications to pre-built vendor math libraries ( extern_kernels.addmm – backed by Intel’s CPU math libraries oneDNN/MKL or NVIDIA’s GPU matrix library cuBLAS), it cannot fuse the matrix multiplication itself with the following activation kernel (the C++ kernel cpp_fused_gelu_0 on CPU, or triton_poi_fused_gelu_0 generated in the GPU kernel language Triton). Every extern_kernels.addmm boundary therefore forces a main-memory write and re-read, matching the red cylinders in Slide 1.

(2) Inside Slide 3: OpenXLA’s Post-Optimization HLO Code

The 2 dot Matmuls and 5 %fusion Clusters (%fusion #1 ~ #5) in Slide 3 correspond directly to the top-level ENTRY SyncTensorsGraph.87 module emitted by OpenXLA after optimization. Four core HLO conventions ( XLA Operation Semantics) make the listing straightforward to read:

  • f32[4,128,512]{2,1,0}: A 32-bit floating-point (f32) tensor of shape (4, 128, 512) whose minor-to-major physical layout {2,1,0} stores dimension 2 (512) contiguously in memory.
  • bitcast: A Zero-Cost Reshape that reinterprets tensor dimensions without moving a single byte in memory (analogous to Inductor’s reinterpret_tensor).
  • dot with Direct Contracting Matmul (lhs_contracting_dims={1}, rhs_contracting_dims={1}): Pure matrix multiplication without bias. PyTorch’s nn.Linear stores fc1 weights p4.32 in shape (2048, 512), which naively requires a Weight Transpose (transpose) into (512, 2048) before multiplying. Optimized HLO eliminates the transpose completely by contracting dimension 1 (512) of the (512, 512) input directly against dimension 1 (512) of p4.32. Likewise, dot.59 contracts dimension 1 (2048) of (512, 2048) directly against dimension 1 (2048) of p2.26 (512, 2048) to produce (512, 512).
  • ROOT tuple.86: The final return node of the HLO graph, writing the completed output tensor once to HBM.
ENTRY SyncTensorsGraph.87 {

// [Read Once from HBM + Zero-Cost Reshape] Reinterpret Input Tensor x (p5.34) as 2D (512, 512)

p5.34 = f32[4,128,512]{2,1,0} parameter(5)

bitcast = f32[512,512]{1,0} bitcast(p5.34)



// [HLO Matmul #1: dot.36] Direct Contracting Matmul of (512×512) and (2048×512) without transpose

p4.32 = f32[2048,512]{1,0} parameter(4)

dot.36 = f32[512,2048]{1,0} dot(bitcast, p4.32), lhs_contracting_dims={1}, rhs_contracting_dims={1}



// [%fusion Cluster #1] parallel_multiply_bitcast_fusion (callee: fused_computation.4.clone)

// Fuses ① fc1 Bias Add (p3.31) + ② Full GELU + Zero-Cost Reshape (bitcast) into a single on-chip loop

p3.31 = f32[2048]{0} parameter(3)

call.5 = f32[512,2048]{1,0} call(p3.31, dot.36), to_apply=parallel_multiply_bitcast_fusion



// [HLO Matmul #2: dot.59] Direct Contracting Matmul with second weight matrix (p2.26) without transpose

p2.26 = f32[512,2048]{1,0} parameter(2)

dot.59 = f32[512,512]{1,0} dot(call.5, p2.26), lhs_contracting_dims={1}, rhs_contracting_dims={1}



// [%fusion Clusters #2 ~ #5] ④ Residual + LayerNorm 4-Stage Fusion Group

p1.25 = f32[512]{0} parameter(1)

// • %fusion #2 (parallel_add_bitcast_fusion): fc2 Bias Add + Residual Add (x + h)

call.4 = f32[1,512,512]{2,1,0} call(p5.34, p1.25, dot.59), to_apply=parallel_add_bitcast_fusion

constant = f32[] constant(0)

// Two-step Sum Reduction across 512 hidden elements (parallel_reduce-window -> reduce)

call = f32[1,512,16]{2,1,0} call(call.4, constant), to_apply=parallel_reduce-window

reduce = f32[512]{0} reduce(call, constant), dimensions={0,2}, to_apply=scalar_add_computation

// • %fusion #3 (parallel_bitcast_multiply_fusion): Elementwise Square for Variance

call.3 = f32[1,512,512]{2,1,0} call(p5.34, p1.25, dot.59), to_apply=parallel_bitcast_multiply_fusion

call.1 = f32[1,512,16]{2,1,0} call(call.3, constant), to_apply=parallel_reduce-window.1

// • %fusion #4 (add_rsqrt_fusion): Reciprocal Sqrt (rsqrt) from Sum Reduction (reduce)

add_rsqrt_fusion = f32[512]{0} fusion(call.1, reduce), kind=kLoop, calls=fused_computation.1

p0.2 = f32[512]{0} parameter(0)

p6.78 = f32[512]{0} parameter(6)

// • %fusion #5 (parallel_multiply_add_fusion): Mean Centering, Normalization & Gamma (p0.2) · Beta (p6.78)

call.2 = f32[4,128,512]{2,1,0} call(p0.2, add_rsqrt_fusion, reduce, p5.34, p1.25, dot.59, p6.78), to_apply=parallel_multiply_add_fusion

ROOT tuple.86 = (f32[4,128,512]{2,1,0}) tuple(call.2)

}

Notice in Slide 3 why stage ④ ( Residual + LayerNorm) forms a 4-stage fusion group (%fusion #2 ~ #5) rather than a single monolithic loop: mathematically, LayerNorm must first aggregate all 512 elements of each token vector via a Sum Reduction (reduce) ( parallel_reduce-window followed by reduce) to obtain the token’s mean and variance before elementwise mean centering and scaling can proceed. XLA respects that reduction barrier while tightly fusing all surrounding pointwise math into %fusion #2 ~ #5 so unreduced intermediate tensors aren’t materialized in memory.

Because this HLO dump comes from the local CPU backend ( PJRT_DEVICE=CPU), the fusion callees carry CPU-multithreaded names ( parallel_*_fusion); on physical TPU hardware, the proprietary libtpu backend goes one step further, pipelining MXU dot outputs straight through VMEM into downstream VPU %fusion loops at the hardware level.

4. Before vs. After XLA Optimization Passes (Slide 2 vs. Slide 3)

Comparing Slide 2 (86 unfused individual ops) against Slide 3 (20 top-level instructions) reveals three classic compiler optimizations that XLA applies automatically under the hood. (In the listings below, broadcast replicates a lower-rank tensor – such as a 1D bias vector of length 2048 – across batch and sequence dimensions so it matches (4, 128, 2048).)

4.1 Eliminating Weight Transpose (transpose) and Redundant Identity Multiply (×1) in Stages ① & ③

  • Slide 2 (Pre-Optimization): Both ① fc1 and ③ fc2 emit explicit Weight Transpose (transpose) ops to flip their weight matrices, plus Redundant Identity Multiply (×1) ops (multiply(…, 1)) produced by mechanically lowering PyTorch’s addmm(beta=1, alpha=1) signature.
  • Slide 3 (Post-Optimization): XLA’s algebraic simplifier folds both transpose ops directly into the contracting dimension metadata (rhs_contracting_dims={1}) of dot.36 and dot.59 – eliminating weight layout shuffling – and deletes every no-op ×1 multiplication.

4.2 Algebraic Strength Reduction and %fusion Cluster #1 in Stage ② GELU

In Slide 2’s pre-optimization HLO, ① fc1 Bias Add and ② GELU span 14 separate unfused instructions – including a costly general-purpose Power Function power(h, 3) ( power.42):

// [Matches Slide 2: 14 Unfused Individual Ops Before Optimization]

reshape.37 = f32[4,128,2048]{2,1,0} reshape(dot.36)

broadcast.38 = f32[2048]{0} broadcast(constant.30), dimensions={}

multiply.39 = f32[2048]{0} multiply(p3.31, broadcast.38) // Redundant Identity Multiply (×1)

broadcast.40 = f32[4,128,2048]{2,1,0} broadcast(multiply.39), dimensions={2}

add.41 = f32[4,128,2048]{2,1,0} add(reshape.37, broadcast.40) // ① fc1 Bias Add (add)

multiply.56 = f32[4,128,2048]{2,1,0} multiply(broadcast.55, add.41) // 0.5 * h

power.42 = f32[4,128,2048]{2,1,0} power(add.41, broadcast.29) // Power Function power(h, 3)

multiply.45 = f32[4,128,2048]{2,1,0} multiply(broadcast.44, power.42)

add.46 = f32[4,128,2048]{2,1,0} add(add.41, multiply.45)

multiply.49 = f32[4,128,2048]{2,1,0} multiply(broadcast.48, add.46)

tanh.50 = f32[4,128,2048]{2,1,0} tanh(multiply.49)

add.53 = f32[4,128,2048]{2,1,0} add(broadcast.52, tanh.50)

multiply.57 = f32[4,128,2048]{2,1,0} multiply(multiply.56, add.53)

reshape.58 = f32[512,2048]{1,0} reshape(multiply.57)

In Slide 3’s post-optimization HLO, XLA replaces power(h, 3) with two fast register multiplications ( x * x * x substitution) and fuses everything from ① fc1 Bias Add through the trailing Zero-Cost Reshape (bitcast) into %fusion Cluster #1 (parallel_multiply_bitcast_fusion / fused_computation.4.clone):

// [Matches Slide 3: %fusion Cluster #1 (fused_computation.4.clone)]

fused_computation.4.clone {

...

add.24 = f32[4,128,2048]{2,1,0} add(bitcast.23, broadcast.54) // ① fc1 Bias Add fused in

multiply.31 = f32[4,128,2048]{2,1,0} multiply(add.24, add.24) // power(h, 3) -> x * x

multiply.30 = f32[4,128,2048]{2,1,0} multiply(add.24, multiply.31) // completes x * x * x

...

tanh.1 = f32[4,128,2048]{2,1,0} tanh(multiply.28)

add.22 = f32[4,128,2048]{2,1,0} add(broadcast.53, tanh.1)

multiply.27 = f32[4,128,2048]{2,1,0} multiply(multiply.32, add.22)

ROOT bitcast.22 = f32[512,2048]{1,0} bitcast(multiply.27) // Zero-Cost Reshape & return

}

4.3 Decomposing batch-norm-training and Rebuilding %fusion #2 ~ #5 in Stage ④ LayerNorm

In Slide 2, stage ④ LayerNorm spans roughly 50 individual operations centered on batch-norm-training and Reciprocal Sqrt (rsqrt) (which computes 1/√x directly without a slow division instruction). That happens because PyTorch’s XLA lowering initially reshapes the (4, 128, 512) activation into (1, 512, 512), allocates dummy scale/offset constants, and invokes XLA’s generic batch-norm-training op. XLA’s optimization pipeline strips away batch-norm-training along with more than 15 temporary constant initializers, expanding and re-fusing the math into hardware-friendly Sum Reduction (reduce) + %fusion Clusters #2 ~ #5 ( Gamma Scale · Beta Shift included) – shrinking the entire graph from 86 unfused ops down to 20 top-level ENTRY instructions.

5. Graph Breaks and the Host-Synchronization Anti-Pattern

5.1 Why Graph Breaks Happen

Does wrapping a model in torch.compile automatically guarantee the clean, single-module HLO graph shown in Slide 3? Not if the forward() method forces the CPU to inspect tensor values mid-flight. The most destructive performance anti-pattern in compiled PyTorch is placing a scalar extraction call like .item(), print(tensor), or a data-dependent Python if branch in the middle of forward():

def forward(self, x: torch.Tensor) -> torch.Tensor:

residual = x

h = self.act(self.fc1(x)) # [Subgraph #1 region]

if h.mean().item() > -100.0: # Triggers Graph Break + Accelerator-to-CPU sync!

h = self.fc2(h) # [Subgraph #2 region]

else:

h = self.fc2(h) * 0.0

return self.norm(residual + h)

While the Accelerator (Device) computes tensor h, the Python if statement must be evaluated by the Python interpreter running on the CPU (Host). The instant TorchDynamo encounters h.mean().item(), it cannot trace across the host-dependent branch, so it aborts single-graph capture and splits the model into two separate subgraphs – a Graph Break.

Two built-in diagnostic APIs expose Graph Breaks and their exact XLA compilation/execution counts immediately:

import torch._dynamo as dynamo

import torch_xla.debug.metrics as met



# 1) Inspect subgraph count and exact Python line causing the Graph Break

explanation = dynamo.explain(anti_pattern_model)(x)

print(explanation.graph_count, explanation.break_reasons)



# 2) Query actual XLA compilation count (CompileTime) and execution count (ExecuteTime)

print(met.metric_data("CompileTime")[0], met.metric_data("ExecuteTime")[0])

Table 3. Graph Break & Execution Diagnostics: Clean Sub-Block vs. .item() Anti-Pattern

Adding a single .item() call doubles both the subgraph count and XLA compilation count to 2 and forces a blocking Device → Host synchronization. Notice too that XLA Executions jump to 3: one launch for Subgraph #1, a second runtime execution to materialize and synchronize the scalar tensor state for .item(), and a third launch for Subgraph #2 once Python finishes evaluating the if condition.

5.2 Inspecting the Split HLO Modules

Opening the HLO files emitted via –xla_dump_to confirms that our unified module has fractured into two separate HloModule executables – reintroducing a costly multi-megabyte HBM round-trip right across the break:

1. First HLO Module (SyncTensorsGraph.48 — Subgraph #1):

HloModule SyncTensorsGraph.48, entry_computation_layout={

(f32[2048]{0}, f32[2048,512]{1,0}, f32[4,128,512]{2,1,0})

-> (f32[4,128,2048]{2,1,0}, f32[])

}

Look at the return tuple to the right of ->: in addition to the scalar mean ( f32[]) destined for the CPU, XLA is forced to materialize Forced Write of Intermediate Tensor h (f32[4,128,2048], 4.19MB) out to HBM so that the second subgraph can consume it later.

2. Second HLO Module (SyncTensorsGraph.59 — Subgraph #2):

HloModule SyncTensorsGraph.59, entry_computation_layout={

(..., f32[4,128,2048]{2,1,0}, ...) -> (f32[4,128,512]{2,1,0})

}

After finishing Subgraph #1, the Accelerator (Device) sits in an Idle Stall waiting for f32[] Sync to transfer back to the CPU (Host) and for the Python if Check to dispatch the next launch – at which point Subgraph #2 must Re-read Intermediate Tensor h ( 4.19MB) back out of HBM.

5.3 The Fix: Preserving a Single Unified Graph with torch.where

Rewriting the conditional logic with the tensor-native torch.where operator eliminates Host Sync altogether ( No Host Sync). As shown in the bottom half of the timeline comparison below, fc1 → GELU → torch.where → fc2 → LayerNorm compiles into a Single HLO Module with No Intermediate HBM Write · No Idle Stall:

cond = h.mean() > -100.0

fc2_out = self.fc2(h)

h = torch.where(cond, fc2_out, torch.zeros_like(fc2_out))

Keep one architectural trade-off in mind: unlike a Python if statement that executes only the taken branch, torch.where(cond, A, B) evaluates both tensor expressions A and B inside the compiled graph and selects elements via a predicated vector mask. For masking, scaling, gating, or threshold clipping like the example above, avoiding host synchronization and Graph Breaks is an overwhelming win; if A and B were two massive, independent neural network towers, however, computing both towers every step would waste FLOPs and require a routing-aware design instead.

Timeline Comparison: Anti-Pattern (.item() + Python if Graph Break) vs. Solution (torch.where Single Unified Graph)

6. Dynamic Sequence Lengths, HLO Recompilation, and Static Bucketing

6.1 Why XLA Specializes on Exact Tensor Shapes

Even when a model has zero Graph Breaks, real-world workloads present a second major hurdle: variable sequence lengths across batches. On CPUs and GPUs, passing dynamic=True to TorchInductor represents sequence length symbolically ( s0) and emits parameterized loop bounds such as for (int i = 0; i < s0; i++), allowing a single compiled kernel to handle arbitrary sequence lengths without recompiling.

TPU hardware works differently. Because the MXU systolic array and VMEM scratchpad lay out tensors in fixed 2D physical tiles such as 128 × 8 ( TPU hardware tiling guide), XLA specializes buffer offsets, tile padding, and loop trip counts on exact compile-time dimension constants. Consequently, every time an unseen sequence length arrives, XLA must run a full HLO Recompile from scratch.

6.2 Measuring the Cost of Unbucketed Variable Sequence Lengths

Holding batch size B=4 and hidden dimension D=512 constant while stepping sequence length S through 8 distinct values from 128 to 512 on the local CPU backend ( PJRT_DEVICE=CPU) exposes the recompilation penalty clearly:

Table 4. Step Latency & HLO Recompilations Without Bucketing Across Varying Sequence Lengths

Inductor pays a one-time compile cost on Step 1 and finishes Steps 2–8 in 13–35 ms with zero recompilations. Without bucketing, OpenXLA hits YES (HLO Recompile) on every single step – driving cumulative compilations linearly up to 8 and inflating step latency to 144-224 ms. Keep in mind that 144-224 ms reflects compiling a tiny 4-op sub-block on CPU PJRT; on a full multi-billion-parameter Transformer running on physical TPUs, global VMEM tile allocation and instruction scheduling turn each new shape into multi-second to tens-of-seconds stalls.

6.3 Eliminating Recompilations with Static Bucketing

The standard production pattern to stop recompilation storms is Static Bucketing. As illustrated below, we define a small set of fixed bucket sizes ahead of time (e.g., 128, 256, 512). Whenever a variable-length input arrives ( Length 95, Length 118, Length 140, Length 210, etc.), we round up to the smallest bucket that fits, fill the trailing positions with 0 Padding, and pass along a boolean valid_mask. Only the first arrival for each bucket triggers WARMUP_COMPILE and populates the XLA Compile Cache; every subsequent step mapping to an already-seen bucket executes immediately via CACHE_HIT (0 Recompile).

Comparison of Without Bucketing (HLO Recompile Every Step) vs. Static Bucketing (XLA Compile Cache Reuse)

Two implementation details are critical when writing Static Bucketing in practice:

  1. Any layer that aggregates across tokens — such as Self-Attention, sequence mean pooling, or cross-entropy loss — must apply valid_mask so 0 Padding tokens never contaminate real token representations.
  2. If you need to trim the output back to raw_len, do not slice dynamically on the accelerator tensor (out[:, :raw_len, :]) inside the compiled graph, because every distinct raw_len slice shape will trigger a brand-new XLA recompilation! Keep all accelerator tensor shapes strictly pinned to bucket_len and either mask the loss or slice after transferring the output to the host.
import torch.nn.functional as F



BUCKETS = (128, 256, 512)



def pad_to_bucket(x: torch.Tensor, buckets=BUCKETS):

raw_len = x.size(1)

bucket_len = next(b for b in buckets if b >= raw_len)

pad_len = bucket_len - raw_len

# Pad sequence dimension (second from last) on the right with zeros -> fixed bucket shape

x_padded = F.pad(x, (0, 0, 0, pad_len), value=0.0)

valid_mask = torch.arange(bucket_len, device=x.device)[None, :, None] < raw_len

return x_padded, valid_mask, raw_len

Feeding 12 continuously varying sequence lengths (from 95 to 480) through pad_to_bucket produces the following step latencies and cache behavior:

Table 5. Step Latency & XLA Compile Cache Hits with Static Bucketing

Warmup compilation ( WARMUP_COMPILE) happens only three times – on Steps 1, 3, and 6 when Bucket Sizes 128, 256, and 512 are encountered for the first time. Across every other step, even though raw sequence lengths never repeat, OpenXLA hits CACHE_HIT (0 Recompile), capping cumulative compilations at 3 and dropping step latency to 0.43-0.80 ms – roughly 300-390× faster than a WARMUP_COMPILE step ( 170 ms).

In Gen 3 native torch_tpu (summarized in Section 2.1), manual Static Bucketing is complemented by upper-bound dynamic compilation via torch._dynamo.mark_dynamic( x, dim=1, min=16, max=512) – which bounds allocation at max=512 while executing variable dynamic trip counts – and a 3-tier persistent compilation cache spanning in-process memory (L1), host shared memory (L2), and remote storage (L3) to amortize warmup across restarts.

7. Key Takeaways and Official References

7.1 Four Engineering Principles for TPU-Native PyTorch

  1. Minimize HBM Round-Trips Through %fusion: Real TPU throughput is bounded by data movement between off-chip HBM and On-Chip Memory (VMEM & Registers), not peak MXU/VPU FLOPs. Compile models through OpenXLA (backend=”tpu” on TPU VMs, or backend=”openxla” for local inspection) so chains of biases, activations, residuals, and normalizations collapse into single on-chip %fusion loops.
  2. Eliminate Host Synchronizations and Graph Breaks: Calling .item(), print(), or Python if on tensor values inside forward() fractures a single HLO module into multiple subgraphs, stalling the accelerator and forcing multi-megabyte intermediate tensors out to HBM. Audit models with torch._dynamo.explain to verify 0 Graph Breaks, and express data-dependent selection with torch.where.
  3. Bound Dynamic Shapes with Static Bucketing: Because XLA compiles fixed-shape tiles, unpadded variable sequence lengths trigger destructive HLO Recompile stalls every step. Pad inputs into a small set of fixed bucket sizes (128, 256, 512, etc.) paired with a valid_mask, avoid dynamic device-side output slicing, and pre-warm every bucket before serving traffic.
  4. Verify IR Locally Before Provisioning Cloud TPUs: Running PJRT_DEVICE=CPU with –xla_dump_to lets you verify single-module HLO capture, inspect %fusion clusters, and confirm plateauing CompileTime counts locally at zero hardware cost before deploying to physical TPU pods.

7.2 Official Documentation and Foundational Papers

Accelerator Architecture & Memory Hierarchy

* Google Cloud TPU System Architecture — Physical organization and specifications of HBM, VMEM, MXU, and VPU across TPU generations

* Google DeepMind How to Scale Your Model: All About TPUs — Deep dive into systolic arrays (MXU), VMEM scratchpads, 128 × 8 tiling, and arithmetic intensity

* AI and Memory Wall (Gholami et al., 2024) — Quantitative analysis of the growing divergence between compute FLOPs and memory bandwidth

Deep Learning Building Blocks

* Transformer: Attention Is All You Need (Vaswani et al., 2017)

* GELU Activation: PyTorch nn.GELU / Gaussian Error Linear Units (Hendrycks & Gimpel, 2016)

* Residual Connections: Deep Residual Learning for Image Recognition (He et al., 2015)

* Layer Normalization: PyTorch nn.LayerNorm / Layer Normalization (Ba et al., 2016)

PyTorch 2.x Compiler Stack

* torch.compile API Reference and Introduction to torch.compile Tutorial

* TorchDynamo Overview and torch.fx Graph Representation

* torch.compiler & TorchInductor Overview and Inductor CPU Debugging Guide

* torch.compiler Troubleshooting (Graph Breaks) and Dynamic Shapes (dynamic=True, mark_dynamic)

* PyTorch C++ / ATen Documentation and Custom Device Integration via PrivateUse1

OpenXLA & TorchTPU Ecosystem

* OpenXLA Overview, XLA Compiler Architecture (%fusion), and XLA HLO Operation Semantics (dot, bitcast, broadcast, reduce)

* StableHLO Specification and PJRT Runtime Integration

* Native TorchTPU Repository (google-pytorch/torch_tpu) and PyTorch/XLA Documentation ( GitHub)

* Cloud TPU v5e Documentation, PyTorch FSDP, and Profiling PyTorch/XLA on Cloud TPU VMs


PyTorch × TPU (TorchTPU & OpenXLA): Compiler Internals and Operator Fusion Optimization was originally published in Google Developer Experts on Medium, where people are continuing the conversation by highlighting and responding to this story.

Total
0
Shares
Leave a Reply

Your email address will not be published. Required fields are marked *

Previous Post
how-do-companies-build-their-own-ai?-a-look-inside-enterprise-ai-architecture

How Do Companies Build Their Own AI? A Look Inside Enterprise AI Architecture

Next Post
how-to-explain-technical-concepts-to-non-technical-people-in-english

How to Explain Technical Concepts to Non-Technical People in English

Related Posts