Hardware: The engine of innovation

Unlike traditional GPUs that partition silicon into thousands of Streaming Multiprocessors with complex Single-Instruction Multiple-Thread scheduling, a TPU chip uses massive "TensorCores" (usually 1–2 per chip) plus auxiliary SparseCores. Operating on large data blocks via a few big cores provides a simpler programming model. By eliminating massive thread queues and dynamic branch logic, execution is exceptionally predictable, allowing compilers to schedule computations with zero runtime overhead.

GPUs rely on warp registers to fetch operands for every math step, while TPU's MXU uses a systolic array where data streams through a 2D grid of processing elements. Inputs flow horizontally and vertically, reusing operands across cycles without writing intermediate results back to registers. Accumulating in full FP32 precision to ensure high numerical range, this structural predictability is why matrix dimension alignment is critical to maximizing execution efficiency.

Most accelerators rely on hardware L1/L2 caches, causing runtime latency jitter and non-deterministic cache misses. The TPU’s local memory hierarchy is entirely compiler-controlled. Vector Memory is a high-speed scratchpad explicitly staged from High-Bandwidth Memory by an async Direct Memory Access unit under XLA compiler control. This makes data movement static and predictable, allowing offline performance simulators to accurately map execution timelines ahead of time.

Standard GPU fabrics use high-power, latency-sensitive electrical switches. TPU pods are wired as 2D/3D torus topologies connected via Google’s Optical Circuit Switches, which use micro-mirrors to route raw light paths. Establishing direct, physical fiber connections at the start of a training job eliminates packet-routing overhead, drastically cuts latency, and lets the compiler dynamically carve massive pods into isolated, fault-tolerant slices on the fly.

We prioritized range over precision to cut memory traffic and boost density with TPUv2’s Brain Float 16 (FP32 range with 7-bit mantissa), then doubled compute throughput by adding 8-bit floating point (E4M3/E5M2) to TPU v5p and v7. TPU 8t/8i introduced native 4-bit floating point (FP4) support directly in the systolic array, delivering up to 4x throughput over BF16. To ensure reliability at low precision, TPUs enforce bitwise determinism, hardware replay, and Functional Built-In Self-Tests against Silent Data Corruption, guaranteeing reproducible runs for large-scale debugging.

While conventional server fleets treat thermal management as a mechanical packaging afterthought, TPU supercomputers integrate thermal dissipation as a foundational architectural constraint. Beginning with TPU v3 in 2018, Google transitioned all frontier training systems to direct-to-chip liquid cooling. Liquid cooling addresses three foundational scaling bottlenecks: