GPU vs TPU vs NPU for AI Workloads

The fundamental difference between GPU vs TPU vs NPU for AI workloads is the layer of the AI stack each chip is designed for and the trade-off each makes between flexibility and specialisation. GPUs are general-purpose parallel processors that sacrifice some efficiency for flexibility — they can train any model, run any framework, and serve any inference workload, making them the universal default. TPUs are Google’s purpose-built tensor math engines that sacrifice flexibility and cloud-agnosticism for cost efficiency — 2x cheaper at scale than GPUs for matched workloads, but only on Google Cloud and only for TensorFlow/JAX-friendly workloads.

NPUs are ultra-specialised inference-only chips that sacrifice training capability and model size for extreme energy efficiency — 40–60x more power-efficient than GPUs for edge inference, running on milliwatts inside your phone or laptop. In practice: GPU for training, TPU for Google-ecosystem cloud training and batch inference, NPU for on-device inference.

NVIDIA’s dominance stems from the CUDA software ecosystem rather than raw hardware performance alone. CUDA has 20+ years of development, 4 million+ developers, and every major ML framework (PyTorch, TensorFlow, JAX, ONNX) optimised for it first. When a developer writes a PyTorch training script, it just works on an NVIDIA GPU with no modification. Porting that same code to run efficiently on an AMD GPU or Google TPU requires significant engineering effort. This software moat means that even when AMD’s MI350X hardware matches or exceeds H100 specifications, the CUDA ecosystem creates real switching friction.

TPUs, despite their cost and efficiency advantages, are constrained to Google Cloud and work best with TensorFlow/JAX — most of the industry’s ML code is written in PyTorch. NPUs are measured in different units (TOPS for inference) vs GPU market share (which is measured in training and cloud inference) — so the 87% figure and the 970M NPU smartphone number measure fundamentally different markets that do not directly compete.

NPUs are inference-only — they cannot train AI models. Training requires iterative calculation of gradients, weight updates, and backpropagation across billions of parameters, which demands large amounts of high-precision floating-point memory (HBM in the tens of gigabytes), flexible programmability for different loss functions and optimisers, and full support for the mathematical operations of automatic differentiation. NPUs are fixed-function inference engines — they execute a frozen, compiled, quantised model against input data, but have no mechanism for updating weights or performing the iterative optimisation that training requires.

All model training happens on GPU clusters or TPU pods in the cloud. Once trained and compressed (quantised to INT8 or INT4), a model is compiled for the target NPU using vendor toolchains (Core ML, SNPE, OpenVINO) and deployed to device for inference. The training cloud and the inference edge device are fundamentally different parts of the AI lifecycle and currently require different hardware entirely.

CUDA (Compute Unified Device Architecture) is NVIDIA’s parallel computing platform and programming model — the software layer that allows ML frameworks and applications to run efficiently on NVIDIA GPUs. It includes a compiler, libraries (cuDNN for deep learning, cuBLAS for linear algebra, TensorRT for inference optimisation), profiling tools, and an extensive ecosystem of pre-optimised code. CUDA matters for the GPU vs TPU vs NPU decision because it represents 20+ years of engineering investment that makes switching away from NVIDIA GPUs genuinely difficult. When PyTorch or TensorFlow executes a training step, the GPU code path is CUDA — the same code does not run natively on AMD GPUs (which require ROCm) or Google TPUs (which require XLA compilation).

For most teams, this means: start on GPU with CUDA, and only evaluate TPU or custom silicon once you have validated your workload and can afford the engineering effort to port. AMD’s ROCm has narrowed the gap (performance difference with CUDA dropped from 40–50% to 10–30% on many workloads in 2025–2026), but the 10x developer activity gap between CUDA and ROCm remains the decisive practical factor.

AI chip performance is measured in several different units that capture different aspects of computational capability. TOPS (Tera Operations Per Second) measures integer operations — typically INT8 or INT4 — and is the most common metric for comparing inference chips including NPUs, where quantised integer math is standard. TFLOPS (Tera Floating-Point Operations Per Second) measures floating-point operations — FP32, FP16, or BF16 — and is most relevant for training and high-precision inference where numerical precision matters.

Higher floating-point precision delivers better model accuracy but lower raw throughput. PFLOPS (Peta Floating-Point Operations Per Second) is simply TFLOPS at a larger scale — the B200 GPU’s 1,100 PFLOPS FP4 means 1.1 quadrillion four-bit floating-point operations per second. Memory bandwidth (GB/s) measures how fast data can move between memory and compute — often the actual bottleneck for large model inference where weights must constantly be loaded from HBM. The critical insight is that TOPS numbers across different chip types are not directly comparable: 45 TOPS on an NPU at 4W is a fundamentally different capability profile than 3,958 TOPS on an H100 at 700W — same unit, completely different scale, purpose, and power envelope.

The economics are straightforward: an NVIDIA H100 costs approximately $3,320 to manufacture and sells for $28,000 — an 88.1% gross margin that represents the “NVIDIA tax” hyperscalers pay on every chip. At the scale of hyperscalers (AWS, Google, Microsoft, Meta each spending tens of billions annually on AI compute), even a partial replacement with in-house chips at manufacturing cost rather than retail price yields multi-billion dollar savings. Google’s TPU programme (begun in 2015) has been the model: Google builds chips precisely calibrated for its workloads (TensorFlow matrix operations), manufactures them at TSMC, and avoids paying NVIDIA’s margin. AWS followed with Trainium (training) and Inferentia (inference). Microsoft launched Maia 100. Meta developed MTIA.

OpenAI is building its first custom ASIC with Broadcom and TSMC for mass production in 2026. The trend is structural: custom ASIC shipments are growing 44.6% in 2026 vs 16.1% for GPU shipments. For enterprises that are not hyperscalers, the economics are different — the design cost of a custom AI chip starts at tens of millions of dollars, which is only justified when chip volume is large enough to amortise that cost below NVIDIA’s $28,000 retail price. Until then, GPUs remain the economically rational choice despite the premium.

In 2026, flagship NPUs can run models up to approximately 7 billion parameters when aggressively quantised to INT4 or INT8 precision. Apple’s M4 Neural Engine (38 TOPS) and Qualcomm’s Hexagon NPU (45 TOPS) can handle models like Phi-3 Mini (3.8B parameters), Mistral 7B (quantised), Gemma 2B, and Llama 3.2 3B at acceptable inference speeds for interactive use. The practical constraint is memory: flagship smartphones have 8–16GB of LPDDR5 shared between the CPU, GPU, and NPU — a 7B model at INT4 requires approximately 3.5–4GB of memory, which is feasible on devices with 12GB+.

Performance is approximately 20–40 tokens per second on flagship NPUs for 3–7B models, which is adequate for text completion and summarisation but slower than cloud LLM APIs. Models above 7–13B parameters remain impractical for NPU deployment in 2026 — they require too much memory and produce output too slowly. Qualcomm’s AI200 data centre chip (entering market in 2026) blurs the NPU/GPU boundary by targeting inference at data centre scale with NPU-style efficiency at much larger model sizes.

For most teams training LLMs in 2026, GPU is the recommended starting point regardless of eventual scale, for three reasons. First, PyTorch is now the dominant framework for LLM training, and PyTorch works natively on CUDA GPUs without the XLA compilation step that TPUs require. Second, the tooling, debugging, and community knowledge for GPU-based LLM training is vastly richer — Hugging Face Transformers, DeepSpeed, Megatron-LM, and every major LLM training library targets CUDA GPUs first. Third, GPU compute is available from any cloud provider, enabling cost comparison and workload portability.

TPU makes sense for LLM training when: you are already on Google Cloud with a significant existing TensorFlow or JAX investment, you are training at a scale where the 2x cost advantage represents millions of dollars in savings, and you have the engineering capacity to port your training code to work efficiently on XLA. Anthropic’s order of 1 million TPUs from Google signals that the largest frontier AI labs at extreme scale are evaluating the economics of TPU at their specific workload profiles — but for most teams training in the 1B–70B parameter range, GPU remains the pragmatic default.

The GPU vs TPU vs NPU landscape is converging and fragmenting simultaneously. Convergence: the boundary between NPU and GPU is blurring as mobile SoCs add more capable NPUs that run increasingly large models, and as specialised inference ASICs (Groq LPU, Positron Atlas) attack the inference market from below GPU price points. The boundary between TPU and custom ASIC is blurring as every hyperscaler builds its own Google-style custom silicon. Fragmentation: the training market, the cloud inference market, and the edge inference market are increasingly served by different chip families rather than one universal chip. For training: GPUs remain dominant, but custom ASICs from hyperscalers are growing 44.6% per year, taking large-scale training workloads away from NVIDIA over time.

For cloud inference: GPUs compete with TPUs, specialised inference ASICs, and the next generation of efficient chips optimised specifically for transformer serving. For edge inference: NPUs are dominant and expanding — on-device LLMs become practical for larger model classes as NPU TOPS ratings and device memory expand annually. IBM noted that 2026 is “the year of frontier versus efficient model classes” — a split between massive cloud-resident models and efficient device-resident models that directly maps onto the GPU/TPU vs NPU hardware distinction.

GPU, TPU, and NPU are the three most widely discussed AI accelerator categories, but the full AI chip landscape is broader. ASICs (Application-Specific Integrated Circuits) are the parent category that includes TPUs and NPUs — any chip custom-designed for a specific workload rather than general-purpose computation. AWS Trainium (training) and Inferentia (inference), Microsoft Maia 100, Meta MTIA, Tesla Dojo, Qualcomm AI200, and OpenAI’s forthcoming custom chip are all ASICs — purpose-built silicon optimised for specific AI workloads. FPGAs (Field-Programmable Gate Arrays) are reprogrammable chips that can be reconfigured in software after manufacturing — AMD (Xilinx) and Intel (Altera) are the dominant FPGA makers.

FPGAs offer lower latency than GPUs for specific inference tasks and better flexibility than fixed-function ASICs, but lower raw performance than either — making them most suitable for ultra-low-latency inference applications like algorithmic trading or 5G baseband processing. The hierarchy runs: CPU (most flexible, worst for AI) → GPU (highly parallel, good for AI, flexible) → FPGA (reconfigurable, specific use cases) → TPU/ASIC (purpose-built, excellent for specific workloads, less flexible) → NPU (fixed-function inference, most efficient for on-device AI). In 2026, the market is decisively moving toward more specialised silicon at every layer of the AI stack.