What Is an NPU (and Do You Need an 'AI PC')?

Quick answer
An NPU is a small, power-efficient accelerator built into a processor, designed for local inference rather than training.
Its selling point is efficiency: a few watts for always-on tasks like transcription, background blur, and small local models.
TOPS measures how many operations per second it claims, but the number is only meaningful for the precisions and models the vendor tested.
An NPU is far slower than a discrete GPU and has no dedicated memory, so it does not change what large models you can run.
Its practical value today is battery life and privacy for small on-device models — not training, and not hosting a large model.

What is an NPU?

A neural processing unit is a block of silicon designed to do one thing efficiently: run the arithmetic of a neural network. It has appeared in phones for years for tasks like face unlock and photo processing, and it now ships inside mainstream laptop processors, which is what the ‘AI PC’ marketing refers to. It sits alongside the CPU and integrated graphics on the same chip, and it is built for a specific trade: much lower performance than a graphics card, in exchange for far lower power draw.

That trade is the whole point. A discrete GPU can consume hundreds of watts and needs its own cooling; an NPU runs inside a laptop’s thermal budget and can stay active continuously without visibly draining the battery. For a model that runs constantly in the background — live captions, noise suppression on a call, a local assistant — doing the work in a few watts instead of waking the GPU is a real improvement in both battery life and heat.

What TOPS actually tells you

NPU marketing is dominated by TOPS, trillions of operations per second. It is a throughput figure like any other, and like gigahertz it says less than it appears to: the number depends on which numeric precision was used, how the operations were counted, and whether the workload was shaped the way the hardware likes. Vendor figures are usually measured at low precision on a favourable benchmark, so two chips quoting 40 and 45 TOPS are not meaningfully different in practice.

More importantly, TOPS says nothing about whether your model runs. An NPU has no dedicated memory of its own; it works out of the same system RAM as everything else, so its ceiling is what the laptop’s memory allows. There is also a software problem: models must be exported to a specific format and often compiled for the particular NPU, and the tooling is immature and fragmented compared with the GPU ecosystem. A model that runs beautifully on one chip’s runtime may fall back to the CPU on another.

NPU vs GPU vs CPU

Three accelerators, three shapes. The CPU is general-purpose and slowest at dense tensor maths, but it can do anything and is always there. A GPU has thousands of cores, its own high-bandwidth memory, and the mature software ecosystem that AI frameworks are built around; it is what training and large-model inference require. An NPU is a small, purpose-built block that does a narrow subset of those operations very efficiently, running out of shared system memory.

The consequence is a clean division of labour. Training and fine-tuning: GPU always. Serving large models with high throughput: GPU, on a machine where the memory is dedicated to it. Small models running continuously on a device you carry: NPU, where the win is measured in watts. A laptop NPU does not let you run a model that a discrete GPU could not, and it will not make a large model fast — it makes a small model cheap to run.

Does it change what you should build?

For most application developers, not yet. If your product calls a hosted model API, the accelerator on the user’s device is irrelevant. If you ship a model inside a desktop or mobile app, an NPU is worth investigating for anything that must run offline or continuously, but expect to test on each device family and to keep a CPU fallback, because runtime support varies and falls back silently when it is missing.

The honest summary is that NPUs make on-device inference cheaper to run rather than newly possible. The interesting applications are the ones that only work if the model is local and always listening — transcription, ambient assistance, privacy-sensitive processing that never leaves the machine — and those are a different product shape from an app whose heavy lifting happens in a data centre. The GPU guide explains the tier above; this one is about the tier that fits in a battery budget.

Where this bites vibecoders
NPUs show up in generated plans as an assumption rather than a decision. An assistant asked to build a local feature may specify on-device inference because the hardware is advertised as capable of it, without checking that the runtime supports the specific chip, that the model exports to that format, or that the fallback path exists when it does not. The result is code that works on the developer’s laptop and silently runs at a fraction of the speed — or fails to load — on someone else’s. Testing on the target hardware, with a CPU fallback, is the only reliable check.

Where AI coding assistants get this wrong

Treating an NPU as equivalent to a discrete GPU for model hosting.
Choosing a model based on an NPU’s TOPS figure without checking format support.
Assuming an on-device model will run on any user’s machine with the same performance.
Omitting a CPU fallback for when the accelerator or its runtime is unavailable.
Comparing TOPS figures across vendors as though they were measured the same way.
Forgetting that an NPU shares system memory with the rest of the machine.

Checklist

Confirm the runtime supports the exact accelerator and model format you ship.
Keep a CPU fallback path and test it.
Measure real latency and battery impact instead of trusting TOPS figures.
Check system memory headroom if the local model is large.
Test on each device family you intend to support.
Use a GPU or an API for anything that must serve large models at scale.

FAQ

Do I need an AI PC to run local models?

No. Local models run on the CPU, on integrated graphics, and on discrete GPUs. An NPU makes small always-on models more power-efficient, which is a battery and heat benefit rather than a new capability. The constraint that decides whether a model runs is memory, not the presence of an NPU.

How fast is an NPU compared with a GPU?

Much slower for AI workloads — often an order of magnitude or more for the operations GPUs are built around. Its advantage is power efficiency, not throughput. An NPU is designed to keep a small model running continuously within a laptop’s thermal budget, not to train or serve large models.

What does TOPS mean?

Trillions of operations per second, a raw throughput figure for the arithmetic the chip can perform. It is measured at a particular numeric precision on a favourable workload, so it is a rough indicator at best. It also says nothing about memory capacity or software support, both of which matter more in practice.