What Is a TOPS Score (and What Does It Actually Measure)?

Quick answer
TOPS means trillions (10^12) of operations per second, a peak throughput number for an AI accelerator.
The number is meaningless without the precision it assumes: INT8 TOPS, INT4 TOPS, and FP16 TOPS are different measurements of the same chip.
Vendors often quote sparse or theoretical peaks that a real model cannot reach, because memory bandwidth and operator support become the limit.
Use TOPS to compare chips in the same family at the same precision, and use model benchmarks to compare systems.
The more useful device metric is performance per watt at a specific model, not peak TOPS.

What does TOPS measure?

TOPS counts operations per second in trillions. An operation is typically a multiply-accumulate, so the raw number is large: a chip rated at tens of TOPS can perform tens of trillions of these per second under ideal conditions. For neural networks, that capacity maps to matrix multiplications, which dominate inference and training.

It is a peak figure. It assumes the chip is fully occupied, the data is already in the right place, and the operations are the ones the hardware accelerates best. A real model spends time on memory movement, activation functions, and operators the chip handles less efficiently, so achieved throughput is lower.

Why are TOPS numbers hard to compare?

Precision changes the answer. A chip can often execute INT8 operations at twice the rate of FP16, and INT4 at twice again, so a vendor can increase the headline number by lowering precision. That is legitimate for models quantised to that precision, and misleading when the model runs at a higher one.

Sparsity is the second variable. Some accelerators skip zero weights and advertise a sparse peak that assumes a specific sparsity pattern. Real models may be dense or have different sparsity, so the sparse number overstates the achievable rate. Memory bandwidth is the third: a chip with fast arithmetic and slow memory waits on data.

| Claim                    | What it assumes                          | Why it may not apply            |
|--------------------------|------------------------------------------|----------------------------------|
| 40 TOPS INT8             | Dense INT8, all units busy               | Model may run FP16               |
| 80 TOPS sparse INT8      | 2:4 sparsity, compatible kernels         | Dense model, no sparsity         |
| 20 TOPS FP16             | Dense FP16                               | Quantised model could run faster  |
| Peak at max clock        | Sustained boost, no thermal limit        | Laptop or phone throttles        |

What should you measure instead?

Measure the model: tokens per second for a language model, inferences per second for a vision model, and latency at the batch size you will actually use. Record the precision and the memory footprint. Then measure power draw if the device is battery-powered or thermally constrained.

For servers, measure throughput at a realistic concurrency and the cost per million tokens. For devices, measure sustained performance after several minutes, because the first thirty seconds often run at a boost clock that cannot be maintained. These numbers answer the product question; TOPS only describes the silicon.

When is TOPS still useful?

It is a rough filter. If two chips in the same family at the same precision differ by a large factor, the higher number is likely faster for workloads that saturate both. It also communicates the class of the hardware in a way that a model-specific benchmark does not.

Use it to shortlist, then discard it for the decision. A specification that does not name the precision, the sparsity assumption, and the memory configuration is marketing rather than engineering. Ask for the benchmark on your model or run it yourself.

Where this bites vibecoders
Hardware marketing in an AI-generated product brief tends to quote the largest TOPS number on the page. The feature then runs a model at a different precision and misses the target. Define the model, precision, and latency first, then check whether the device sustains it.

Where AI coding assistants get this wrong

Comparing TOPS figures without checking the precision.
Assuming sparse peak numbers apply to a dense model.
Ignoring memory bandwidth when the model does not fit in cache.
Using peak clock figures for a thermally constrained device.
Treating TOPS as a proxy for tokens per second.
Choosing hardware before confirming framework support.

Checklist

Find the precision behind every TOPS claim.
Check whether the figure assumes sparsity.
Confirm the memory capacity and bandwidth.
Benchmark the actual model at the target batch size.
Measure sustained performance, not the boost peak.
Compare performance per watt for device workloads.

FAQ

What does 40 TOPS mean in practice?

It means the accelerator can theoretically perform 40 trillion operations per second under the conditions the vendor specifies, usually a particular precision and density. It does not mean a specific model will run at a specific speed. The achieved rate depends on memory, operators, and thermal limits.

Is TOPS the same as FLOPS?

They are related but not identical. FLOPS counts floating-point operations per second; TOPS counts operations more broadly, often including integer operations used in quantised inference. A TOPS figure at INT8 is not comparable to a FLOPS figure at FP32 without conversion and context.

Why do NPUs quote higher TOPS than GPUs?

Often because they quote a lower precision, count different operations, or exclude memory movement. A device NPU may also be optimized for a narrow set of models. The comparison is not apples to apples, which is why model benchmarks exist.

Does higher TOPS mean lower latency?

Not necessarily. Latency depends on the model’s critical path, memory access, batch size, and scheduling. A chip with high peak throughput can have higher latency for a single small request than a lower-TOPS chip optimized for that case. Measure the latency you need.