Token Throughput

What is token throughput?

Ask how fast an AI service is and you're really asking about token throughput — how many tokens the system can process per unit of time, usually "tokens per second". The bigger the number, the faster the same job gets done.

Is it the same as latency?

No, but they're related
Latency is how long one request takes from send to response; throughput is how much the system can process per second overall. Think delivery: latency is how long your parcel takes, throughput is how many parcels the warehouse ships per day. High throughput is what keeps the service up when lots of people pile on.

What shapes throughput?

Hardware and compute
More and stronger GPUs mean more tokens computed at once.
Model size
A big model costs more per token, so on the same hardware throughput usually drops.
Batching and parallelism
Processing several requests together can lift throughput a lot.
Optimizations
Quantization, speculative decoding and KV caching all push throughput higher.

Why it matters

For users, low throughput means queues, stutters and long waits. For providers, throughput decides cost: at the same compute, higher throughput means cheaper cost per token — which is how a product stays both affordable and able to absorb traffic.

Bottom line: token throughput is the AI's work rate — more tokens per second, faster service that holds up under load.