AI tokens per megawatt: measuring data center output and efficiency

Start with the work, not the chip count

NVIDIA describes AI factories in terms of output, including token throughput, energy efficiency, utilisation and cost per token. That framing shifts attention from how many accelerators a building contains to what the installed system can deliver. It is a vendor’s framing of AI infrastructure, not a replacement definition for every data center.

A language model processes and generates tokens: units in its representation of text. More output is useful only if it serves the task. A response that arrives too late, fails an accuracy requirement or generates unnecessary material can raise a token counter without improving the application. Enterprise databases, storage and many scientific workloads also need measures other than generated text tokens.

NVIDIA MLCommons

The framework is bigger than four headings

In March 2026, Jensen Huang set out NVIDIA’s five layers: energy, chips, infrastructure, models and applications. The infrastructure layer includes land, power delivery, cooling, construction and networking. The point is their dependence on one another.

This magazine uses Land, Power, Cooling and Compute as editorial desks. They are our way of following those connections, not NVIDIA’s exact taxonomy or an endorsement. Tokens belong within Compute because they help describe some AI workloads; the desk also covers networks, storage and the software that makes hardware useful. A physical building and an AI application require several kinds of evidence.

NVIDIA, Jensen Huang

Throughput needs a response-time promise

An offline workload can wait while a system processes a batch. A live conversation has to start responding and keep responding at an acceptable pace. MLCommons separates these scenarios in MLPerf Inference. Its language-model benchmarks use token throughput alongside time to first token and time per output token, with defined accuracy requirements.

That makes a practical comparison more demanding than copying the largest number from a chart. Keep the model, input and output lengths, numerical precision, workload pattern and latency conditions beside the result. Ask whether it measures the whole installed system or only part of it, and whether the configuration is available or a preview. A benchmark is a defined test, not a guarantee of production traffic.

MLCommons MLCommons

An open supercomputer cabinet with rows of blue and red coolant hoses between dark doors — An open Frontier cabinet with the blue and red coolant hoses that serve its compute blades, 2022. Oak Ridge National Laboratory / CC BY 2.0

Choose the meter before comparing efficiency

Tokens per second divided by power gives an energy-efficiency relationship only when both measurements cover the same interval and boundary. Total tokens divided by joules is another way to express it. GPU-only power and whole-facility power produce different answers; neither should be presented as the other.

PUE can help describe the facility overhead around IT equipment, but it cannot establish model accuracy, server utilisation or cost per useful answer. Keep the electrical boundary explicit and state what other equipment is included. A more efficient cooling plant improves one part of the system; the application still determines whether the computation was worthwhile.

Google Data Centers MLCommons

A better operating question

Ask how much acceptable work the system delivered within its service target and measured energy budget. Record failed requests, response-time tails and the workload mix alongside the total. For a commercial comparison, add the stated accounting period and the costs actually included, rather than treating the electricity bill as the whole cost of production.

This keeps tokens in their proper role: a useful measure with conditions attached. A strong AI infrastructure story follows the chain from supplied electricity, through cooling and servers, to an application that does its job. It does not confuse a busy machine with a valuable result.

MLCommons NVIDIA