The problem with energy per token

As adoption of generative AI grows, the energy demand of this technology is becoming an increasingly important consideration for data center operators, end users and policymakers. However, infrastructure power consumption alone reveals little about the useful work being performed: the same AI cluster may deliver very different levels of output depending on how it is configured and used.

Energy-per-token metrics aim to address this problem by linking energy consumption directly to the AI-generated output. Tokens are fragments of words that represent the basic units of data manipulated by most large language models. By expressing efficiency as joules per token (or its derivatives such as tokens per watt), data center operators can compare AI models, hardware platforms, and inference architectures using a common unit tied to application activity. The metric is also attractive because it can be translated into economic and sustainability measures, including cost per token and carbon emissions per token.