Cutting AI carbon requires more than energy efficiency

One major criticism of generative AI is its carbon footprint, primarily driven by high energy consumption. A recent report explained that energy per token for inference is driven primarily by two factors: infrastructure utilization and throughput (see The problem with energy per token). Throughput reflects how efficiently a model generates tokens during active inference periods, while utilization reflects how efficiently the underlying hardware is utilized over its operational lifetime.

Converting energy per token into carbon means multiplying token energy by the carbon intensity of the electricity used at that point in time. However, it is rarely that simple in practice.