GPU power management is a work in progress

Specialized IT hardware designed to perform generative AI training and inference is much more efficient at the job than generic IT systems. However, it is becoming clear that GPU-heavy servers are subject to many of the same problems that negatively impact the efficiency of traditional enterprise IT: low infrastructure utilization, high idle power and wasteful peak performance states.

The data center industry is at the precipice of an escalating power problem. Infrastructure for generative AI consumes a substantial amount of power and accounts for a significant portion of new data center capacity. Training of large language models (LLMs) currently dominates AI infrastructure needs, with the objective of squeezing the most performance out of hardware to shorten training times. Even if total energy is often a secondary consideration for AI training runs, system utilization levels are high, which means relatively high (if suboptimal) power efficiency.