The goal of commissioning is to prove a data center’s readiness to support the IT load at design capacity by demonstrating and documenting the performance of power and cooling systems. Direct liquid cooling (DLC) presents challenges at all stages of commissioning, in part because it physically couples facilities and IT equipment in new ways. Best practices for commissioning liquid-cooled mission-critical environments are still maturing.

Coolant distribution units (CDUs) are at the center of the issue. The commissioning of CDUs and the connected equipment in the technology cooling loop is not always straightforward for a range of reasons. Some manufacturers of CDUs have (or used to have) limited experience with either the minutiae of integrating and testing a CDU in a larger, complex fluid network, or the wider data center commissioning process. Some of these companies had not sold any products to data centers before introducing their CDUs. This does not imply that the manufacturers cannot build a performant and reliable product — but factory witness testing (FWT) of their CDUs to verify their capabilities under real-world conditions is crucial.

For operators who commission their data centers using a conventional, thorough commissioning script, any quality or performance issues will add costs and delays. FWT helps uncover these issues early in the commissioning process, giving manufacturers that do offer robust FWT capacity an advantage. FWT is still not standard practice, although it is becoming a more common requirement. When speed takes priority, and/or CDUs are retrofitted into an existing facility, FWT is more likely to be overlooked. For example, hyperscalers deploying superclusters for generative AI training might proceed without FWT to produce a trained model quickly, and will rely on the manufacturer’s assurances of performance and reliability.

Defining test conditions

Verifying CDU performance and reliability may be among the first hurdles the operator faces in commissioning DLC systems, but it will likely not be the last. Data center commissioning typically comprises five sequential “levels,” beginning with factory testing of individual equipment units and culminating in a combined test of electrical and cooling systems in the data center.

Critically, DLC also complicates these system tests because the tests require liquid-cooled load banks, which are not yet standardized or widely available. Liquid-cooled load bank designs and best commissioning practices are still evolving. Uptime Intelligence has described liquid-cooled load banks that are still under development in a previous report (see Validating the use of high-density DLC), and future research will cover improvements and the push toward standardization.

This report focuses on CDUs that connect to facility water for heat exchange. A CDU’s nameplate capacity (in kW) and test performance depend on several variables, including:

• IT coolant type and physical properties (e.g., density, viscosity, specific heat capacity).
• IT coolant supply temperature, return temperature, flow rate.
• Approach temperature (difference in °F or °C between incoming facility water and outgoing IT coolant).
• Filtration method (particle size, integrated or discrete filter placement) and overall pressure drop in the fluid loop.

Escalating CPU and GPU heat output over the past few years shifted the typical values of these variables, but also drove some standardization. IT coolant flow rates are higher: ASHRAE and the Open Compute Project Foundation (OCP) now recommend 1.5 L/min·kW (liters per minute per kilowatt) of heat load as a standard for CDU testing, up from less than 1 L/min·kW several years ago. Narrow approach temperatures (3-4°C or 4.8-9.0°F) are also becoming more common.

Constructing and operating test facilities for FWT can be costly and complex, even when designing for a narrow range of test conditions. Past changes in CDU test variables suggest they will likely continue to evolve, even after the introduction of standards. Designing a flexible test facility to handle future CDUs adds further complexity.

Even in the short term, a single well-defined test program may not be sufficient for all end users. Some end users build data centers with design assumptions that differ from the standardized or most common values described above. Often, these customers prefer to tailor the FWT to match their own expected operating conditions (sometimes called an “application test”).

However, expense and complexity alone would not justify proceeding without FWT if it would render the manufacturer’s CDUs unacceptable to their customers. Conventional commissioning procedures depend on FWT to plan and execute the subsequent stages that demonstrate the cooling system’s capability. Bringing a data center online with CDUs that are not factory tested requires multiple stakeholders to approve an exception to this standard, which could introduce risk.

Outlook for factory witness testing

The minority of data center customers willing to fit out a data center without going through FWT first are likely to be the ones that are under the greatest pressure to deliver an operational facility quickly. Developers of frontier generative AI models (which are in relative infancy) currently compete over speed to market, and this urgency flows down to supporting training infrastructure. Operators hosting these training workloads are the most likely to deviate from the conventional commissioning script and forgo full FWT — with the blessing of their IT end users.

First, generative AI training represents a large share of commercial DLC deployed today and consumes most of the manufacturing capacity of DLC manufacturers. The large high-density superclusters used to train the frontier models create enough latent demand for CDUs to attract new competitors— even if they cannot offer FWT.

Unlike most conventional enterprise IT, AI training superclusters are homogeneous in their IT and cooling designs, and are essentially composed of repeatable units. Organizations working with urgency to produce a trained large language model and accelerate data center commissioning may be more willing to engage third-party testing on only a small number of CDUs. They may then assume that the remaining units are identical and will perform similarly.

This urgency to achieve speed to market is currently unique to generative AI and is likely to ease as AI applications and their supporting infrastructure mature. Manufacturers not offering FWT or application testing for their CDUs may consider adding or expanding testing capabilities, as the priorities of their largest customers are likely to shift over time.

Today, operators handling workloads that require a higher standard of resiliency and a conventional commissioning process are less likely to need to accommodate liquid cooling in their facilities in the first place. As liquid cooling becomes more common, eventually including more mission-critical workloads, major CDU manufacturers may invest in FWT and application testing capabilities to meet their customers’ commissioning needs.

Other related reports published by Uptime Institute include:
Validating the use of high-density DLC