Event Recap
RECAP | ROUNDTABLE | Extreme Cold Weather - Data Center Considerations
The United States has experienced extreme cold temperatures and weather conditions across much of the country, but in particular in areas of the southwest. Texas issued a statewide emergency. Major data centers in Texas ran on engine-generator to reduce grid load and stay online. There have been reports of data center outages as well.
The purpose of this impromptu roundtable was for attendees to provide their insight and experiences in regard to data center operations, during and preparing, for extreme cold conditions. As an example, are you doing anything extra with your engine-generators, fuel, mechanical equipment, natural gas and water supply, etc.? Do you know how extreme cold impacts the equipment design capacity of your data center? Do you have a proactive plan to address extreme temperatures, cold and hot, and what constitutes an effective plan? And don’t forget the onsite staff – what’s in place to support them during these conditions?
Roundtable comments:
The roundtable had over 20 attendees and over 15 member companies represented, all primarily interested in sharing experiences on temperature extremes and to compare notes on how to address these conditions. A number of attendees were from San Antonio and Dallas, Texas area, as well as traditional cold weather areas like the Minneapolis, MN area. One attendee commented that inclement weather planning helps data center operations a lot, but planning for a 100 year cold extreme is not normal. They are hoping to get a number of those “gotchas” from the session. Another attendee who deals primarily with data center risk indicated they are looking for what worked and what didn’t work in the way of mitigation.
Ryan Orr, Uptime Institute VP of Topology Services, kicked off the conversation by asking how many know the specific minimum temperature your data center is designed for. A couple of attendees appear to know the minimum design air temperature, but it appeared understanding the minimum design temperature and how this impacts capacity was not something commonly known. Ryan went on to explain how Uptime Institute uses ASHRAE design data for extreme temperatures, where looking at the N=20 value is the norm, but we encourage you to use something more if you think it is necessary. As an example, the ASHRAE N=20 low end temperature for Tulsa, OK, is -5 degrees F, but during this recent cold spell Tulsa got down to -19 degrees F. The key point is to first understand your boundaries and then, if there is a concern, put appropriate operations measures in place to address the concerns. An attendee commented they had to look beyond their present operational plans to address this rare extreme cold risk case.
An attendee from Minnesota where extreme cold is more common stated you need to plan and drill for these extremes. In MN, evaporative cooling is used quite a lot. During extremes, their technicians need to conduct more frequent rounds to look for things, like ice buildup, etc. Being able to run cooling tower fans in reversal is a key item as well.
The focus shifted to engine-generators (EGs). Here are some of the items discussed.
• Consider purchasing blended diesel fuel to eliminate wax accumulation in extreme cold. An attendee in MN commented they purchase extreme blended fuel and then run off the fuel throughout the season so they can replenish it after the winter.
• Examine the configuration of your fuel oil delivery system – inside vs. outside delivery. Most fuel oil deliveries are conducted outside, which could be an issue in extreme cold. Consider running fuel lines underground for cold temperature protection.
• Options can be purchased on EGs for cold running environments to help mitigate risk.
• In outdoor EG applications, engine coolant heat exchangers need to be examined.
• For outdoor diesel tanks, water build up in extreme cold can cause a freezing issue. Desiccant breathers can help, but onsite staff still need to be more concerned of this and do more frequent checking.
• Refueling for a number of attendees was an issue during the event. Fuel trucks were unable to get to the site. Therefore, they are now looking at more fuel storage for longer runtime, which comes with more concerns and exposures.
Ryan mentioned fuel is more sensitive than people think to cold temperatures. Therefore, it needs to be deeply looked at. It’s not just the potential for freezing the line that can get you, but also how it impacts fuel quality (for example, viscosity, gelling, etc.).
A number of attendees in the Texas area indicated they went on EGs because of the unstable power grid. The utility indicated they were implementing rolling blackouts, but in the beginning they ended up being mostly blackouts. However, a number of attendees indicated once utility went to rolling blackouts, they came off EGs because of greater concerns over the potential fuel supply.
The conversation shifted to other mechanical equipment. Ryan asked if anyone had a component that had cold starting issues, or was the approach to just run everything. One attendee indicated they had issues cold starting a chiller that sat for too long in the cold. Ryan touched on how we’ve seen issues in Russia where they modified their operations standard to factor in the cold. We’ve also seen these types of cold starting issues with any refrigerant based equipment. As sites utilize more automation and rotation of equipment, you may decide in extreme conditions to start all components and run them, or take out of auto mode and manually start and rotate equipment. One attendee commented if air cooled chillers are your basis, there are limits for extremely low ambient temperatures because of evaporator tubes freezing. Another attendee commented they run 15% glycol solution in Texas.
Next, staffing considerations were discussed. One attendee indicated they purchased cots, dry food, and have other accommodations on site, like a shower room, in case critical staff have to stay onsite for an extended period of time. Another commented how they planned ahead and adjusted their staffing schedule to make sure people could get to the site. A third attendee mentioned how they had an employee stay onsite 2-1/2 straight days. They discussed this with the employee beforehand about staying, so the site and the employee were prepared. An important point brought up by an attendee was that law enforcement have not allowed people on the roads during extreme conditions, so now they have letters that key employees keep in their vehicles stating they are essential employees.
Lastly, Ryan brought up how make-up water and supply water should also be a concern. The risk to a long term extended outage of water should be analyzed. As we saw in Texas during and after the extreme cold event, public water systems can be an issue with lines freezing, water mains breaking, or having centralized distribution pumps without power, not leaving adequate system pressure. An approach that many sites implement is installing deep water wells with enough flow to maintain site operations.
To summarize, here are some major talking points.
• Need to understand the boundaries of your design conditions. Most people do not know the minimums their data centers are designed to cope with.
• Understand the difference between allowable temperature for equipment operation and cold start. The differences are not always obvious. Need to work closely with manufacturers for all equipment to understand difference between the two.
• Understand the fuel impact during cold weather. It’s not always just freezing concerns, but low temperatures can lead to fuel quality degradation or other issues such as waxing or condensate build up in exterior tanks after warming up. Also, consider type of fuel used and it's temperature rating. Some are now considering the amount of fuel storage they have onsite because of fuel supply concerns
• Don’t forget your staff. Have provisions onsite to deal with potential extended stay conditions. Have letters to show essential employee status in case law enforcement is not allowing travel.
• Understand the risk of loss of water. Extended loss of water could cause significant problems for sites that are reliant on evaporative cooling, but have no onsite wells. Most sites are not prepared for an extended water outage.
Note: Links are included for other extreme cold content from Inside Track and DataCenter Knowledge.
The purpose of this impromptu roundtable was for attendees to provide their insight and experiences in regard to data center operations, during and preparing, for extreme cold conditions. As an example, are you doing anything extra with your engine-generators, fuel, mechanical equipment, natural gas and water supply, etc.? Do you know how extreme cold impacts the equipment design capacity of your data center? Do you have a proactive plan to address extreme temperatures, cold and hot, and what constitutes an effective plan? And don’t forget the onsite staff – what’s in place to support them during these conditions?
Roundtable comments:
The roundtable had over 20 attendees and over 15 member companies represented, all primarily interested in sharing experiences on temperature extremes and to compare notes on how to address these conditions. A number of attendees were from San Antonio and Dallas, Texas area, as well as traditional cold weather areas like the Minneapolis, MN area. One attendee commented that inclement weather planning helps data center operations a lot, but planning for a 100 year cold extreme is not normal. They are hoping to get a number of those “gotchas” from the session. Another attendee who deals primarily with data center risk indicated they are looking for what worked and what didn’t work in the way of mitigation.
Ryan Orr, Uptime Institute VP of Topology Services, kicked off the conversation by asking how many know the specific minimum temperature your data center is designed for. A couple of attendees appear to know the minimum design air temperature, but it appeared understanding the minimum design temperature and how this impacts capacity was not something commonly known. Ryan went on to explain how Uptime Institute uses ASHRAE design data for extreme temperatures, where looking at the N=20 value is the norm, but we encourage you to use something more if you think it is necessary. As an example, the ASHRAE N=20 low end temperature for Tulsa, OK, is -5 degrees F, but during this recent cold spell Tulsa got down to -19 degrees F. The key point is to first understand your boundaries and then, if there is a concern, put appropriate operations measures in place to address the concerns. An attendee commented they had to look beyond their present operational plans to address this rare extreme cold risk case.
An attendee from Minnesota where extreme cold is more common stated you need to plan and drill for these extremes. In MN, evaporative cooling is used quite a lot. During extremes, their technicians need to conduct more frequent rounds to look for things, like ice buildup, etc. Being able to run cooling tower fans in reversal is a key item as well.
The focus shifted to engine-generators (EGs). Here are some of the items discussed.
• Consider purchasing blended diesel fuel to eliminate wax accumulation in extreme cold. An attendee in MN commented they purchase extreme blended fuel and then run off the fuel throughout the season so they can replenish it after the winter.
• Examine the configuration of your fuel oil delivery system – inside vs. outside delivery. Most fuel oil deliveries are conducted outside, which could be an issue in extreme cold. Consider running fuel lines underground for cold temperature protection.
• Options can be purchased on EGs for cold running environments to help mitigate risk.
• In outdoor EG applications, engine coolant heat exchangers need to be examined.
• For outdoor diesel tanks, water build up in extreme cold can cause a freezing issue. Desiccant breathers can help, but onsite staff still need to be more concerned of this and do more frequent checking.
• Refueling for a number of attendees was an issue during the event. Fuel trucks were unable to get to the site. Therefore, they are now looking at more fuel storage for longer runtime, which comes with more concerns and exposures.
Ryan mentioned fuel is more sensitive than people think to cold temperatures. Therefore, it needs to be deeply looked at. It’s not just the potential for freezing the line that can get you, but also how it impacts fuel quality (for example, viscosity, gelling, etc.).
A number of attendees in the Texas area indicated they went on EGs because of the unstable power grid. The utility indicated they were implementing rolling blackouts, but in the beginning they ended up being mostly blackouts. However, a number of attendees indicated once utility went to rolling blackouts, they came off EGs because of greater concerns over the potential fuel supply.
The conversation shifted to other mechanical equipment. Ryan asked if anyone had a component that had cold starting issues, or was the approach to just run everything. One attendee indicated they had issues cold starting a chiller that sat for too long in the cold. Ryan touched on how we’ve seen issues in Russia where they modified their operations standard to factor in the cold. We’ve also seen these types of cold starting issues with any refrigerant based equipment. As sites utilize more automation and rotation of equipment, you may decide in extreme conditions to start all components and run them, or take out of auto mode and manually start and rotate equipment. One attendee commented if air cooled chillers are your basis, there are limits for extremely low ambient temperatures because of evaporator tubes freezing. Another attendee commented they run 15% glycol solution in Texas.
Next, staffing considerations were discussed. One attendee indicated they purchased cots, dry food, and have other accommodations on site, like a shower room, in case critical staff have to stay onsite for an extended period of time. Another commented how they planned ahead and adjusted their staffing schedule to make sure people could get to the site. A third attendee mentioned how they had an employee stay onsite 2-1/2 straight days. They discussed this with the employee beforehand about staying, so the site and the employee were prepared. An important point brought up by an attendee was that law enforcement have not allowed people on the roads during extreme conditions, so now they have letters that key employees keep in their vehicles stating they are essential employees.
Lastly, Ryan brought up how make-up water and supply water should also be a concern. The risk to a long term extended outage of water should be analyzed. As we saw in Texas during and after the extreme cold event, public water systems can be an issue with lines freezing, water mains breaking, or having centralized distribution pumps without power, not leaving adequate system pressure. An approach that many sites implement is installing deep water wells with enough flow to maintain site operations.
To summarize, here are some major talking points.
• Need to understand the boundaries of your design conditions. Most people do not know the minimums their data centers are designed to cope with.
• Understand the difference between allowable temperature for equipment operation and cold start. The differences are not always obvious. Need to work closely with manufacturers for all equipment to understand difference between the two.
• Understand the fuel impact during cold weather. It’s not always just freezing concerns, but low temperatures can lead to fuel quality degradation or other issues such as waxing or condensate build up in exterior tanks after warming up. Also, consider type of fuel used and it's temperature rating. Some are now considering the amount of fuel storage they have onsite because of fuel supply concerns
• Don’t forget your staff. Have provisions onsite to deal with potential extended stay conditions. Have letters to show essential employee status in case law enforcement is not allowing travel.
• Understand the risk of loss of water. Extended loss of water could cause significant problems for sites that are reliant on evaporative cooling, but have no onsite wells. Most sites are not prepared for an extended water outage.
Note: Links are included for other extreme cold content from Inside Track and DataCenter Knowledge.
Request an evaluation to view this report
Apply for a four-week evaluation of Uptime Intelligence; the leading source of research, insight and data-driven analysis focused on digital infrastructure.
Already have access? Log in here
Posting comments is not available for Network Guests