RECAP | ROUNDTABLE | Physical data center security and assessing risks

Data center operators have made considerable investments toward physical security. Infrastructure and access control measures, as well as staff procedures, are routine in most, if not all, mission critical facilities. However, the likelihood of sabotage has grown, the surface area for attacks has expanded, and the methods used by intruders are increasingly sophisticated. Rhonda Ascierto, VP of Research for Uptime Institute, joined this roundtable where attendees discussed physical security best practices, threats that are most concerning, and how best to assess site physical risks.

The session started by referencing the report Rhonda authored last year, Data Center security: Reassessing physical, human and digital risks (see the link below).

Rhonda then kicked off the session by saying data centers overall do a great job on physical security. She then provided an example of an individual who was targeting AWS data centers and how data centers are not as invisible as they were in the past.

An attendee started the comments by saying they do a formal risk assessment every year. However, the tool they internally use changes a lot. Most of the assessment is about physical security and risk. Bomb threats was an item mentioned by their team, so they have had to review their bomb threat reaction steps.

Another attendee commented physical security became paramount after 9/11. Data center siting became important, moving away from top tier cities, and there was clear evidence of physical hardening of data centers. As time went on, the real threat became more cyber, and the IT aspects of security. Now, over the last couple years, we are shifting to new threat vectors like physical attacking of data centers. Generally, focus has been on data center software systems (i.e., BMS) and what threats can come through. We need to reassess periodically based on changing conditions and environment what are the vulnerabilities.

Rhonda commented that cyber security and BMS, anything with an IP address, is vulnerable. She referenced talking with a hacker who could find DCIM, BMS, PLCs, and equipment from data centers very easily online. The attack surface has grown to include cyber and physical attacks of a site.

An attendee added we all have very robust IT security requirements. He commented what keeps him up at night is the human factor and personnel. He asked the question how often do you do background checks? Do you really know who the people are that walk into your facility? He indicated they do a robust background check twice a year because they learned from multiple incidents that checks should be frequent. Another attendee indicated they include the requirement for background checks in their service agreements. He also commented delivery people are the most vulnerable people you are letting get close to your data center. Also, day laborers being supplied from unknown sources is a concern.

Next, a comment was made that factory passwords supplied by vendors are a key risk, and that everyone should change those passwords immediately (ex., factory set Liebert unit passwords). Rhonda agreed and stated the #1 risk is password reuse - not changing them, and using the factory installed password.

An attendee commented that IT equipment you get rid of and do not clean up, and then recycle, is a big risk. There can still be a lot of sensitive information on this IT equipment if not thoroughly cleaned and wiped. Anything that leaves your data center needs to be sanitized to a Department of Defense (DoD) level. Rhonda added the most vulnerable equipment is legacy equipment due to lack of ownership and responsibility. This legacy equipment wasn’t designed with cyber in mind making legacy equipment the weakest link for cyber-attacks.

An attendee commented they partnered with their IT Security team and the team found vulnerabilities in their DCIM and BMS. Rhonda commented a best practice is penetration testing of these systems. The testing is usually conducted by a firm hired to periodically try to break in, both cyber and physical. One attendee stated penetration testing is normally done randomly, which is a concern. The concern is you conduct the test and it takes down the BMS, as an example. Testing needs to be coordinated with compliance and governance conducted via audits. Another attendee commented the flip side is you need to know what to do if you actually have a cyber-attack. They have developed a break-the-glass type of procedure of how to disconnect the production network for both IT and the facilities side, then they drill on it.

Rhonda then added the lack of security around adding biometrics is a personal concern. The more layers of security you have, the harder it is to intrude. She considers adding in layers of complexity to be a best practice.

An attendee commented their company is in the process of air-gapping facilities for security, essentially isolating their critical facilities from an IT and network perspective so that any issues at one location cannot impact any others.

To recap, here’s a quick list of some “best practices” mentioned during the session:
• Reassess and audit security periodically, usually annually. Be mindful that assessments need to change because the attack surface and conditions around you will change.
• Conduct frequent background checks of all vendors and personnel accessing site.
• Change factory supplied passwords on all equipment immediately.
• Legacy equipment is the weakest link for cyber-attacks so pay particular attention to this equipment.
• Make sure to sanitize and dispose of all equipment properly.
• Have IT Security look for vulnerabilities in your data center systems (i.e., DCIM, BMS, EPMS, etc.) and meet with IT Security on a periodic basis.
• Conduct penetration testing of key data center systems. Hire firms to periodically try to break in, both cyber and physical.
• Review site access frequently. It was mentioned monthly audits are conducted to comply with SOX.

Regarding auditing, here’s a quick list of prevailing industry standards for internal IT security auditing:
• SOX compliance, Sarbanes-Oxley Act for financial compliance
• PCI audit, payment card industry data security standard
• ISO27001, international standard for information security (ISO/IEC 27001:2013)

Rhonda then asked the group if anyone was aware of HR-type software being used that scans the social media aspects of a person’s life. She heard from a big cloud company about this and was curious if this is a method others are using to be aware of vendor and personnel potential risks. It was indicated financial companies have been known to check social media when checking and establishing credit, but there were no comments about this being used to check on vendors and personnel. However, it seemed the impression was they wouldn’t put it past their company.

Lastly and in closing, it was stated, “We don’t know what a terrorist looks like, essentially what’s impacting people’s lives.” Again, highlighting that the people we allow onsite and to access our systems are the primary security concern.