Preparing Our Data Centers For Hurricane Sandy

Hurricane Sandy is poised to pound the mid-Atlantic and Northeast today through Wednesday, with intense winds and torrential rains that could cause widespread power outages and floods, including in areas where Rackspace operates data centers. The teams at our Virginia facilities, and also at our headquarters in San Antonio, have been working round the clock to ensure that our customers’ data remains safe and that they continue to receive the Fanatical Support that they’ve come to expect, even during what many forecasters predict will be a “perfect storm.”

We realize that there are always things that can go wrong (in good weather and bad). But we’re working hard behind the scenes – and drawing on our 14 years of experience — to take preventative action to avoid service disruptions. While we don’t expect any issues, when things do go wrong, we strive to be as responsive and transparent as possible.

Currently, our two Northern Virginia data centers – IAD1 in Herndon and IAD2 in Ashburn – are the most likely to feel the effects of Hurricane Sandy. Northern Virginia is inside the projected forecast cone for the storm, but is not expected to get hit directly.

As of Monday, the National Weather Service said Northern Virginia stands a 90 percent to 100 percent chance of experiencing tropical force surface winds, which are between 39 mph and 73 mph. Northern Virginia also has a 10 percent to 20 percent chance of hurricane force winds. Because of this, a “High Wind Warning” is in effect now through 8 p.m. EDT tomorrow with winds predicted to increase significantly from 1 p.m. today through Tuesday morning. Sustained winds could reach 35 mph to 45 mph, while gusts could hit between 60 mph and 70 mph.

Hurricane Sandy is also expected to produce between five and 10 inches of rain, with up to 12 inches in localized areas. IAD1 and IAD2 are built on land rated by the Federal Emergency Management Agency (FEMA) as being above the 500-year flood marks for the area. Both are built on high ground and not in close proximity to areas that may experience water rise due to a storm surge or flash flooding.

Power outages, too, remain a concern in the mid-Atlantic and northeast. Currently, the generator fuel tanks in both IAD1 and IAD2 are full and the generators can run for roughly 60 hours – that’s 2 ½ days — without refueling. If needed, we can refuel the generators while they are running. Our emergency fuel provider is on notice that we may require additional fuel, which will be delivered within 24 hours of a service call. Additionally, our facility partner stands at the ready and maintains its own emergency fuel delivery contracts. We have also contacted our generator maintenance and repair vendor, which will place a diesel engine specialist on site at IAD1 if required.

We have also ensured proper staffing throughout the event and will continue to offer round the clock customer support services. For our on-site staff we have prepared emergency overnight kits that include inflatable mattresses, blankets, towels and a hygiene kit. The data centers are stocked with food and drink, including military Meals Ready to Eat (MREs) and plenty of bottled water. We have ensured that critical staff is on-site and that the rest of our data center staff is safe and accounted for before, during and after the storm.

We have coordinated ticket queue coverage with the other global data center teams and IAD will remain in constant contact with those teams throughout Hurricane Sandy. We have also blocked and closed the IAD1 and IAD2 scheduled maintenance calendars for the duration of the storm, while all maintenance scheduled before the calendars were blocked will be completed as scheduled.

This isn’t our first brush with a weather event of this magnitude. In August 2011, Hurricane Irene moved through Northern Virginia and caused widespread damage throughout the mid-Atlantic and Northeast. During Irene, neither our Herndon nor Ashburn facilities required transfer to generator power as the utility grid remained unaffected. Also, in late June 2012, a derecho, also known as a “land hurricane,” moved through Northern Virginia causing damage and power outages, some of which lasted up to three weeks. Neither facility required generator power.

Preparations will be ongoing as the storm moves closer. We feel that these preparations and our track record for withstanding major storms will offer our customers continual service without disruption as we brace for Hurricane Sandy.

To our customers and everyone else in Hurricane Sandy’s path, please stay safe.

Jacques Greyling joined Rackspace Hosting as a Linux tech in the United Kingdom over a decade ago. He worked his way up through the company and has held many successful positions including Linux Team Lead, Head of Customer Care, Director of Operations, UK and Managing Director, UK. In 2007, after experiencing several data center problems, Rackspace knew it needed someone with the discipline to make tough decisions about data center operations and management. They also needed someone who deeply understood and cared about the company and preserving its unique culture. Consequently, in 2009, Greyling moved from the UK to Dallas, Texas where he spent the next seven months. He sometimes stayed at the office for 72-hour shifts that included everything from doing incident maintenance to giving tours. In April of 2010, Rackspace’s COO, Mark Roenigk, asked Jacques to lead the company’s Global Data Center Infrastructure team. From an entry-level tech position to a data center executive leader, Jacques has truly climbed a ladder of hard work and dedication at Rackspace. In his leadership role, he finds the greatest reward in “putting people you would never expect in certain positions and helping them succeed. Taking people that you see have potential and putting them where they can be challenged.” Prior to Rackspace, Greyling worked at Nissan and Datacentrix respectively. He attended the University of Pretoria in South Africa, where he earned his degree in Engineering. In his free time, he enjoys gardening, home improvement projects and spending time with his family. Jacques Greyling resides in San Antonio with his wife of 12 years, Annalize and his four-year old son, Xander.



Please enter your comment!
Please enter your name here