Heartbleed: Stopping The Bleeding With Expertise At Scale

When we heard about the “Heartbleed” vulnerability in OpenSSL last week, your specialists at Rackspace made it our mission to help our customers avoid heartache.

The Heartbleed vulnerability in OpenSSL, the encryption software found in most Linux systems, may allow an attacker to read chunks of memory from a remote system, meaning an attacker could access your servers remotely and pull back sensitive data including passwords, session tokens, or private keys. No trace of the attack remains on the system after the attacker has taken the data. The attacker can run the attack multiple times and gain access to different data from the server depending on what’s being stored in RAM.

We quickly patched our own infrastructure and immediately began proactively working with our customers to fix their systems based on our customer rules of engagement (some customers have opted to patch the systems themselves, or we do not have access to their servers as part of the services or at their request).

For our managed cloud and managed hosting customers we patched OS images, applications, websites – everything you run at Rackspace. We offered you a level of support and service that other cloud providers can’t touch. We were there to alert, guide and patch. We took a proactive role in doing the work. We had the test tools, we knew the distributions and versions affected, and we prepared the patches. And we did this for thousands of customers. We offered our expertise at scale to make sure our customers’ systems were no longer vulnerable to Heartbleed.

Let’s take a look at the numbers. In the days following the discovery of Heartbleed:

  • We opened 6,465 tickets for our automated processes in the US, and 1,609 in the UK
  • We logged into more than 14,000 servers and restarted at least one service on each
  • Roughly 30 percent of Managed Cloud servers had been patched before we ran the automated process (either by us or by the customer)
  • We notified more than 70,000 Email and Apps accounts
  • Our Managed Cloud customers opened more than 600 Heartbleed-related tickets
  • We handled more than 1,200 chats in which Heartbleed was mentioned (on average, a Racker handles about 300 chats per month)

Rackers throughout the company worked around-the-clock to ensure we helped each and every customer.

This level of hands-on service isn’t something all cloud companies do. Many left you on your own to make sure your systems were either free of the vulnerability or to patch them yourself. Not us. We staffed the phones, chats and tickets with our teams of experts for around-the-clock coverage to protect you from Heartbleed.

This is part of the Fanatical Support promise we offer every one of our customers. There is no such thing as an unsupported Rackspace customer. From our largest to our smallest, we offer support. We’re a true trusted partner. We don’t simply provide our customers the raw infrastructure and force them to go it alone. Everybody ought to have support.

And even for the customers we couldn’t patch due to access restrictions, we were happy to accept their phone calls, chats and emails and walk them through the process of patching their servers.

With other cloud providers, you pay top dollar for an additional layer of service and support before they’ll even take your call or even provide you a phone number. And without that, your best chance at support lies in online forums where you may or may not find the answers you need.

It’s our philosophy that during a crisis, contact with an actual human is exponentially more valuable than being pushed to a website. You can’t ask a website follow up questions. You can’t ask a website for clarification.

For our managed cloud and hosting customers, the updates were seamless. You didn’t have to figure out the patching process on your own. You could trust us to do what we do best – provide Fanatical Support from a team of specialists. While other cloud providers offer only infrastructure at scale, we were there when it mattered most to provide expertise at scale and make sure you weren’t impacted by Heartbleed.

If you have specific questions about Heartbleed, we encourage you to contact your Fanatical Support team. Updates and technical solutions are available in the Rackspace Community: https://community.rackspace.com/general/f/34/t/3596.

Rackspace also hosted a Google+ Hangout last week to discuss Heartbleed. Check out the recording for more details on how our specialists went about locating and updating servers that needed it.

John Engates joined Rackspace in August 2000, just a year after the company was founded, as Vice President of Operations, managing the datacenter operations and customer-service teams. Two years later, when Rackspace decided to add new services for larger enterprise customers, John created and helped develop the Intensive Hosting business unit. Most recently, John has played an active role in the evolution and evangelism of Rackspace's cloud-computing strategy and cloud products. John meets frequently with customers to hear about their needs and concerns, and to discuss Rackspace's vision for the future of cloud computing. John currently serves as Chief Technology Officer. John is also an internationally recognized cloud computing expert and a sought-after speaker at technology conferences, including CA World, the Goldman Sachs Techtonics Conference and Cloud Expo. He speaks on the future of cloud computing, enterprise cloud adoption, data center efficiency, green data center best practices, and more. Prior to joining Rackspace, John was a founder and General Manager at Internet Direct, one of the original Internet service providers in Texas. John is a graduate of the University of Texas at San Antonio and holds a B.B.A. in Accounting.


Please enter your comment!
Please enter your name here