major.io words of wisdom from a systems engineer

Rackspace Outage

I’ve received a lot of IM’s and e-mails from friends and readers of this blog about the Rackspace outages on November 11th and 12th. I work for a company that believes in full disclosure, so if you want the facts, they’re already available to the public:

Rackspace Information Center

TechCrunch - Quick, Plug The Internet Back In: Major Rackspace Outage

Laughing Squid - Massive Power Outage At Rackspace’s Dallas Data Center

37Signals - Downtime Explanation

ValleyWag - Truck driver in Texas kills all the websites you really use

I don’t know of any additional information besides what is contained within these articles and blog posts. However, I can tell you that I’ve never worked for a company before that pulled together in such large numbers to get on the phones and respond to tickets long after shifts should have been over. Obviously, it was a horrible “perfect storm” type of situation, and no one would wish for it to happen to anyone.

Over the last five years, I’ve had dedicated servers and VPS accounts with seven companies. Out of those seven, five have had major outages. After those outages, I can honestly say I received a timely and courteous response from one of the companies in that list. In one situation with a certain Texas hosting company, I had no network connectivity for almost 72 hours with no response to phone calls or trouble tickets.

After it’s all said and done (and it’s not done yet), I find myself to be very proud of the company for which I work. Server parts will eventually fail, as will networks, generators and power grids - it’s inevitable. The important part is that the will of those who are providing the support never fails.

Our will is strong, and it continues to stay that way.