Amazon: Cloud outage made worse by generator, software glitches |
Connect with TechFlash on our Facebook page for all the latest technology news headlines and commentary, plus information and access to special events, photos from events, promotions and more.
Amazon’s major power outage in Ireland on August 7 was made worse by the failure of backup generators and a software error.
Amazon says the problem, which affected a data center that hosts both the Amazon’s Elastic Compute Cloud (EC2) and Microsoft’s Business Productivity Online Services (BPOS), started when a 110kV 10 megawatt transformer failed. Normally, “when utility power fails, electrical load is seamlessly picked up by backup generators.” But in this case, one of the Programmable Logic Controllers did not “complete the connection of a portion of the generators to bring them online,” according to Amazon.
And to make matters worse Amazon said it discovered a software error that affected the Amazon EBS, or Elastic Book Store, which provides block level storage for Amazon Cloud customers. The error, which has since been corrected, made it more difficult to recover data.
In a company blog post summarizing what went wrong Amazon, apologized to its customers and said it was going to give out 10-day credits. Customers impacted by the EBS software bug that accidentally deleted blocks in their snapshots will receive a 30 day credit for 100 percent of their EBS usage, Amazon said.
Meanwhile, Amazon says it is reassessing what caused the power failure:
The initial fault diagnosis from our utility provider indicated that a lightning strike caused the transformer to fail. The utility provider now believes it was not a lightning strike, and is continuing to investigate root cause.
Amazon also issued an apology to its customers:
Last, but certainly not least, we want to apologize. We know how critical our services are to our customers’ businesses. We will do everything we can to learn from this event and use it to drive improvement across our services. As with any significant operational issue, we will spend many hours over the coming days and weeks improving our understanding of the details of the various parts of this event and determining how to make changes to improve our services and processes.
If you are commenting using a Facebook account, your profile information may be displayed with your comment depending on your privacy settings. By leaving the 'Post to Facebook' box selected, your comment will be published to your Facebook profile in addition to the space below.