Data center tenants carry servers out of Fisher Plaza the morning after the fire.

Allrecipes.com had been working for the past year to build geographic redundancy into its online systems, planning to open a new co-location facility in New York by the end of August.

Last week, the wisdom of that plan became clear -- but not in a good way. Allrecipes was among dozens of sites shut down by the fire and subsequent power outage at Seattle's Fisher Plaza data center.

"For us, it was like a six-week-too-early bummer," said Lisa Sharples, president of the Seattle online recipe company, saying it would have migrated quickly to the New York servers had they been ready.

The incident late Thursday at Fisher Plaza East exposed a big gamble that many tech companies take by limiting themselves to a single location when setting up their online infrastructure. Allrecipes wasn't alone, with major sites such as Microsoft Bing Travel, Authorize.net and many others being taken offline as a result of the Fisher Plaza outage -- in some cases for more than 24 hours.

Data centers may seem infallible, and Fisher Plaza has been touted as "one of the most advanced communications hubs in the world." But failures happen all the time, and it's up to individual companies to have effective disaster recovery plans, said Jesse Robbins, who was previously in charge of availability for all of Amazon.com’s websites.

"You can expect an outage in a data center every couple of years even in the best-engineered facilities,” said Robbins, whose Amazon nickname was Master of Disaster. “The people that I hold responsible for outages are not the data centers but the people who built systems that rely on a single data centers or that depend on disaster recovery plans that they write once and never put into practice -- and end up getting caught with their pants down."

One problem is the expense inherent in establishing that geographic redundancy, said Ashutosh Tiwary, CEO of Seattle online back-up recovery startup Doyenz and an expert in the field of online backups.

"For mission critical applications, a common way to mitigate this risk is to build a geographically dispersed data center," Tiwary said via email. "However, many organizations particularly the small and mid-sized ones, don’t do this because of the large costs, low perceived risk, and the lack of perceived (return on investment)."

It's also not as simple as adding a second facility, said Troy Davis, CEO of Seven Scale, operator of Cloudvox, and a board member at the Seattle Internet Exchange.

"A Web service maintains info about past transactions with its users in the form of database rows, in-memory data structures, and files on disk," explained Davis, a former network engineer at Loudeye and RealNetworks, via email.

"The challenge with adding a second data center is how to keep both sites in sync, so that a failure is seamless to users," Davis added. "There are many strategies for keeping state in sync (and avoiding the need to), but it boils down to extra development time and implementation cost. It's not an easy thing to do, nor always worthwhile for the service."

But former Amazon "Master of Disaster" Robbins said the best approach is still a “multi-data center strategy where they don't have a dependency on any single data center and they can easily survive an outage affecting any data center without any customer impact.”

A combination of virtualization technology and cloud computing also provides a solution, letting web companies take advantage of networks of geographically dispersed facilities. Doyenz CEO Tiwary, whose company offers one such service, says it "provides many of the benefits of a redundant data center in the cloud without the cost of building and maintaining one."

The other big question following the incident was why Fisher Plaza used water to douse the electrical fire that triggered the incident, as opposed to an alternative fire-supression material. Fisher officials said last week that the water forced them to keep the onsite generators offline, leaving the building without power until backup generators could be brought in.

Robbins, who today is CEO of Seattle startup Opscode, said it’s common for data center facilities to have sprinkler systems. He said in the case of fire, data centers often rely first on an inert gas, flooding the affected room with a gas that displaces oxgen and disrupts combustion. But if that doesn’t work they go to sprinklers.

Water for fire protection is probably more common than many people would think, agreed Davis, the Seven Scale CEO. Inside data centers, choices about fire supression are often left to individual tenants, so long as they meet minimum requirements. But in this case, Fisher Communications said the fire happened in a common electrical room where Fisher Plaza East gets power from Seattle City Light.

"If that's the case, none of the tenants (Internap, Adhost, and others) could have prevented this: the fire suppression systems they chose within their datacenters never came into play," Davis said. "And once water is used, there is little that anyone can do until it has drained or been pumped and the fire department is satisfied."

Sharples

For Allrecipes.com, the Fisher Plaza outage hit at a particularly bad time. Many of Allrecipes' users were trying to log on to get 4th of July recipes during the outage. AllRecipes typically gets about 700,000 unique visitors each day, but on July 3rd, company president Sharples expected more than 1 million. Instead, the site was down for more than 24 hours.

It marked the longest interruption in service since the Web site was founded. Sharples said the outage also caused confusion among some of the company's advertisers who were not accustomed to seeing the site go down. As a result, Allrecipes is offering additional advertising impressions to its clients, including the possibility of running ads during upcoming holidays such as Labor Day.

Allrecipes was of the original tenants at Fisher Plaza, and Sharples said they've had a good working relationship with hosting provider Internap as well as building operators. They have no plans to move as a result of the fire.

However, she said, they are "very focused" on getting their New York co-location facility up and running.

Backup power at Fisher Plaza.

MONDAY UPDATE: This is the text of the status report issued Monday morning by Internap Network Services to its Fisher Plaza data center tenants:

"At this time, all cooling and power equipment servicing the SEF facility continues to operate normally from the interim power solution in place. There is an 8-hour refueling rotation to keep generators online, and providing power to the facility. The latest update from the Fisher Building staff and Internap Field Operations staff shows no indications of problems since the restoration of services.

"Customers with IP connectivity in either the SEF or SEF003 PNAPs continue to see their IP connectivity services up and stable; customers colocated to the SEF datacenter continue to see power and environmental conditions to their cage/cabinets up and stable.

"As previous notifications have outlined, the facility is currently being powered by mobile generators while the facility power infrastructure is being examined further to determine what the correct course of repair should be. Fisher Building staff do not anticipate any further interruptions in power at this time; and as previously noted, the Fisher engineering staff and various vendor representatives remain on-site to closely monitor the current situation. Internap Field Operations engineers will also be monitoring the situation closely, and are available to assist customers as needed. Please contact the Internap NOC to request Remote Hands services if necessary.

"Due to the current stability of power at the facility with the mobile generators, the NOC is going to supply updates every several hours unless there is a change in status, at which point notices will be sent immediately."