Generator arrives at Fisher Plaza last week.

It was one week ago today that Fisher Plaza's data center facilities were knocked out by an electrical fire and power outage, taking down dozens of Web sites in the process. According to the latest reports from Internap Network Services, the facility is running smoothly on interim power, with two additional mobile generators now on site to provide extra redundancy. 

As of yesterday, officials were expecting new electrical switchgear to arrive as early as today, to replace equipment damaged in the incident. Earlier this week, Internap told tenants that the plan is to build a new switchgear room. Internap said Fisher Plaza officials believed it would be "several months" before a permanent solution is in place.

Read on for excerpts from Internap status reports issued this week.

Date: Tue, 7 Jul 2009

Please be advised that the landlord for Fisher Plaza has informed us that they are in the process of developing near term & final restoration plans to deal with the facility impact from last week's fire. The objective of the plans is to proceed as quickly as possible to provide a robust level of generator redundancy as well as a final solution to restore full commercial and generator power to everything at Fisher Plaza East.

Near Term: Proceeding immediately, the landlord will 1) obtain two additional generators and paralleling gear needed to provide redundancy across the 3 central risers affected by the outage. This will provide N+1 until we can return to full commercial power and allow required maintenance on the generators (required every 10 days) without any downtime.

Long Term Utility Restoration: The restoration plan involves building a completely new switchgear room outside the current vault and Switchgear Room. This would separate the Main 1 and Main 2 systems providing more reliability, new equipment, and the ability to begin construction while they are clearing out the damaged gear from the damaged Switchgear Room. All of the old Main 1 & 2 switchgear from the damaged switchgear room will be replaced with new equipment needed to provide a new Main 1 service and the paralleling gear to bring up all four of the building's 1.5 MW generators on line together.

The landlord stressed that this is a very fluid plan and that suppliers are aggressively sourcing equipment, design, build, and shipping times. Once they are able to develop a specific timeline as well as Method of Operation (MOP) they will share it with Internap, and we will of course share it with our customers. They are expecting several months until full restoration.

Date: Thu, 9 Jul 2009

The Internap NOC has received a status update from the Fisher Building regarding the progress being made towards the additional temporary mobile generator capacity/redundancy described in an earlier communication. The two additional generators arrived on site at Fisher 7/8/09 at 15:00 PDT. The east end of John Street from the garage exit to the corner of 5th avenue has been closes. On-Site Services has installed the fence around the generators and security is posted at this location 24/7. The switchgear is en-route from Los Angeles via relay drivers and is schedule to be delivered on Friday. At this point the time for Friday's delivery is undetermined, but when we receive this information we will pass it on. Sigma Six is on stand-by and will be ready to test the gear when it arrives. Cables have been delivered and Cummins and Prime are in the process of laying them out and preparing them for the new generators.

Date: Fri, 10 Jul 2009

At this time, all cooling and power equipment servicing the SEF facility continues to operate normally from the interim power solution in place. The latest update from the Fisher Building staff and Internap Field Operations staff shows no indications of problems since the restoration of services.

At this time, Internap can confirm that there are now 2 additional mobile generators on site at the Seattle Fisher facility. Currently Fisher personnel are working with their vendors to finalize the Method of Operation (MOP) to add these generators to provide n+1 redundancy.