Several years ago, when my company moved into our current building, some decisions were made at a level over my boss’ head. We have UPS power in our server rooms, but we don’t have standby generators for the building. At the time, it was deemed too expensive.

This morning, that decision came back to bite us. Very early this morning, our power went out. My boss called me a few minutes after 5:00, asking me to come in. We were running on battery backup and would need to shut things down in an orderly manner if the outage went too long.

I wouldn’t wish this experience on anyone, friend or foe. We did learn a few interesting things during this experience.

• Our server room has no emergency lighting.  Working in a dark server room is hell.
• The lock on our server room door isn’t connected to emergency power. My boss had to run upstairs to the vault to get the key.
• Air conditioning in our server room isn’t on emergency power. The server room got hot VERY quickly.
• Several of our key servers are our most power-hungry servers. They’re all on the same UPS and drained that battery pretty quickly.
• Having a malfunctioning KVM during a crisis is somewhere near Dante’s third circle of hell.
• When power did come back, several of our VMWare boxes didn’t automatically restart the virtual machines. That was quickly corrected.

All told, we came through this okay and it was a great learning experience. My databases fared well. I didn’t have to do anything to recover those.  Fortunately, we don’t have a factory floor that runs around the clock. Our biggest concern was users outside the US being able to still get to Exchange and Sharepoint. We did have one casualty among primary systems. One of our servers did overheat and may need to get a replacement processor.

When we found out why the power went out, my boss was none too happy. The power company had scheduled an outage without telling us nor the property management company. They were cutting over a building down the street. When we know these things are coming, we can handle them in a much more orderly fashion.