Recipe for Exchange Flambe

A flambe consists of a filling of some sort, something flammable, and an ignition source.

Ingredients

The gooey filling
2 Windows 2003 Servers running Exchange 2003 SP2, clustered
1 shared external hard drive array (RAID 5)

The flammable parts
0 IT staff who were involved in setting up current environment
0 documentation of current environment

The ignition source
1 recently hired Network Administrator who is on his way out after three months because he's a dumbass
1 new Network Administrator (to replace the above)
1 incompetent support person

Assembly Instructions

We're going to assemble our flambe in layers. First a bit of prep work

1. Old Network Administrator gives new Network Administrator access to the domain administrator account. On his first day.
   
2. Old Network Administrator leaves New Network Administrator to poke around Active Directory to learn the structure of things.
3. It is Old Network Administator's last day, so he is gone. Old Network Administrator didn't set up any of the current environment, but he had some knowledge of it.
   
4. New Network Administrator realizes most servers have some level of local security implemented, so he downloads and installs a software package on all the servers.

At this point we have an unstable environment, and nobody but the New Network Administrator has any idea that anything has changed. So now we need to start working on the gooey filling.

1. Something in the software breaks the driver for the card that connects the Exchange server to the external RAID array.
2. The RAID array starts flashing a light indicating a connectivity error.

There are a few directions to go from here:

• Call support
• Reboot the servers, then call support if it doesn't reconnect
  
• Power down the servers, reboot the array, reboot the servers, then call support if it doesn't reconnect

Sometimes the obvious choice is the worst. Time to add some fuel.

1. New Network Administrator calls HP Support.
2. Support identifies one drive has a significant number of errors.
3. New Network Administrator is instructed to upgrade the firmware on the drive.
4. After the firmware is upgraded the drive is offline.
5. It is pulled and reseated to allow the array to reinitialize it.
6. The RAID controller doesn't recognize the new firmware, and it is flagged as failed.

Connectivity to the RAID array is spotty, and the servers are failing back and forth every few seconds. The Quorum, Transaction and Data drives are on different arrays and end up getting mismatched. Let's add another layer of fuel.

1. New Network Administrator discovers the hardware failure in Device Manager and applies all outstanding Windows Updates.
   
2. Because the servers are failing back and forth at a rate of about every five seconds, the connectivity to the array is sporadic and this causes the drives to start thrashing.
3. With one drive already in a failed state, another one starts flashing a warning.

The only thing left to do is light it up!

1. New Network Administrator pulls the drive flashing a warning. While another drive is offline.
   

And there you have it: one very dead Exchange 2003 cluster! All it takes is a little bit of knowledge, a lot of ignorance, and an incompetent support person.

An HP support rep finally came on site, dug through some logs, and determined that the firmware that was installed was completely wrong for our environment. Also, there is a known problem with one particular Windows update that breaks the installer for the drivers, so new drivers can't be installed and the old ones can't be reinstalled.

He replaced the failed drive, let the array rebuild, replaced the drive that had the bad firmware, and by some miracle we were able to recover three of our five Exchange storage groups. Now the New Network Administrator and the PC Tech are piecing together backups and trying to reconstruct messages from logs on our SMTP relay.