VMWare ESX, virtualized DNS and an ISCI SAN

Since the fire we had last year we have replaced all our old servers with a new virtualized infrastructure. We're running VMware ESX 3.5, an HP BLc-3000 blade chassis with six blades, and an HP AiO1200R ISCSI SAN. It is working great and I have a writeup about that decision-making process that I will be publishing shortly.

Today I wanted to bring up one of the potential pitfalls when you're creating a fully virtualized environment. This past weekend we had to cut building power for an extended period of time, so the network administrator brought down everything in our server room. As he brought everything back online he realized that Virtual Center, the control console for VMware ESX, could not talk to the SAN because it required DNS resolution.

The Problem

Our DNS servers are virtualized with storage on the SAN. He ran into a chicken-and-egg situation where he had dependent services that relied on each other.

It took him a while to realize that DNS was the issue. The logs on the SAN side simply said "Could not connect ISCSI LUN". On the VMware side the virtual machines said "storage not available". Figuring out why the two were unable to connect took some careful analysis. Solving it proved difficult because our departmental wiki also used SAN storage, so he had no access to our documentation. In a flash he found himself back in the same situation he was in after the fire, when he could not access critical documentation because the servers with it were not available.

The Solution

So how did he solve it? Luckily he still had the old primary domain controller hanging out, which had all the DNS information. He was extremely lucky, and he knows it. To keep from having to rely on luck, how should you configure your VMware environment so this doesn't happen to you? There are a couple of ways to tackle it.

Use local storage for your virtualized name servers.

Pros

• Name servers will load without SAN access.
• Resilient to SAN outages.

Cons

• Cannot mix guest VM's that require SAN storage. The ISCSI initiator in VMware ESX loads when ESX boots. By having your DNS server on the same physical host as another VM that requires SAN storage, the guest on SAN storage will not be able to start.

Use a non-virtualized DNS server.

Pros

• Resilient to SAN outages.

Cons

• If using a Windows server, also requires you run Active Directory services.

Use hosts files.

Pros

• Resilient to SAN outages.
• May improve performance slightly since lookups will always be from local cache.

Cons

• Requires you add hosts files to the Virtual Center server, SAN server, and every ESX host server.
• Can be a maintenance burden if your environment changes frequently and you have to constantly add/remove ESX hosts.

We have opted for the last option. Our VMware host environment is fairly static, so maintaining hosts files will be a minimal maintenance issue. The resilience we gain from it make it very worthwhile. Oh, and we printed a copy of our wiki page that has all the hostnames and IP addresses of every server we have, and put it in the safe. :-) You do have a similar list, and a fireproof safe... right?

A script to check remote computers for directories

At work we needed a way to check servers to see if certain software had been installed. The easiest way was to check for the software's installation directory. There isn't an easy way to do this remotely, though, so I wrote a script to take care of it: dircheck.vbs.

While I was writing this tool I learned a lot about VBScript. For starters, you can't interact with stdin or stdout using the default VBS command interpreter. If you try to write information to the user's console it will display everything in a popup. To fix this, you can use the special cscript interpreter:

cscript dircheck.vbs

If you execute the above command you will get command line help for the utility. Full source code is obviously included, so please feel free to use it however you need to.

How to copy SQL Server DBMail configuration to another server

I'm setting up a new SQL Server from scratch and wanted to copy the existing DBMail configuration from the old server. I did some searches and the best I could find were pointers to the msdb.dbo.sysmail_* system tables. I did some trial and error and got everything copied over, so here's how I did it.

1. Log into the new server
   
2. Create a server link from the new server to the old server
3. Copy the DBMail configuration
   

I had to log into the new server and do the server link there. From my workstation SQL Server considered it a redirection, and that is a security violation. Save yourself some headaches and just start at the new server. Note that the following SQL script will delete any existing DBMail configuration in the target SQL Server. If you want to keep the existing configuration you'll need to take out the DELETE and SET IDENTITY_INSERT statements and manipulate the account_id and profile_id in the related tables.

SET IDENTITY_INSERT sysmail_account ON
INSERT INTO sysmail_account (account_id, [name], [description], email_address, display_name, replyto_address, last_mod_datetime, last_mod_user)
SELECT * FROM oldserver.msdb.dbo.sysmail_account
SET IDENTITY_INSERT sysmail_account OFF
GO

DELETE sysmail_configuration
GO
INSERT INTO sysmail_configuration (paramname, paramvalue, [description], last_mod_datetime, last_mod_user)
SELECT * FROM oldserver.msdb.dbo.sysmail_configuration
GO

DELETE FROM sysmail_profile
GO
SET IDENTITY_INSERT sysmail_profile ON
INSERT INTO sysmail_profile (profile_id, [name], [description], last_mod_datetime, last_mod_user)
SELECT * FROM oldserver.msdb.dbo.sysmail_profile
SET IDENTITY_INSERT sysmail_profile OFF
GO

DELETE FROM sysmail_principalprofile
GO
INSERT INTO sysmail_principalprofile (profile_id, principal_sid, is_default, last_mod_datetime, last_mod_user)
SELECT * FROM oldserver.msdb.dbo.sysmail_principalprofile
GO

DELETE FROM sysmail_profileaccount
GO
INSERT INTO sysmail_profileaccount (profile_id, account_id, sequence_number, last_mod_datetime, last_mod_user)
SELECT * FROM oldserver.msdb.dbo.sysmail_profileaccount
GO

DELETE FROM sysmail_servertype
GO
INSERT INTO sysmail_servertype (servertype, is_incoming, is_outgoing, last_mod_datetime, last_mod_user)
SELECT * FROM oldserver.msdb.dbo.sysmail_servertype
GO

DELETE FROM sysmail_server
GO
INSERT INTO sysmail_server (account_id, servertype, servername, port, username, credential_id, use_default_credentials,
enable_ssl, flags, last_mod_datetime, last_mod_user)
SELECT * FROM oldserver.msdb.dbo.sysmail_server
GO