Windows server times are very, very important

We have a custom VB.Net app that connects to a customer's web service so our users can exchange information with the customer. I spent most of today struggling to get an Active Directory Certificate Services code signing certificate working, and once I had that working I turned my attention to this one.

The Sr. Network Admin had been working on it for a while, and users were becoming increasingly panicked. They hadn't been able to connect for over a week and the customer was getting impatient. We understood that some apps wouldn't work until our infrastructure was back up, but we finished that on Saturday. Everything should work.

When users tried to connect they got the following error

System.Net.WebException: The remote server returned an error: (407) Proxy Authentication Required. ---> System.ComponentModel.Win32Exception: The clocks on the client and server machines are skewed

I checked the client PC and its clock was fine, so this meant a server clock is off. Not surprising considering everything in our computer room has been through a fire, cleaned in a solution of some kind, dried, reassembled, and stuffed in a rack. But which one? We have a total of four possible proxies involved. I installed the application on my computer, generated the error, then checked my Event Viewer. Looky what I found

Event Type: Error
Event Source: Kerberos
Event Category: None
Event ID: 5
Date: 7/22/2008
Time: 3:18:45 PM
User: N/A
Computer: CROBINSON
Description:
The kerberos client received a KRB_AP_ERR_TKT_NYV error from the server host/isa-vpn.domain.com. This indicates that the ticket used against that server is not yet valid (in relationship to that server time). Contact your system administrator to make sure the client and server times are in sync, and that the KDC in realm DOMAIN.COM is in sync with the KDC in the client realm.

I logged into isa-vpn and lo and behold its date was June 13, 2001 and its time was 10:21 PM. I fixed this and the application started working. As an added bonus users were able to log into the VPN, which the Sr. Network Admin had also been working on for the last three days.

This highlighted three things. First, our servers were not set up to use a central time server. Second, nobody checked the server times after they were brought back online. And finally, the CMOS battery in the server is dead.

Something else to add to our DR and maintenance plans.