Tuesday, March 27, 2012

OCS or Lync Front End not starting after patching

I have run into the issues described in this bug several times, and googling all the error messages always brings me to the Microsoft KB 968100, which has all the errors I was seeing in my OCS event log:

Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7024
Description:
The Office Communications Server Front-End service terminated with service-specific error 3287186749 (0xC3EE7D3D)
Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7022
Description:
The Office Communications Server Front-End service hung on starting.

Log Name:      Communications Server

Source:        OCS Server
Date:          7/12/2010 11:35:17 AM
Event ID:      12304
Task Category: (1000)
Level:         Error
Keywords:      Classic
User:          N/A

Computer:      ocsfe.contoso.com

Description:
The component Live Communications User Services (application: Default) reported a critical error: code C3EE7D3D (SQL server cross database chaining option is disabled for either rtc or rtcdyn database. This may be caused by attaching or restoring an existing database. Since this condition causes performance issues, service will be stopped and will not start until the problem is resolved.

Cause: Check the eventlog description.

Resolution:
Please enable cross database chaining option (using "sp_dboption '', 'db chaining', TRUE") for both rtc and rtcdyn databases and restart the service.). The service has to stop.
Additionally, the following event is logged repeatedly in the OCS log:

Event Type: Error
Event Source: OCS MCU Infrastructure
Event Category: (1022)
Event ID: 61013
Description:
The process DataMCUSvc(1692) failed to send health notifications to the MCU factory at https://ocststpool1.ned.ifo.com:444/LiveServer/MCUFactory/, 20776, 11/8/2007 3:12:41 PM.
And the following errors are logged in the OCS log at about the time that the service started encountering problems:

Event Type: Error
Event Source: OCS Server
Event Category: (1000)
Event ID: 12330
Description:
Process: 'RtcAggregate.exe' Exit Code: C3E83201
(SIPPROXY_E_SHUTDOWN_REQUESTED).
Cause: This could happen due to low resource conditions or insufficient privileges.
Resolution:
Try restarting the server. If the problem persists contact Product Support Services.
Event Type: Error
Event Source: OCS User Services
Event Category: (1006)
Event ID: 30915
Description:
Initialize failure. Failed to initialize the Service component.
Error code: C3E83201
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
After investigating the resolution in the article, (which is incorrect and doesn't have the "EXEC" in front of the command they give you) I found my databases already had the chaining option enabled. 

I did more searching, and the fix is so simple, it really should be in the KB.  You likely just patched, and you utilized the handy ServerUpdateInstaller.exe that Microsoft provides...

For example, for OCS 2007 R2 CU11, the download page here:
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=19178

The patches all worked, and you thought you were done...   until the Front End service hung on starting and dropped all the above errors in your event log.

You need to also download the OCS2009-DBUpgrade.msi and use that to update your RTC and RTCDyn Database schema to match the patches you installed.

This is done by running:
OCS2009-DBUpgrade.msi POOLNAME=pool01

Keep in mind, the poolname here is not needed to be FQDN, just the netBIOS name of the pool is needed.

Hopefully I put enough of the errors and key words in here that more people find this out with less downtime.