OCS or Lync Front End not starting after patching

I have run into the issues described in this bug several times, and googling all the error messages always brings me to the Microsoft KB 968100, which has all the errors I was seeing in my OCS event log:

Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7024
Description:
The Office Communications Server Front-End service terminated with service-specific error 3287186749 (0xC3EE7D3D)

Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7022
Description:
The Office Communications Server Front-End service hung on starting.

Log Name:      Communications Server

Source:        OCS Server
Date:          7/12/2010 11:35:17 AM
Event ID:      12304
Task Category: (1000)
Level:         Error
Keywords:      Classic
User:          N/A

Computer:      ocsfe.contoso.com

Description:
The component Live Communications User Services (application: Default) reported a critical error: code C3EE7D3D (SQL server cross database chaining option is disabled for either rtc or rtcdyn database. This may be caused by attaching or restoring an existing database. Since this condition causes performance issues, service will be stopped and will not start until the problem is resolved.

Cause: Check the eventlog description.

Resolution:
Please enable cross database chaining option (using "sp_dboption '', 'db chaining', TRUE") for both rtc and rtcdyn databases and restart the service.). The service has to stop.

Additionally, the following event is logged repeatedly in the OCS log:

Event Type: Error
Event Source: OCS MCU Infrastructure
Event Category: (1022)
Event ID: 61013
Description:
The process DataMCUSvc(1692) failed to send health notifications to the MCU factory at https://ocststpool1.ned.ifo.com:444/LiveServer/MCUFactory/, 20776, 11/8/2007 3:12:41 PM.

And the following errors are logged in the OCS log at about the time that the service started encountering problems:

Event Type: Error
Event Source: OCS Server
Event Category: (1000)
Event ID: 12330
Description:
Process: 'RtcAggregate.exe' Exit Code: C3E83201
(SIPPROXY_E_SHUTDOWN_REQUESTED).
Cause: This could happen due to low resource conditions or insufficient privileges.
Resolution:
Try restarting the server. If the problem persists contact Product Support Services.

Event Type: Error
Event Source: OCS User Services
Event Category: (1006)
Event ID: 30915
Description:
Initialize failure. Failed to initialize the Service component.
Error code: C3E83201
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
After investigating the resolution in the article, (which is incorrect and doesn't have the "EXEC" in front of the command they give you) I found my databases already had the chaining option enabled. 

I did more searching, and the fix is so simple, it really should be in the KB.  You likely just patched, and you utilized the handy ServerUpdateInstaller.exe that Microsoft provides...

For example, for OCS 2007 R2 CU11, the download page here:
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=19178

The patches all worked, and you thought you were done...   until the Front End service hung on starting and dropped all the above errors in your event log.

You need to also download the OCS2009-DBUpgrade.msi and use that to update your RTC and RTCDyn Database schema to match the patches you installed.

This is done by running:

OCS2009-DBUpgrade.msi POOLNAME=pool01

Keep in mind, the poolname here is not needed to be FQDN, just the netBIOS name of the pool is needed.

Hopefully I put enough of the errors and key words in here that more people find this out with less downtime.

Using PowerShell to Notify of NTBackup failures

I recently had a need to send email notifications on a Windows 2003 server when a backup through NTBackup failed. Preferring PowerShell to VBscript, I installed PowerShell on the server, and set out to script the notification.

I decided to send a notification based on whether or not the NTBackup application has logged an error in the application event log in the past 24 hours. I chose this way because in Windows 2008 and later, there is a built-in scheduled task that can be customized to send an email based on an event ID, and I am already using this on some other servers. So it made sense to keep the same method of notification in Windows 2003.

I could have also chosen to parse through the ntbackup log and look for errors, or check for the existence of a new backup file created within the last 24 hours.

I didn't find a PowerShell version of NTBackup notification anywhere else online, and many of the VBscript versions require using blat. One advantage of the PowerShell method is that you don't have to install anything third party on the server. All you need is PowerShell.

Please note this script works only with PowerShell 2.0, and only in versions of Windows 2003 and 2003 R2. In this version of the script, I am storing the SMTP account and password in plaintext (of course I didn't put a real account in the script below!), so if this is a problem in your network, I would suggest using get-credential to pass the credential in a secure manner instead.

'Set the computer name
$computer = gc env:computername
'Set Email parameters
$EmailFrom = "backup_service@chrislehr.com"
$EmailTo = "robin.lehr@chrislehr.com"
$Subject = "Backup Failed on " + $computer
$SmtpServer = "mail.chrislehr.com"
$smtp = new-object Net.Mail.SmtpClient($smtpServer)
$smtp.Credentials = New-Object System.Net.NetworkCredential("backup_service","P@ssw0rd")

'Find all NTBackup events in the event log within the last 24 hours with type error. If the count is greater than 0, send an email.
$events = get-eventlog -computer $computer -LogName application -after (get-date).addhours(-24) -EntryType error| where-object {$_.source -eq "ntbackup"}
$body = "A Backup job has failed on " + $computer + "in the past 24 hours. Please investigate"
If (($Events).count -gt 0) {$smtp.Send($emailFrom, $EmailTo, $Subject, $body)}

Virtual Directory Errors

I recently installed SP1 on an Exchange server for a client, everything appeared to go fine however after rebooting we ran into various issues with OWA, Autodiscover, ActiveSync, PowerShell and EWS. Also we were unable to launch EMC or Powershell. We were receiving the following error about the various virtual directories. We ran through all the troubleshooting that the Exchange team recommends ( http://blogs.technet.com/b/exchange/archive/2010/02/04/3409289.aspx ) however nothing resolved it. The error we were getting was:

Log Name:      Application
Source:        ASP.NET 2.0.50727.0
Date:          2/10/2012 2:05:46 PM
Event ID:      1310
Task Category: Web Event
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      computer.corp
Description:
Event code: 3008
Event message: A configuration error has occurred.
Event time: 2/10/2012 2:05:46 PM
Event time (UTC): 2/10/2012 9:05:46 PM
Event ID: 1dbea0f1695f48648c8988fb70acec0a
Event sequence: 1
Event occurrence: 1
Event detail code: 0


Application information:
    Application domain: /LM/W3SVC/1/ROOT/Autodiscover-10-129733815458960203
    Trust level: Full
    Application Virtual Path: /Autodiscover
    Application Path: D:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Autodiscover\
    Machine name: ServerName

Process information:
    Process ID: 4600
    Process name: w3wp.exe
    Account name: NT AUTHORITY\SYSTEM

Exception information:
    Exception type: HttpException
    Exception message: Could not load file or assembly 'Microsoft.Exchange.Security, Version=14.0.0.0, Culture=neutral, PublicKeyToken=xxxxxxxxxxxxxxx' or one of        its dependencies. The system cannot find the file specified. (D:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Autodiscover\web.config line 855)

Request information:
    Request URL: https://computer.corp:443/Autodiscover/Autodiscover.xml
    Request path: /Autodiscover/Autodiscover.xml
    User host address: XXX.XXX.XXX.XXX
    User:
    Is authenticated: False
    Authentication Type:
    Thread account name: NT AUTHORITY\SYSTEM


Thread information:
    Thread ID: 1
    Thread account name: NT AUTHORITY\SYSTEM
    Is impersonating: False
    Stack trace:    at System.Web.Compilation.BuildManager.ReportTopLevelCompilationException()
   at System.Web.Compilation.BuildManager.EnsureTopLevelFilesCompiled()
   at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters)

After calling PSS we determined that it actually was something got botched with the SP1 install and the web.config files were not correctly pointed to the install location. So for us what was in the web config was:

 <codeBase version="14.0.0.0" href="file:///%ExchangeInstallDir%bin\Microsoft.Exchange.Sqm.dll" />

however for our install it should be:

<codeBase version="14.0.0.0" href="C:\Program Files\Microsoft\Exchange\V14\bin\Microsoft.Exchange.Sqm.dll" />

Difference in Policy Versions

I recently was working on a cross forest migration from Exchange 2010 SP1 to an environment that had Exchange 2010 RTM. Microsoft claims that it is supported however we ran into an issue with ActiveSync devices. I could not find any relevant posts or articles with this because it is always best practice to make sure that your Exchange servers are on the same version. We received the following error after migrating and attempting to configure ActiveSync:

Log Name:      Application
Source:        MSExchange ActiveSync
Date:          2/9/2012 1:39:43 PM
Event ID:      1008
Task Category: Requests
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      company.corp
Description:
An exception occurred and was handled by Exchange ActiveSync. This may have been caused by an outdated or corrupted Exchange ActiveSync device partnership. This can occur if a user tries to modify the same item from multiple computers. If this is the case, Exchange ActiveSync will re-create the partnership with the device. Items will be updated at the next synchronization.
URL=/Microsoft-Server-ActiveSync/Proxy?User=Username&DeviceId=Applxxxxxxxxx4T&DeviceType=iPhone&Cmd=Settings
--- Exception start ---
Exception type: Microsoft.Exchange.Data.Storage.InvalidSyncStateVersionException
Exception message: Sync state 'Policy' loaded from the user's mailbox has a newer version '37' than the Client Access server version '35'.  Please update the Client Access server to the latest version.
Exception level: 0
Exception stack trace:    at Microsoft.Exchange.Data.Storage.SyncState.Load(Boolean reloadFromBackend, PropertyDefinition[] additionalPropsToLoad)
   at Microsoft.Exchange.Data.Storage.CustomSyncState.GetSyncState(SyncStateStorage syncStateStorage, Folder syncStateParentFolder, SyncStateInfo syncStateInfo, StoreObjectId storeObjectId)
   at Microsoft.Exchange.Data.Storage.SyncStateStorage.GetCustomSyncState(SyncStateInfo syncStateInfo)
   at Microsoft.Exchange.AirSync.GlobalInfo.LoadFromMailbox(MailboxSession mailboxSession, SyncStateStorage syncStateStorage, ProtocolLogger protocolLogger)
   at Microsoft.Exchange.AirSync.Command.OpenSyncStorage(String deviceType, String deviceID, Boolean shouldOpenGlobalSyncState)
   at Microsoft.Exchange.AirSync.Command.WorkerThread()
--- Exception end ---.

We attempted to try and retrieve a list of any partnerships however the results were null. After a call to PSS we were told that we would have to upgrade the CAS to SP1 as there was no way to remove the legacy partnership information without exporting the mail to PST and re-creating mailboxes. 

Lync for Windows Mobile is now available!

Some pretty awesome mobility news today.  The Windows Mobile Lync app is now available in the marketplace!  Now we just need the Android and iOS versions!

http://www.wpcentral.com/lync-mobile-windows-phone-now-live-marketplace

Of course, there is some administrative changes needed to implement Lync mobile devices.
http://www.agileit.com/Blog/Lists/Posts/Post.aspx?ID=898

Finding all file shares on the network

I adapted this script from Michel Steveman's post on printer inventory.

#Purpose: get all file shares and output to Excel

#Create array of servers to retrieve information from by reading masterservers.txt. Could also plug in an AD query here.
$servers= get-content masterservers.txt

#create new Excel WorkBook
$Excel = new-Object -comobject Excel.Application
$Excel.visible = $True
$Excel = $Excel.Workbooks.Add()
$Sheet = $Excel.Worksheets.Item(1)
$Sheet.Cells.Item(1,1) = "Server Name"
$Sheet.Cells.Item(1,2) = "Share Name"
$Sheet.Cells.Item(1,3) = "Share Path"
$intRow = 2
$WorkBook = $Sheet.UsedRange
$WorkBook.Font.Bold = $True

#Get File Share Information and write to Excel
ForEach ($server in $servers)
{
$Shares = Get-WmiObject -Class win32_share -ComputerName $server | where-object {$_.type -eq 0}
foreach ($Share in $Shares)
{
$Sheet.Cells.Item($intRow, 1) = $Server
$Sheet.Cells.Item($intRow, 2) = $Share.Name
$Sheet.Cells.Item($intRow, 3) = $Share.Path
$intRow = $intRow + 1
}
}

$WorkBook.EntireColumn.AutoFit()
$intRow = $intRow + 1
$Sheet.Cells.Item($intRow,1).Font.Bold = $True