Wednesday, February 25, 2009

Restoring Exchange using Microsoft DPM 2007 SP1

This article will walk you step-by-step through performing a restore on Microsoft Exchange 2007 using Data Protection Manager 2007 SP1.

To recover Exchange data, you must first have an active recovery storage group mounted. To do this, open the Exchange Management Console (EMC), go to tools, and then click "Troubleshooting Assistant." You must first enter the server and user information and click next.


Next, you must select a task. Choose "Create a recovery storage group."


Now you must select the recovery storage group that contains the mailbox you want to recover. For this example, we will choose the executive storage group. Click next.

The next screen asks you to name the recovery storage group and verify the location of the exchange data and logs. When done, click "Create Recovery Storage Group."


The last screen will show you whether or not the creation was successful:


Next, recover the database to the recovery storage group. Log onto your DPM server, and go to the Recovery tab of the DPM 2007 console. Expand the Exchange server, and click on the storage group from which you want to restore the mailbox database:


Right-click on the database in the right-hand side of the screen and click "Recover…" Click Next on the review screen.

Select "Recover to Recovery Storage Group."


Specify the Exchange server, the Storage Group Name (Recovery Storage Group), and the database (Executive Database):


Select to mount the databases after they are recovered:


Click Next, review your recovery settings, and then click "Recover."

Now to recover a single mailbox, you will need to open the Exchange Management Shell on your exchange server. The following commands are examples of different mailbox recovery scenarios:

  • To recover a single mailbox from the RSG database to the active mailbox 'username', run the following command in the Exchange Management Shell:

Restore-Mailbox -identity 'UserDisplayName' -RSGDatabase 'RSG\mailbox
database'

  • Use the following command to restore a mailbox in a Recovery Storage Group into a folder in the same active mailbox or a different active mailbox:

    Restore-Mailbox -RSGMailbox 'SourceUsername' -RSGDatabase 'RSG\Mailbox Database' -id 'DestinationMailbox' -TargetFolder 'FolderName'

  • To recover email in a particular date range into a folder in the same active mailbox or a different active mailbox, use this command:

Restore-Mailbox -RSGMailbox 'SourceUsername' -RSGDatabase 'RSG\mailbox database' -id 'DestinationUsername' -TargetFolder 'FolderName' -StartDate 'mm/dd/yy' -EndDate 'mm/dd/yy'

A note about "single mailbox restores" in DPM 2007: If you use DPM 2007 to restore a single mailbox, DPM still restores the entire mailbox database to your Exchange server, and you still have to use powershell to pull the mailbox data out of the database. Personally, I don't use this feature. It seems repetitive to me. But if you do use it, make sure you have enough space to restore the whole database.

Monday, February 16, 2009

OCS 2007 R2 Group Chat Installation - Part 3, Client install

Earlier, in Part One, we installed Group Chat Server and in Part Two, we learned how to connect to the admin tools and to create a channel and allow people in it. Now it is time to Deploy the client.

The Group Chat client requires the .NET Framework 3.5 SP1 and Visual C++ runtime installed. Unfortunately, Microsoft provided only an executable client install, so we cannot easily GPO this installation without an MSI file. There are the below options for the clientsetup.exe executable:

Working with this, I created a logon script to do this installation. If a client does not have .NET 3.5 SP1 installed, and you skip that install, the unattended install will give an error.

\\2008dc\netlogon\dotnetfx35setup.exe /qb \\2008dc\netlogon\Clientsetup.exe /Unattend

An added dose of fun - .NET requires a reboot, and /qb tries to force that reboot.

The actual manual install process is very simple - just take all the defaults.

Getting the software installed is just one part - getting it configured is another. First I will cover the manual configuration. Then I will discuss deploying these settings in an enterprise.

The more critical knowledge here is how to deploy this to clients and have the logins work out of the box. Using the "Automatic Configuration" provided, my first sign in was less than desirable. I got the below error:

And was unable to search for a channel (it just seems to hang there trying) so I created a new configuration named domain.com like I did in Part 2. I had to investigate further.

So I decided to try the Administrator account. It worked fine. So for some reason - the "Server Address" had to be "Administrator@domain.com" and then when I actually log on, I use my Chris@domain.com SIP URI. Very odd behavior. Above is the NON working config. Below is the working config.


Odd for sure. Again, when I sign into the application, I am clearly me, not admin:

Now, configuring another piece of client software might be easy for us, but not for most users, so it's time to learn how to deploy these settings.

So I downloaded the OCS 2007 R2 ADM files and went to create a new GPO, and quickly found that none of the GPO names in the XLS document for Group chat are in there.

It seems as is there should be an additional ADM coming, or an updated one with these settings.

I did find that you can manipulate these by dropping an XML file into the workstation directory of:

C:\Users\%username%\Application Data\Microsoft\Group Chat\Common\Accounts (on Vista) and by editing or replacing the file at:
C:\Users\chris\Application Data\Microsoft\Group Chat\Group Chat Console\Accounts\_default.account_.xml

So I made a second script:

copy \\2008dc\netlogon\_default.account_.xml "%userprofile%\Application Data\Microsoft\Group Chat\Common\Accounts"
copy
\\2008dc\netlogon\Internal.xml "%userprofile%\Application Data\Microsoft\Group Chat\Group Chat Console\Accounts"

Since these folders won't exist until the application is installed, you may want to stagger your install and your configuration GPOs/Scripts.

If anyone has any better way to roll out the configuration, please let me know, I really do feel like these are some non-enterprise level workarounds.

Sunday, February 15, 2009

OCS 2007 R2 Group Chat Installation - Part 2, Administrative Tools Installation

Earlier, in Part One, we installed Group Chat Server and now it is time to install the administrative tools. I have already done this a few times and ran into some oddities, that hopefully I can help you not run into!

In this instance, I am installing the tools on my OCS Standard pool server (which is also my group chat server) We can begin by running the 'AdminSetup' installer.


The installer will warn you if you do not already have MS Visual C++ redistributable installed. If you run their installer of this, do note, you will be cleaning up the root of your C drive.


Icky. Easy enough to clean up though. Once this is installed, its easy to click through all the defaults and complete the Admin tool console install.

Once installed, we can launch the Admin tools console. The "proper" name doesn't fit in the start menu, really.

So I attempt to logon:


I see "Connected to OCS server" and then I get this:

"Cannot sign in because of a problem with the chat room service. If the problem persists, please contact your system administrator"

Time to not trust Automatic Configuration. I will come back to this in the client deployment some more, but for the admin console, a manual configuration is OK.

Choosing Edit Accounts here, I made a new Account and used the below settings:

I then chose the Domain.com account and signed in without error. In the screen above, the Host is my OCS std pool, the domain.com is my AD FQDN and note the capitalization on Administrator to match my SIP URI.

Now that I am in the tool - lets create a test channel and set it up.

File>New>Chat Room brings up this dialog:
This gave an error. Channel names cannot have spaces. I removed this moving forward.

Once the room is created you need to add members to the channel. Since this is a lab domain, I decided to add my members to the Root of the server. In a production environment, I would likely be much more careful about security.

Now, Part 3 will come soon, and discuss the client installation, but I will give a brief preview here as the client is obviously needed to test this.


Here, you can see presence integration.

That's all for now. Check back soon for the client deployment, which I hope to include GPO settings to configure the client as well.

Thursday, February 12, 2009

OCS 2007 R2 Group Chat Installation - Part 1, Server Installation

If there is one thing in Group Chat that is similar to OCS, it's definitely the icons. If you were not aware, the Group Chat functionality was an acquisition of a company named Parlano in the summer of 2007. Because of this, the integration with the rest of OCS R2 isnt quite where I would expect it to be. I expect in future SP's and versions this will become more tightly integrated.

A note, on Group Chat pre-requisites. I did this install on my existing standard pool which is not recommended. It should be on a separate server.

This is a screenshot of the IIS features I have installed. I have not yet found any article on group chat pre-requisites on a fresh server. I will need to do this soon and update this post.

Downloads:
ServerSetup
ClientSetup
AdminSetup

I will review OCS 2007 R2 Group Chat in three or possibly four parts.

  • Server Installation
  • Administation Installation and configuration
  • Client deployment
  • Usage, recomendations

Lets dig in with the Server Setup.exe. I wish that this was more OCS integrated from the start. We begin with a VERY non MS DOS window. This launches the Group Chat Server install, that looks and feels familiar.



From there, Accept terms, enter user/company, choose features… two choices, Chat Server and Compliance Service. Both install by default.

Get a warning about MSMQ not being installed. This might be due to my eval install rather than real standard. Either way, if you plan on using Group Chat, or archiving/monitoring, you need MSMQ.

And then…. I am removing backup files:


And I did this twice to be sure. Yep, that's a window popping under to install. And "Server configuration" with a new icon. Of course, these are the pains of integrating Parlano, whom Microsoft acquired only 2 years ago.


Choose server and DB name:


Note here: You do need to manually create this Database.

Next screen, you can choose a different DB for compliance (likely policy recommended)

I chose the same for technical ease.



It's around this point I noticed the menu on the left gave no knowledge of how many steps remained. Not a big deal, but would be nice to see that the list has a start and a finish.

Here is where we set our Super Users:



Enter your OCS pool name, then choose Browse to select the MTLS cert to use.


The next few screens, I am setting a few service accounts up. For simplicity, I used my Domain Admin account for most of these. In a production environment, I would ABSOLUTELY have separate accounts for these.


For some reason, the browse button on this dialog was not working for me. Not a big deal. This should be a UNC path. I made it on the same machine for my instance.

I did not delve into the Compliance Adapters at all.


Next screen was Web Service Settings. Same deal on the UNC share:

Final Overview before hitting Install:

When you hit Finish, you will be alerted if any of the service accounts are granted the log on locally or log on as a service right.

Then we flip back to the MS installer and we are done!


Once this is installed - you will only have the OCS R2 Group Chat icon in your administrative tools.


Launching this allows you to view status of services and stop/start them as well as modify the service accounts.

I am not yet sure why the Web Service is listed as stopped. The W3SVC is running in the Services control panel, however.

Within this, File - Configure Server Settings allow you to modify pretty much ANY of the settings you specified in the installation. I am not going to review each of these, as I think it would be repetitive.

Next up - the Administrative Tool!

Monday, February 09, 2009

OCS 2007 R2 Deployment, Part 4

We have already prepared Active directory, Installed OCS 2007 R2, and configured the server and certificate.
In order to have internal IM and LiveMeeting capabilities, we only need to do a few more things.
  • Create required DNS entries
  • Enable users for OCS 2007 R2
  • Deploy Communicator Clients
  • Enable Audio/Video on clients with new hardware


So, first off.. If you are only doing internal OCS, you only need to be concerned with your internal LAN DNS.

Communicator looks for several DNS entries. If you turn on event logging in the client, you learn it looks (in this order - replace domain.com with your internal FQDN)

  • SRV record for SIP

  • sipinternal.domain.com

  • sip.domain.com

  • sipexternal.domain.com

I created sip.domain.com as a CNAME to my OCS 2007 R2 server:



Enabling your users is very simple from ADUC once the OCS administrative tools are installed. Since OCS 2007 R2 is 64 bit only, I have yet to find the 32 bit admin tools. (If you find this, please link me and I will update this post!)

Right click the user in ADUC and choose Enable user for OCS


Choose the server or pool, the SIP naming convention (I recommend using email address), then next and finish.

Deploying the Communicator 2007 R2 client (OC 2007 R2, but I find that naming to close to not be confusing) can be done in MANY ways. It is supplied as an MSI, so you can:

  • Put the MSI on a file share and email users the instructions
  • Deploy via GPO
  • Logon Scripts
  • Deploy using SMS/SCCM/SCE

Once deployed, for domain machines with Communicator installed, it is as simple as launching in order to get the IM and presence functionality. Beyond this, there is audio/video which requires PC's with webcams and headsets with microphones. If this is not pre-existing, I typically budget 80-120$ per node for the addition of hardware as needed.

After installing our webcams and microphones, internally we were able to do VOIP and Video calls immediately after deploying the client.

Impact of ESX snapshot backups on Microsoft database servers

Up until Update 2, ESX 3.5 uses a VMWare tool called the Sync Manager as part of the snapshot backup process. The Sync manager quiesces the file system (pauses all incoming I/O requests and dumps dirty data to disk) before the snapshot backup is taken. This allows the backup to be crash-consistent.

If you were to take a snapshot without pausing the I/O requests and then restore the snapshot, the virtual machine will start up for the first time thinking that it is recovering from a power failure type of crash. But the recovered system will not be able to find the I/O requests that were stored in the memory (RAM) at the time the snapshot was taken. So data loss is possible.

What does this have to do with database servers? For servers housing databases (Active Directory, Exchange, or SQL servers), stopping I/O requests with the Sync manager halts incoming requests to the database without notifying the database of what is happening. The database is waiting on that information to arrive – so when the data doesn't come in when expected, errors are logged to the event log and in some cases, the databases become corrupt. I happened to find this out on an active directory domain controller, and the sequence of errors looks like this:

First, you'll see event ID 1 logged with source LGTO_Sync. This is the Sync driver starting to do its work quiescing the file system.

On domain controllers, AD requests will begin to fail. The description will differ based on the request, but the Event ID stays the same.

For domain controllers running DNS, dynamic updates will fail as well:

For DHCP servers, you will see this:

On Exchange servers, you'll see Autodiscover errors:

If you are seeing these errors, stop using the sync manager now. Eventually you will corrupt your database.

Workaround
Stop quiescing guest database servers before taking snapshots, and start adding snapshots of virtual machine memory to your backups. Most backup applications allow you to do this. If yours doesn't, you can script it using vimsh.

Example:
vimsh -n -e "vmsvc/createsnapshot [VmId] [snapshotName] [snapshotDescription] [includeMemory]"

vimsh -n -e "vmsvc/createsnapshot XXXX FIRST_SNAPSHOT MY_FIRST_SNAPSHOT_1"

By taking a snapshot of the guest machine's memory, you are creating a full snapshot. When you restore, you restore the memory on top of the file system. When the machine starts, it will be able to access all the necessary information in memory to start normally - no crash.

Resolution
To combat the Sync Manager problem, ESX has released update2, which includes an ESX VSS tool that integrates with Microsoft VSS. It works by using the windows operating system to hold I/O requests, eliminating the need for the sync driver. When the operating system is in charge of halting its own I/O activity, the databases are notified that a backup is taking place. The databases can then pause their own processing of requests, and no errors occur.

This update is relatively new, and many third-party backup applications do not support update 2 yet, which is why I have offered the workaround here.

One last note about Microsoft domain controllers and Vmware snapshot backups
In 2006 (revised in Dec. 2008), Microsoft released KB 888794 (http://support.microsoft.com/kb/888794/en-us), which states that

"Active Directory does not support any method that restores a snapshot of the operating system or the volume the operating system resides on. This kind of method causes an update sequence number (USN) rollback. When a USN rollback occurs, the replication partners of the incorrectly restored domain controller may have inconsistent objects in their Active Directory databases. In this situation, you cannot make these objects consistent. "

In reality, the BURFLAGS registry referred to in Microsoft KB290762 (http://support.microsoft.com/kb/290762) can be set so that the virtual DCs are nonauthoritative, and an existing domain controller can be set to authoritative. This will allow the USN to be overwritten by the authoritative domain controller, and no USN rollback will occur.