Thursday, October 08, 2009

Exchange 2007 SCR How to - Part 2

This is a continuation from Part 1 where we configured the SCR replication.

Failing over to the target SCR server

In this example, I am not having an ACTUAL failure. I am choosing to dismount the DB on the source server. I typically also will check the "do not mount the database store on startup" just so that if I do get to stopping/starting any services later that I don't accidentally remount the DB that I am attempting to fail over. And once I successfully fail over, I delete the now "old" SG and database from the EMC as well as the EDB file and transaction logs that were associated. I like to keep out of date data tidy like this. If you disagree, at the minimum, you should move these files into a well labeled folder so you know exactly what/when the files were from so that 6 months later (or wherever your comfort level is) you can do housecleaning in an educated manner. In order to ensure we have live data, I sent myself an email just before dismounting the database. You can see behind it the Get-StorageGroupCopyStatus showing the status as Healthy, with the hard coded 50 log replaylaglength.

First, we dismount the active database (in a real failure, something did this for you)

Dismount-Database OECU-EXCH1\ExecDB

Once the source database is dismounted, we can begin the SCR activation process. The first step here is to create a new SG and a new DB. DO NOT USE the folders/files/paths of your SCR data! For example, create a new SG named "RecoverySG" and a new DB named "RecoveryDB" and have their paths be new/unused folders and paths. (this is the part I mentioned above is not clear enough in the technet article)Run powershell as admin so it can do file operations!

New-StorageGroup -Name RecoverySG -LogFolderPath 'D:\Exchange Logs\RecoverySG' -SystemFolderPath 'D:\Exchange Logs\RecoverySG'

New-MailboxDatabase -name RecoveryDB -StorageGroup RecoverySG -EdbFilePath 'D:\Exchange Databases\RecoveryDB\REcoveryDB.edb'

Notice again - those are NEW and empty paths and files. If you attempt to use your SCR data at this point, you will have undesirable results! Now we run the restore command. This checks the status of the log shipping, and will attempt to copy missing log if files if needed. It also disables the original SCR and makes the database viable for mounting.

Restore-StorageGroupCopy "EXCHANGESOURCE\ExecSG" -standbymachine EXCHANGETARGET

Now is a good time for a quick reminder on what SCR is and how it does log shipping. SCR essentially just copies log files to the target, and when a backup occurs, the target will also replay those log files. So if we skip the next step, we risk bringing up a database that essentially is only as up to date as the last backup. If that was just before this test, it may not be a big deal, or may not be noticed (especially in lab where you don't have live mail flow, etc) So now we need to run ESEUTIL /R to replay the log files. This is done by running eseutil from the location of the log files like so:

eseutil /r E00

The /r Exx is the prefix for that databases logs (you can check by looking in the log folder directory for that storage group)

This should replay the logs and bring the database to a clean shut down state. You can confirm this by running eseutil /mh and specify the EDB file. The database state should be Clean Shutdown.

The below command updates the new RecoverySG's paths to match the paths of our SCR Database. The "configurationOnly" flag here tells it NOT to move the existing file, but to just change the configuration.

Move-StorageGroupPath "EXCHANGETARGET\RecoverySG" -SystemFolderPath "D:\Exchange Logs\ExecSG" -LogFolderPath "D:\Exchange Logs\ExecSG" -ConfigurationOnly

Now we need to do the same thing for the Database - point the "new" DB at our "recovered" data.

Move-DatabasePath "EXCHANGETARGET\RecoveryDB" -EdbFilePath "D:\Exchange Databases\ExecDB.edb" -ConfigurationOnly

Now we need to set the database to allow file restore. This is what will allow this database to be mounted.

Set-MailboxDatabase "EXCHANGETARGET\RecoveryDB" -AllowFileRestore:$true

If you skip the above step, when you attempt to mount the database, you will receive an error that appears to be permissions related.

Mount-Database "EXCHANGETARGET\RecoveryDB"

This is where most admins breathe a sigh of relief, but we aren't done - we need to move user s to this DB. Well, not really. What this really does is updates these user's AD objects to have their Exchange server and homeMDB at their new location in the RecoveryDB.

Get-Mailbox -Database "EXCHANGESOURCE\ExecDB" where {$_.ObjectClass -NotMatch '(SystemAttendantMailboxExOleDbSystemMailbox)'} | Move-Mailbox -ConfigurationOnly -TargetDatabase "EXCHANGETARGET\RecoveryDB"

The "where" clause in the middle of this is to prevent you from moving the System mailboxes that are unique to each DB. When you run this command - assuming your SCR target is in a different AD site - keep in mind you will need to sync AD to have your users in the main site start coming back online. You can trigger this in AD Sites and Services or various other ways, or just wait for replication to occur. At this point the users and database should all be online and client access to their data should be restored. If any HT servers in your organization had mail delivered for these users during the outage, it would deliver now. I recommend using OWA to test data as you may have client connectivity issues to troubleshoot as well with Outlook or Outlook Anywhere.

We can see the test email that I sent just prior to dismounting the database on the SCR source, so we know the logs replayed correctly. If your data is "out of date" then the likely thing that did not work is your ESEUTIL /R to replay the logs. You may be able to dismount the database and replay these again, but if you leave the DB mounted for a while and new messages flow in, your log sequence will likely be corrupted or broken. If you do this, you likely will need to go to restoring your DB from the previous night and then attempting to replay the SCR log files again.

Reseeding back to the original source server

Reseeding back is the exact same process, but your source and target are flip flopped. So you re-seed your "live" data that is on your DR server back to your main server. First, clean up the old DB/SG and the files/folders under it on your target server. Then, you can choose to rename/modify any of the DB or SG names or paths to your liking. (This can be skipped if wanted, but needs to be done before you configure SCR) Then, you can repeat the creation of an SCR replica and reseed the data back. Once data is seeded and healthy on the target, you repeat the failover process to "fail back" Once you fail back, clean up all the SG/DB paths/names once again on the DR server. Don't forget to recreate the SCR seed to your DR location after!


Anonymous said...

Hey Chris,

You did a great job with the "Exchange 2007 SCR How to" post. Very good info and to the point. Before implementing SCR in my environment I had to read lots of confusing documentation and never right to the point like your post. I wish I could find this blog before my implementation, it would have saved me lots of time. Thanks for taking the time to share your knowledge with the Exchange Community!

Also, I would like to get your oppinion on something related to the Activation process of the SCR but when you have CCR as the SCR Source. That's basicaly what I have and I need to document the Activation Process as part of our Disaster Recovery Plan.

My config is:

CCR as the SCR Source in our Production site and Stand-Alone server as the SCR Target in our DRP site running the Mailbox Server role. I have another server at the DRP site running the HUB and CAS roles

Would the Activation process using Database Portability would be different to the steps you described in your post?

I'm a little confused with the info provided in KB691321

where they say: "if the SCR source is a clustered mailbox server and the SCR target is a standalone mailbox server instead of a standby cluster, the full server recovery process takes longer. The reason for this is that a standalone Mailbox server has its own Mailbox server identity which must first be removed (by uninstalling Exchange) so that Setup /RecoverCMS can be run"

They talk about uninstalling exchange as part of the activation process to use Setup /RecoverCMS but this contradicts with they say in the beginning of the same KB where they indicate: "You can activate an SCR target using database portability, using the server recovery option (Setup /m:RecoverServer)" so not quite sure they are involving the Setup /RecoverCMS if I have Stand-Alone as Target and plan to use DB Portability

Would I need to uninstall Exchange as they said?

Any feedback would be greatly appreciated!

I can be reached at:

Thanks in advance and keep up the great work!


Frank (Houston, TX)

Cameron said...

I would rework the section near the end with the Get-Mailbox command. I couldn't get it to work without finding another document on the process.

"How to Restore a Mailbox Database to a New Server with Database Portability"

The problem seemed to be with the pipe command. I'm wondering if your blog removed them.

Chris Lehr said...

I think I have fixed this. For some reason MS word to blogger drops the pipe character and I didn't notice it. - thanks for pointing that out!

Unknown said...

Hi Chris,

I have a question when you posted "Get-Mailbox...."

This cmdlet works when you have the SCR source, but how can i move the users to the scr target when the scr source is unavailable?

Thanks in advance

TK said...

I am trying to capture this stuff conceptually in an outline form. If I try to boil this 2 part article down to a 500 foot high view, would the following be correct?

1. Follow all the steps in Article Part 1 and verify that replication has copied the database and logs from Source onto the Target using the exact same file paths as exist on Source.

2. You do not need to have proactively set up duplicate Storage Groups on the Target. In fact you maybe should definitely not do this? I'm guessing that the creation of the SG and DB is taken care of later automatically during the restore process.

3. However you can proactively set up temporary Storage Groups and Databases on the Target that do not in any way match those on Source, either in name or in their respective data paths. This is for the database portability functionality that will be used in a failover scenario.

4. PRIOR TO FAILOVER TIME: Prior to a Restore operation on Target, if you had proactively created those temp SGs and DBs on the Target, ensure that you have mounted & dismounted them and then emptied them of all files in their respective data paths locations.

5. FAILOVER TIME: Do the Restore command on the Target. But where is this database being restored to? To the temp SG and DB? If so how did it know to do that since the restore command is not specific in that sense?

All that command said is: Restore-StorageCopy "SourceServer\StorageGrpoupName -standbymachine TargetServer.

Is it because we only have created a temp SG on the Target that it knows where to restore to? What if you have several temp SGs. This is where I am still confused.

6. Since I'm not clear at all on the last step I should pause here, but to continue, the above will not replay those 50 held-back log files?

Since step 5 does not apparently replay those last 50 logs then you need to do an eseutial /r E00? At this point I'm not conceptualizing where the database really is nor what SG it is in, so I will wait until I can get further clarity before chattering on. Thanks