Exchange 2010 - Recovery Scenario #2 - Recover from a DAG member loss

In this scenario, I have a 3 three server DAG, and I use Windows Server backup to backup my Exchange 2010 Active database. On the server with the active copy, I hit the virtual power button. The Exchange services fail over to another server in the DAG right away.

The Microsoft documentation:

Recover a DAG member Exchange Server

Remove the copy of the Database in a DAG:

This will warn that it cannot communicate with the server. That is expected.

Then, you can remove the server from the DAG:

Reinstall Windows 2008 R2 from DVD (remember, DAG requires Enterprise!)

Reset computer account in Domain (Right click Reset in AD Users and Computers)

Name and IP the server, confirm the date/time is correct (since in a DAG, I also needed to IP my DAG network)

Install Exchange Pre-Requisites

Install Exchange 2010 using:

setup /m:recoverserver

If you skipped the DAG removal steps above, setup will fail with:

Once setup succeeds, you need to reboot the server (at this point, I would also patch as needed - at the time of this writing 2008 R2 and Exchange 2010 RC have no additional patches)

Since I have a DAG, I am able to re-add it to the database availability group and allow the database to reseed. If you were in a single server environment, this is where your backup would come into play. This might be scenario #3

Add the recovered server back to the DAG


Add a mailbox database copy to the recovered server.

Assuming you have a real DAG and you might have 300GB of data to re-seed by adding a copy, this is where the Windows Server Backup may play part. You may be able to restore the Exchange data to an alternate location, and before you add the database copy, move the restored EDB to the folder path for the database. This would allow you to skip time consuming reseeding, as long as your restored EDB was the most recent backup taken of the database.

Optionally, you can activate the database on the recovered server as well.

Exchange 2010 - Recovery Scenario #1 - Mailbox or items

In this post I wrote about how you can now backup Exchange 2007 SP2 and Exchange 2010 mailbox databases with Windows Server backup.

Since them, word of Exchange 2010's release has come, and with that, the question of "when will my backup vendor provide updates that are compatible with Exchange 2010" and "what can I do in the meantime"

One of the easiest solutions is using Windows Server backup, and then allow your existing backups to do file level backups of that data. The question once this is in place, of course is how do I recover from that?

So I intend to cover three scenarios:

1. Single Item or Mailbox recovery - accidental delete, assuming you passed or misconfigured deleted item and deleted mailbox retention.
   
2. Loss of a DAG member - How to recover from losing a single member of a DAG.
   
3. Entire Server recovery - Building/site failure, need to return to service and restore data from backups. (this will be single server from backup)

Additionally, I am using a database that is in a DAG for this, but am writing it as if it was standalone, as the #2 and #3 scenarios would be addressed by the Database Availability Group.

So, on to scenario #1 - I have disabled my mailbox in the EMC, and run the below powershell to force the database clean:

Clean-MailboxDatabase geodb1

Now I see my mailbox under "Disconnected Mailbox" in a normal scenario, this is what Exchange 2003 and up has offered, where I could right click my mailbox and choose to re-connect it to my user account:

Of course, I want to go to backups so I reset my database to have a 0 day deleted item mailbox retention and refreshed this screen and my mailbox was no more!

Do note, the settings in the above screenshot are NOT recommended. Default is 30 days, and I recommend leaving it there or higher!

Next, we must recover the data "to an alternate location" using Windows Server Backup
Choose your Backup Date, then your recovery type should be "Applications"

Choose Exchange:

(I included the show details, which is the store GUID)

I chose here to recover to another location (Note: c:\RDB1 is NOT where my RDB's EDB/logs/anything are)

Do note that "this option will copy just the application data" - there are additional steps after this!

Finally, launch the recovery.

Once completed, you will have the file structure of the database in the path specified:

Now that we have our data files, the recovery is similar to Exchange 2007 SCR's database portability.

Run ESEUTIL /R from the log file directory

Then we can run:
Eseutil /mh geodb1.edb

And determine the DB is healthy:

Next few steps edited on 11/3/2009 for missing content:

Now we can create our new Recovery mailbox database using:
new-MailboxDatabase -Recovery -Name rdb1 -Server exch2010 -EDBFilePath "c:\rdb1\rdb1.edb" -LogFolderPath "c:\rdb1\"

Then we need to allow Restores:
Set-MailboxDatabase -AllowFileRestore:True

Copy the EDB file to the path EDBFilePath of the RDB1 Database, renamed it appropriately, and then it should mount successfully (NOTE: Logs didn't need to be copied since ESEUTIL /R replayed them into the EDB, however if you do copy them into place, Exchange will see they are replayed and move on)

Once mounted, we can use

get-MailboxStatistics -database rdb1

to see that the data is there:>

Now, in the Exchange documentation, it states that:

Restore-Mailbox -Identity chris -RecoveryDatabase rdb1

would recover the data into the mailbox. The problem is, we don't have a mailbox with that GUID any more. If I re-enable a new mailbox for chris, he will get a new mailbox GUID.

Enable-Mailbox chris
Restore-Mailbox chris -RecoveryDatabase rdb1

I get:

This makes sense, it cannot match GUID's and stops - more on this in a second.

However, you are able to run a recovery operation (similar to Export-Mailbox in Exchange 2007)

Restore-Mailbox -RecoveryMailbox chris -Identity chris -RecoveryDatabase rdb1 -TargetFolder "Recovery"

And the results, all of the content in a subfolder named "Recovery"

I attempted a few other things to see if I could restore directly into the mailbox, but was not able to find any luck.

Important to note - if I was recovering for a user that missed their deleted item retention time, I can use the restore-mailbox to specify by subject, dates, folders and more. Because I mail disabled the user, I am not able to restore directly.

OCS Voice Ignite Training - Registered!

Yay!

We have been trying for most of the year to get me a seat at one of these, and instead, I think we got two seats so I will get to tag along with one of our Cisco Voice guys as well (this should be helpful in backfilling voice knowledge for me!)

Pretty stoked to get there. Irving, TX in February!

Installing Exchange 2010 quickly using PowerShell

In Exchange 2007, I typically used ServerManagercmd.exe to quickly deploy required Exchange 2007 parts. In Exchange 2010, when I ran ServerManagercmd, I get the warning that:

Servermanagercmd.exe is deprecated, and is not guaranteed to be supported in future releases of Windows. We recommend that you use the Windows PowerShell cmdlets that are available for Server Manager.

This is replaced by Powershell commands:
Get-WindowsFeature
Add-WindowsFeature
Remove-WindowsFeature

When you run those - you will get an error. Need to run this first:
Import-Module ServerManager

So let's see how fast we can make this go! I am installing for ALL roles. If you need to split out roles, you should read the MS documentation at:
http://technet.microsoft.com/en-us/library/bb691354(EXCHG.140).aspx

You can of course use the Servermanagedcmd -i that they give you, but knowing it's deprecated and I will be doing this 40 times next year, I wanted to know the new way. So here it is!

Install 2008 R2 off a CD/ISO
Set computer time, networking, machine name and domain
Install the AD tools using Powershell
Add-WindowsFeature RSAT-ADDS
Re-Boot
Upon reboot, launch PowerShell as Administrator and copy paste the below (again, this is for MBX, HT, CAS on a single server, check the link above for more detailed pre-requisite planning)
Add-WindowsFeature Web-Metabase, Web-Lgcy-Mgmt-Console, Web-Server, Web-ISAPI-Ext, Web-Metabase, Web-Lgcy-Mgmt-Console, Web-Basic-Auth, Web-ASP, Web-Digest-Auth, Web-Windows-Auth, Web-Dyn-Compression, Web-Net-Ext, RPC-over-HTTP-proxy, AS-NET-Framework, NET-HTTP-Activation

Set the TCP .net sharing service to automatic startup
Set-service NetTcpPortSharing -startuptype automatic

Optional - download and install the x64 version of the Microsoft Office Filter Pack (this allows office attachments content to be searched and indexed.

And from the Exchange 2010 install directory: .\setup.com /mode:install /roles:mb,ht,ca

Now, if you take this and extend it to using PowerShell's remote capabilities, you can prep a BUNCH of 2010 servers quickly!

Exchange 2010 - What is an arbitration mailbox?

If you found this searching you most likely found out about arbitration mailboxes much the way most people will. By either finding out they accidentally deleted them, or by finding out that you need to move, disable or remove them in order to delete a database, or uninstall Exchange 2010, or remove the mailbox role.

From TechNet:
"Arbitration mailboxes are used for managing approval workflow. For example, an arbitration mailbox is used for handling moderated recipients and distribution group membership approval."

This is part of the Moderated Transport features that are new in Exchange 2010.

A lot more information about using arbitration mailboxes can be found here: Understanding Moderated Transport

In short, arbitration mailboxes are where messages awaiting moderation are stored, as well as information about moderator decisions are kept.

Now back to getting over the two most common immediate needs for arbitration mailboxes.

I deleted my arbitration accounts from my AD
This isn't really all that bad. I did it the first time I installed Ex2010 and had a panic moment before I found the fixes. Pretty simply, you need to rerun the AD preparation steps from the 2010 media.

Setup.com /PrepareAD
Setup.com /PrepareSchema
Setup.com /PrepareDomain

Only /prepareAD is required to recreate these accounts, but I left the other steps in here as well just for documentations sake.

I am trying to remove Exchange 2010, or a database, or the Mailbox role and am being told there are arbitration mailboxes preventing me from continuing
This is also not too bad. When you try to remove the first DB in Exchange 2010, there are a few arbitration mailboxes that will prevent database deletion. You have the choice of moving, removing, or mail-disable these mailboxes. Since you cannot see these in the Exchange Management Console, you need to launch Exchange Management Shell (EMS)

Get-Mailbox -Arbitration

This will list the arbitration mailboxes. To narrow it down to a specific database, you can edit this to:

Get-Mailbox -Arbitration -Database DB1

If you are used to PowerShell cmdlets in Exchange 2007, one big change to recall here is that specifying servername\databasename won't work anymore. This is one of the reasons why the database names need to be unique to the organization - so you don't have to specify servers anymore!

Once you have your "get" command returning the correct list of mailboxes, it's time to move, disable or remove them. Disabling the last arbitration mailbox is not allowed, so I recommend moving them as the first preference here.

Move:
Get-Mailbox -Arbitration -Database db1 | New-MoveRequest -TargetDatabase db2

Disable:
Get-Mailbox -Arbitration -Database db1 | Disable-Mailbox -Arbitration

Remove:
Get-Mailbox -Arbitration -Database db1 | Remove-Mailbox -Arbitration -RemoveLastArbitrationMailboxAllowed

If there is enough interest a little later, I may do a write up on using the arbitration mailboxes, but at this point there is still a lot of other Exchange 2010 things to learn and figure out!

Virtualized DC issue with time synchronization

This was a pretty simple mistake but it took me a while to figure out. We were noticing everything on the domain was 10 minutes behind other devices we used and in troubleshooting, I configured my PDC emulator to sync with pool.ntp.org using NTP, and it did, and seconds after, the time would revert back.

Of course, the issue here was that the DC was virtualized and Hyper-V time synchronization was taking preference and syncing to the Hyper-V server's local time which had fallen out of sync. The fix here was either disabling time synchronization on my DC, or enabling NTP pool synchronization for my Hyper-V server. I chose the latter, and moments later all machines were the correct time.

Credit where credit is due:
http://www.aperture.ro/index.php/2009/01/windows-time-sync-hyper-v-enabled-domain-controller-dilemma/