Exchange Named properties limits and risk mitigation

Huge thanks to my co-worker, Matt Bastek (ill link a email or URL when he gets me one) for a lot of the leg work on this one in hunting it down. I merely played the role of reporter here.

Lately, we have heard a few customers reporting Event ID 9667 in their Application event log stating a failure to create a named property.

And reporting that some users were no longer able to receive email. The NDR sent to another internal user was:

#550 5.2.0 STOREDRV.Deliver: The Microsoft Exchange Information Store service reported an error. The following information should help identify the cause of this error: "MapiExceptionNamedPropsQuotaExceeded:16.18969:3D010000, 17.27161:00000000AC000000000000000000000000000000, 255.23226:00000000, 255.27962:7A000000, 255.27962:56000000, 255.17082:00090480, 0.16993:80030400, 4.21921:00090480, 255.27962:FA000000, 255.1494:00000000, 255.26426:56000000, 4.6363:0F010480, 2.31229:00000000, 4.6363:0F010480, 2.22787:00000000, 2.22787:00000000, 2.22787:00000000, 4.5415:00090480, 4.7867:00090480, 4.4475:00090480, 4.4603:00090480, 4.5323:00090480, 255.1750:00090480, 0.26849:00090480, 255.21817:00090480, 0.24529:00090480, 4.18385:00090480". ##

We investigated, and found out that there is a named properties table that is written to with additional headers from any internet messages. You can add these headers yourself, but often it is third party applications that are scanning, routing, marking emails with their stamp.

Here is the Microsoft article we found on the issue:
http://technet.microsoft.com/en-us/library/bb851495.aspx

Typically these are harmless and do not pose a risk. For many applications they are relied upon.

The issue we found with mail stopping delivery was due to the named properties database hitting its maximum limit as defined by the registry. This is 8192 rows by default, but is user configurable at HKLM\System\CurrentControlSet\Services\MSExchangeIS\SERVERNAME\STORENAME-GUID\

DWORD Value "NonMAPI Named Props Quota" Based on our findings this has a hard limit of 32767.

From here, we learn the default maximum header size is 64kb.
http://technet.microsoft.com/en-us/library/bb124345.aspx

64kb is ABOUT 2300 unique named properties. Knowing this, I sent a 61k message, hoping to fly under the radar. The email never came in and sat in the queue. I downed the VM and bumped RAM from 1024 to 2500 at this point.

In order to monitor the named properties rows, you need to enable the perfmon dll for Exchange.
http://support.microsoft.com/?kbid=254606

Then you can view/monitor your row count:

The below screen was the result of a single email on a test domain with 10 new headers.

I sent approximately 10k of random x-headers for this jump from 1200 to 1602

Here is the email - looks harmless to ANY user:

Unless you look at the headers:

So I sent a few emails to get the job done.

Now, it was about here I noticed I didn't get all my emails! So I checked the queue viewer and found that it was queuing up.

OK, great, maybe there was another layer of protection here. I started trying smaller snippets of header after this. Smaller gains in more emails, but I still only sent a handful.

Now, this is a VM with 2GB of RAM, so a memory issue isn't shocking. After waiting and submitting a few smaller emails as well, the works started flowing and they all went through:

And finally, I sent the final blow. All told, I only sent 25 emails for a total of around 120kb of data.

And what happened?

Not much, unfortunately.

In the event log, a warning, then a failure:

After that, I tried sending:

OWA to OWA internally - worked.
SMTP in without additional headers - worked.
SMTP in with additional headers - worked. Errors as shown above, but no non delivery of messages.

No service really denied. All aspects of email continue to work, but your event log begins to flood with those 9667 event id's These events are apparently pretty harmless.

Possible Risks:

• If your NamedProperties were full and you tried to add a new SMTP based service relying on new X-headers, and it might not function right.
• If you had existing NamedProperties applications (anti virus, anti spam, sharepoint, OCS, etc) these might have issues functioning correctly.
• Your application event log could be filled with the warnings and subsequent errors.
• Eventually, you could begin to have email denied to recipients in that database. While I was not able to reproduce this (this is a zero load Exchange server) the entire project here started because a customer running Exchange 2007 SP1 RU4 complained about this.
  

According to Microsoft, there are two fixes.

1. Up the registry entry at HKLM\System\CurrentControlSet\Services\MSExchangeIS\SERVERNAME\STORENAME-GUID\ (effective until you hit the hard limit of 32767)
2. Create a new DB and move mailboxes to the new store. Time costly option for little gain other than quiet event logs.

Details on both of these are here:
http://technet.microsoft.com/en-us/library/bb851493.aspx

Overall, I am not sure if this is really a risk, and if it is, I would love to know how.

Please, let me know.