"Insufficient Resources" in Microsoft Message Queues (MSMQ) and interference with antivirus software.

Problem

Enterprise Vault may experience performance related issues due to a low KernelMemThreshold threshold for the Microsoft Message Queues (MSMQ).  The MSMQ reports insufficient resources when kernel memory reaches 80% utilization.  By default the kernel does not begin the clean up unused memory until it 90% utilization is reached.

Error Message

Event ID 2778
The error c00e0056 occurred whilst calling the method CArchivingAgent
V-437-2778

Cause

Scheduled virus scans can leak memory in the PoolPagedMemory.

Solution

In all Windows Server environments greater than 2000 SP3, increase the kernel memory threshold limit as suggested in Microsoft Knowledge article 811308.  Increasing value of the registry key for KernelMemThreshold gives the kernel a chance to clean up unused memory before MSMQ reports insufficient resources.

Microsoft Support Article 811308 - Addition of KernelMemThreshold:
Note: The below text references information directly from Microsoft Support article 811308.
 

Important: This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:
    322756 How to back up and restore the registry in Windows
 

In certain circumstances, you may want to raise the threshold because an application, a driver, or a system component is consuming kernel memory. Garbage collection is done by the kernel only when the paged pool memory consumption reaches 80 percent. However, Message Queuing stops functioning at 80 percent of paged pool memory consumption. If you set the kernel memory threshold above 80 percent, this makes sure that Message Queuing does not go into "low memory mode" until the Windows Memory Manager starts cleanup.

In other circumstances, you may want to lower the threshold to limit the amount of kernel memory that is used by Message Queuing.

To change the kernel memory threshold, follow these steps:

1. Install the hotfix that is mentioned in this article. (If running Windows 2000 SP4, if version is greater proceed to step 2)
    
2. Add the following registry key:
   HKEY_LOCAL_MACHINE\Software\Microsoft\MSMQ\Parameters\KernelMemThreshold of data type DWORD

          The default is 80. The value can range from 1 to 99. Anything else is converted to 80.

Note When you create this value the Decimal option must be selected. If you select Hexidecimal, the default value of 80 is used.

To change the kernel memory threshold on a clustered Message Queuing server, follow these steps:

1. Install the hotfix that is mentioned in this article on each cluster node. (If running Windows 2000 SP4, if version is greater proceed to step 2)
    
2. Add the following registry key of data type DWORD on each cluster node:
   HKEY_LOCAL_MACHINE\Software\Microsoft\MSMQ\Parameters\KernelMemThreshold the following registry key of data type DWORD on the active node in the cluster: HKEY_LOCAL_MACHINE\Software\Microsoft\MSMQ\Clustered QMs\MSMQ$[Clustered resource name]\Parameters\KernelMemThreshold

Note: Allowing antivirus software to scan the storage directory may result in message corruption.

Note: See Microsoft Support article 312362 for additional information on the tuning of PoolUsageMaximum and PagedPoolSize.    These settings further optimize MSMQ performance.

Applies To

 Identified in Windows 2000 SP4 and addressed in a Hotfix.
    Microsoft Support Article 811308.