Ingestion of items incorrectly flagged as S/MIME are indefinitely retried

Problem

During archiving, items are first written to the Enterprise Vault (EV) Storage Queue and will then need to be ingested to the EV Partition from the Storage Queue.

Certain items have been identified as being invalid S/MIME items. Opening these items in Outlook would fail, as would any attempt to archive them from a mailbox. However, if MS Exchange Journaling is enabled, and one of those items is on a journal report, then Enterprise Vault Journaling will successfully add the item to the storage queue.

The problem then occurs trying to ingest this item, as it gives a MAPI_E_NOT_FOUND error. This is treated as a recoverable error and, once the ingestion retries are exhausted (default:
100 times, 15 seconds between each), the item will be constantly re-queued.

This problem will have 2 main consequences:

1. Depending on the number of items with this problem the rate of ingestion from the Storage Queue to the partition may be severely reduced, as each affected item will keep a thread busy for ~25 minutes per attempt. If all the Storage Archive threads are busy retrying items (by default 25 threads), the Storage Queue location can fill up rapidly as ingestion to the partition is not keeping up with the number of new items being written to the Storage Queue.  This may eventually cause the EV Services to stop due to insufficient free space on the disk.
    
2. When an item is encountered which triggers the continuous retry, indexing of subsequent items will stop.
   EV Indexing will only index items which have been successfully ingested into the partition. When a retried item is encountered, the Indexing Process believes that it has caught up with the archiving process and believes that there are no more items to index. This will lead to a backlog of items awaiting to be indexed
    

Error Message

1. Warning 29057 in the Event log may indicate that this issue is occurring

Type :        Warning
Date :        11/02/2015
Time :        13:11:27
Event :        29057
Source :        Enterprise Vault
Category :    Storage Archive
User :        N/A
Computer :    EVServer
Description:
A problem was encountered while trying to process a saveset in the storage queue, but it appears that the problem may be recoverable.
The saveset will be requeued and another attempt will be made to process it later.


Details: <0x8004010f>
Storage Queue Location: D:\EVStorageQueue
Storage Queue Batch File: D:\EVStorageQueue\EVStorageQueue_1C3BE654F18A7CB4AADC47D47B85B69131210000EVServer_20141201201445_00400.EVSQ
Transaction Id: 90010762A8C08D8A425707D643C39181
Vault Store Entry Id: 1C3BE654F18A7CB4AADC47D47B85B69131210000EVServer
V-437-29057

 

2. A Dtrace of the StorageArchive Process will also highlight Error Status 0x8004010f every time the item is retried and eventually tracing  'All 100 retries of InsertQueuedItemCommand have failed' when Warning Event ID 29057 is logged in the Event Log

Dtrace snippet:

120883    16:46:13.376     [14792]    (StorageArchive)    <13852>    EV:L    {CPoisonPill::CPPControl::UnregisterPP} (Entry)
120884    16:46:13.376     [14792]    (StorageArchive)    <13852>    EV:L    {CPoisonPill::CPPControl::UnregisterPP}|Unregistering PP - 0x3f0ef14
120885    16:46:13.376     [14792]    (StorageArchive)    <13852>    EV:L    {CPoisonPill::CPPControl::UnregisterPP} (Exit)
120886    16:46:13.376     [14792]    (StorageArchive)    <13852>    EV:H    {CVault::InsertSaveset} (Exit) Status: [<0x8004010f>]
120887    16:46:13.376     [14792]    (StorageArchive)    <13852>    EV:H    {CStoreAccessor::InsertItem} (Exit) Status: [<0x8004010f>]
120888    16:46:13.376     [14792]    (StorageArchive)    <13852>    EV:H    {InsertQueuedItemCommand::Execute:#42} _com_error exception: [<0x8004010f>]
120889    16:46:13.376     [14792]    (StorageArchive)    <13852>    EV:H    {InsertQueuedItemCommand::Execute} (Exit) Status: [<0x8004010f>]
120890    16:46:13.376     [14792]    (StorageArchive)    <13852>    EV:H    {EVCommand::Trace:#42} All 100 retries of InsertQueuedItemCommand have failed. hr=<0x8004010f>
120891    16:46:13.376     [14792]    (StorageArchive)    <13852>    EV:H    {EVCommandExecutor::ExecuteCommand} (Exit) Status: [<0x8004010f>]
120892    16:46:13.376     [14792]    (StorageArchive)    <13852>    EV:H    {ArchiveSavesetRequest::InsertItem} Status: [<0x8004010f>]

Cause

Workaround
The workaround below reduces the impact that this issue has on the Ingestion and Indexing

1. Reducing the time spent retrying items
   With the current default settings, when an item is retried, there will be 100 attempts every 15 seconds, meaning the thread will not be doing anything else for 25 minutes.
   The maximum number of retry attempts for an item can be reduced to a smaller value such as 10 before it is re-queued again to retry later.
   The retry interval can be left at 15 seconds which means that the threads will be busy for 2.5 minutes only.
   
   The behavior is controlled using the two registry keys under  HKLM\SOFTWARE\Wow6432Node\KVS\Enterprise Vault\Storage
   
    
   1. InsertQueuedItemMaxRetries
      Type: DWORD
      Value: Between 0 and 1000
      Default Value: 100
      Suggested Value: 10
       
   2. InsertQueuedItemRetrySleepIntervalSeconds
      Type: DWORD
      Value: Between 1 and 600 seconds
      Default Value: 15
      Suggested Value: 15 (same as default)
       

Solution

This issue has been addressed in the following release:
 
Enterprise Vault 11.0.1 Cumulative Hotfix 2 Release
https://www.veritas.com/docs/000025301
 

Enterprise Vault 12
The following link contains information about Enterprise Vault 12
https://www.veritas.com/docs/000099905

Note: For information on how to obtain Enterprise Vault 12 see: How to obtain the license key and installation download for Veritas products



Additionally when archiving to Non-Centera Partitions, the following registry key  needs to be created to process malformed attachments without single Instanced during archiving.
Location  HKEY_LOCAL_MACHINE \SOFTWARE \Wow6432Node \KVS \Enterprise Vault \Storage
REG_SZ   InsertQueuedItemNonRecoverableErrors
Value (HEX) 0x8004010f