Problem
During archiving, items are first written to the Enterprise Vault (EV) Storage Queue and will then need to be ingested to the EV Partition from the Storage Queue.
Certain items have been identified as being invalid S/MIME items. Opening these items in Outlook would fail, as would any attempt to archive them from a mailbox. However, if MS Exchange Journaling is enabled, and one of those items is on a journal report, then Enterprise Vault Journaling will successfully add the item to the storage queue.
The problem then occurs trying to ingest this item, as it gives a MAPI_E_NOT_FOUND error. This is treated as a recoverable error and, once the ingestion retries are exhausted (default:
100 times, 15 seconds between each), the item will be constantly re-queued.
This problem will have 2 main consequences:
- Depending on the number of items with this problem, the rate of ingestion from the Storage Queue to the partition may be severely reduced, as each affected item will keep a thread busy for ~25 minutes per attempt. If all the Storage Archive threads are busy retrying items (by default 25 threads), the Storage Queue location can fill up rapidly as ingestion to the partition is not keeping up with the number of new items being written to the Storage Queue. This may eventually cause the EV Services to stop due to insufficient free space on the disk.
- When an item is encountered which triggers the continuous retry, indexing of subsequent items will stop.
EV Indexing will only index items which have been successfully ingested into the partition. When a retried item is encountered, the Indexing Process believes that it has caught up with the archiving process and believes that there are no more items to index. This will lead to a backlog of items awaiting to be indexed
Error Message
- Warning 29057 in the Event log may indicate that this issue is occurring
Type : Warning
Date : 11/02/2015
Time : 13:11:27
Event : 29057
Source : Enterprise Vault
Category : Storage Archive
User : N/A
Computer : EVServer
Description:
A problem was encountered while trying to process a saveset in the storage queue, but it appears that the problem may be recoverable.
The saveset will be re-queued and another attempt will be made to process it later.
Details: <0x8004010f>
Storage Queue Location: D:\EVStorageQueue
Storage Queue Batch File: D:\EVStorageQueue\EVStorageQueue_1C3BE654F18A7CB4AADC47D47B85B69131210000EVServer_20141201201445_00400.EVSQ
Transaction Id: 90010762A8C08D8A425707D643C39181
Vault Store Entry Id: 1C3BE654F18A7CB4AADC47D47B85B69131210000EVServer
V-437-29057
- A Dtrace of the StorageArchive Process will also highlight Error Status 0x8004010f every time the item is retried and eventually tracing 'All 100 retries of InsertQueuedItemCommand have failed' when Warning Event ID 29057 is logged in the Event Log
Dtrace snippet:
120883 16:46:13.376 [14792] (StorageArchive) <13852> EV:L {CPoisonPill::CPPControl::UnregisterPP} (Entry)
120884 16:46:13.376 [14792] (StorageArchive) <13852> EV:L {CPoisonPill::CPPControl::UnregisterPP}|Unregistering PP - 0x3f0ef14
120885 16:46:13.376 [14792] (StorageArchive) <13852> EV:L {CPoisonPill::CPPControl::UnregisterPP} (Exit)
120886 16:46:13.376 [14792] (StorageArchive) <13852> EV:H {CVault::InsertSaveset} (Exit) Status: [<0x8004010f>]
120887 16:46:13.376 [14792] (StorageArchive) <13852> EV:H {CStoreAccessor::InsertItem} (Exit) Status: [<0x8004010f>]
120888 16:46:13.376 [14792] (StorageArchive) <13852> EV:H {InsertQueuedItemCommand::Execute:#42} _com_error exception: [<0x8004010f>]
120889 16:46:13.376 [14792] (StorageArchive) <13852> EV:H {InsertQueuedItemCommand::Execute} (Exit) Status: [<0x8004010f>]
120890 16:46:13.376 [14792] (StorageArchive) <13852> EV:H {EVCommand::Trace:#42} All 100 retries of InsertQueuedItemCommand have failed. hr=<0x8004010f>
120891 16:46:13.376 [14792] (StorageArchive) <13852> EV:H {EVCommandExecutor::ExecuteCommand} (Exit) Status: [<0x8004010f>]
120892 16:46:13.376 [14792] (StorageArchive) <13852> EV:H {ArchiveSavesetRequest::InsertItem} Status: [<0x8004010f>]
Cause
The root cause of this problem is that items are being flagged as S/MIME, but are not actually S/MIME formatted.Workaround
The workaround below reduces the impact that this issue has on the Ingestion and Indexing
- Reducing the time spent retrying items
With the current default settings, when an item is retried, there will be 100 attempts every 15 seconds, meaning the thread will not be doing anything else for 25 minutes.
The maximum number of retry attempts for an item can be reduced to a smaller value such as 10 before it is re-queued again to retry later.
The retry interval can be left at 15 seconds which means that the threads will be busy for 2.5 minutes only.
The behavior is controlled using the two registry keys under HKLM\SOFTWARE\Wow6432Node\KVS\Enterprise Vault\Storage
- InsertQueuedItemMaxRetries
Type: DWORD
Value: Between 0 and 1000
Default Value: 100
Suggested Value: 10
- InsertQueuedItemRetrySleepIntervalSeconds
Type: DWORD
Value: Between 1 and 600 seconds
Default Value: 15
Suggested Value: 15 (same as default)
- InsertQueuedItemMaxRetries
Solution
This issue has been addressed in the following release:
Enterprise Vault 11.0.1 Cumulative Hotfix 2 Release
https://www.veritas.com/docs/000025301
Enterprise Vault 12
The following link contains information about Enterprise Vault 12
https://www.veritas.com/docs/000099905
Note: For information on how to obtain Enterprise Vault 12 see: How to obtain the license key and installation download for Veritas products
Additionally when archiving to Non-Centera Partitions, the following registry key needs to be created to process malformed attachments without single Instanced during archiving.
Location HKEY_LOCAL_MACHINE \SOFTWARE \Wow6432Node \KVS \Enterprise Vault \Storage
REG_SZ InsertQueuedItemNonRecoverableErrors
Value (HEX) 0x8004010f