Compliance Accelerator Searches show Index Volumes stuck in Not Started, Queued or Processing statuses

Problem

Enterprise Vault (EV) Compliance Accelerator (CA)/Surveillance Searches with Intelligent Review enabled appear to stall. Some Index Volumes complete, while others remain in Not Started, Queued or Processing statuses. Searches that do not have Intelligent Review enabled complete as expected.

Intelligent Review is a feature that uses a learning engine to mark items intelligently based on the actions that Reviewers have taken on earlier items. For example, after a Reviewer has marked a spam message or out-of-office reply as Irrelevant, when other items that have similar characteristics are found, they can be similarly marked by Intelligent Review's learning engine automatically. When Intelligent Review is enabled in a Search, the learning engine processes items whilst the Search is in progress to expediently provide the Intelligent Review results when the items are added in to Review.

Error Message

A Dtrace on the CA server of AcceleratorService64 lists the following exception (formatted with line breaks for easier reading):

[1234]  (AcceleratorService64)  <Process search results - worker thread (1):5678>  EV-L  {-} {TraceDLCommand} {C...} Executed usp_SearchItem_Bulk_Add in ... ms. {TraceDLCommand}
[1234]  (AcceleratorService64)  <Process search results - worker thread (1):5678>  EV-H  {SearchQueues} {C...} Bulk inserted items for SearchResult ID: ... SearchID X
[1234]  (AcceleratorService64)  <Process search results - worker thread (1):5678>  EV-L  {-} {IntelligentReviewUtils} {C...} Starting to update relevance using IR 2.0 on search results for SearchID X SearchResultsID : ..
[1234]  (AcceleratorService64)  <Process search results - worker thread (1):5678>  EV-L  {-} {IntelligentReviewUtils} {C...} Successfully updated relevancy on searchItems for which relevance already present in tblItemrelevance table for SearchID X SearchResultsID : ...
...
[1234]  (AcceleratorService64)  <Process search results - worker thread (1):5678>  EV-H  {-} Exception: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index Info:{SearchQueues} {C...} Exception while trying to update relevance on searchItems for searchID : X 
Diag: 
Type:System.ArgumentOutOfRangeException 
ST:   at System.ThrowHelper.ThrowArgumentOutOfRangeException(ExceptionArgument argument, ExceptionResource resource)
      at KVS.Accelerator.Search.IntelligentReviewUtils.UpdateRelevancyOnSearchItems(Int32 SearchID, Int64 SearchResultsID)
      at KVS.Accelerator.Search.SearchQueues.SearchResultsProcessWork(ThreadSafeQueue& theQueue, Object queuedItem) Inner:None

Cause

The Index Volumes move through various statuses throughout the life of a Search. Here is a brief description of the most common statuses:

Not Started (StatusID 11): The Index Volume is added to a list which is maintained in table tblSearchVault in the Customer database. The Search against this Index Volume has not yet started.
Queued (StatusID 15): Once the list is complete, the Index Volumes are added to a queue to be sent to EV Indexing so that Indexing can run the Search.
Searching (StatusID 12): EV Indexing acknowledges the Search request for each Index Volume, parses and recompiles the Search criteria into its own format, starts running the Search against the Index Volumes and returns items (results). These items are added in XML batches of 1000 (by default) to table tblSearchResults in the Customer database via the VSA TEMP folders on EV and the Accelerator server. The items from each batch are then extracted into table tblSearchItems in the Customer database. If Intelligent Review is enabled, the items in tblSearchItems are then processed by the learning engine. 
Processing (StatusID 18): The last batch of items for the Index Volume is received from EV Indexing and added to tblSearchResults. The processing of items from tblSearchResults into tblSearchItems continues through completion.
Finished (StatusID 13): All items in the last batch in tblSearchResults have completed extraction into tblSearchItems and have completed Intelligent Review processing (if enabled).

Other statuses include:

Error (StatusID 14): There was an issue reported by EV Indexing for the specific Index Volume.
Retrying (StatusID 16): There was a temporary or transient issue connecting to the EV Indexing server or running the Search. The Index Volume is being retried.
Paused (StatusID 525): The Search was paused which then pauses any Index Volumes that are in Searching or Processing statuses.

An issue was found whereby an unexpected exception in the Intelligent Review processing during the Searching or Processing phase caused the thread executing the processing to hang and not be released. This resulted in the status of those Index Volumes not being updated when expected and stalling the processing of the remaining Index Volumes.

Solution

There are no plans to address this issue by way of a patch or hotfix in earlier versions of the software at the present time.  However, the issue has been addressed in the revision of the product specified at the end of this article. 
 
Please contact your Sales representative or the Sales group for upgrade information including upgrade eligibility to the release containing the resolution for this issue.

This issue is fixed in the following release(s), available in the Download Center at https://www.veritas.com/support:
- Enterprise Vault 15.2