Accelerator Searches fail with each Index listing time out error code x80041c71

Problem

Accelerator Searches fail with 64-bit Indexes listing time out error code x80041c71. The issue self-corrects after a period of time. This issue was first reported on 20 May 2016 and self-corrected the same day within 2-3 hours of being reported.

 

Error Message

Error in the Enterprise Vault Event Log on the Enterprise Vault Index Service server:
 
Log Name:      Veritas Enterprise Vault
Source:        Enterprise Vault
Event ID:      41315
Task Category: Index Query Server
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      EVServer1.mylab.local
Description:
A search failed with error "Search engine timeout".
 
Archive: JournalArchive
Archive ID: 17476B74168FDE24CBE01F1CBF22AA35E1110000mylab
Index volume ID: 1234
User: mylab\vaultadmin
Maximum results: 50000
Timeout (seconds): 14400
Sort order: snum 

v-437-41315

 
Error in the Enterprise Vault Event Log on the Compliance Accelerator or Discovery Accelerator server:

Log Name:      Veritas Enterprise Vault
Source:        Accelerator Service Processor
Event ID:      297
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      CAServer.mylab.local
Description:
APP AT - Customer ID: 2 - Error processing Vault to Search off the queue: Searching Vault Thread - worker thread for index server: 2 (2)
Vault ID: 12345678
Number of tries: 5
Cause:The search timed out.      (0x80041c71)

v-437-297

 

Cause

This issue's root cause is a processing issue with the search timeout value passed to the EV search engine.  The investigation has discovered this issue could re-occur periodically with a cadence of approximately 49-50 days. As such, a workaround was identified and is valid for Compliance Accelerator (CA) and Discovery Accelerator (DA), as described below. Implementing the workaround will alleviate the immediate timeout issue if and when it should reoccur. Any Searches exhibiting these symptoms should be re-run. It is best if the Searches are Rejected and Discarded, then new Searches created for the required Search Criteria.

Workaround:

A workaround for this timeout issue is to lower the Search timeout (hours) from the default timeout setting of 4 to 1. The Search option Search timeout (hours) specifies the maximum timeout per Index Volume in a Search. Reducing the timeout value for this setting to 1 does alleviate the timeout error upon Search submission, but could still be listed for individual Index Volumes that take longer than this value (in hours) to complete their Search.

The steps to follow for each CA or DA customer are as follows:

1. Launch the CA or DA Client using an account that has permissions to change configuration settings (such as the Vault Service Account).
2. Click on the Configuration tab to select this tab.
3. Click on the Settings sub-tab to select it.
4. Scroll down the list of configuration sections to locate and expand the Search section.
5. Scroll down the list of options in the Search section to locate the Search timeout (hours) option.
6. Change the Value column entry from the default value of 4 to 1.
7. Click on any other row in the Search section's options to move the edit focus and enable the Save button function.
8. Click on the Save button in the lower right of the page.
9. Click on the OK button in the pop-up dialog box to acknowledge the requirement to restart the Customer's Background Tasks.
10. Close the CA or DA Client.
11. Restart the Customer's Background Tasks (CBT) or the Enterprise Vault Accelerator Manager Service (EVAMS) on the CA or DA server to complete the change.

To restart the Customer's Background Tasks:

1. Log onto the CA or DA server as the Vault Service Account.
2. Launch Internet Explorer (IE) to the EVBAAdmin site (http://localhost/EVBAAdmin).
3. If needed, expand the listing of CA or DA customers in the left panel by clicking on the plus (+) sign to the left of the CA or DA server name.
4. Click on the CA or DA customer name in the left panel to display its properties in the right panel.
5. Right click on the CA or DA customer name in the left panel to display a listing of available options.
6. Click on the Customer's Background Tasks option.  This should remove the check mark to the left of the option and cause the Status section in the properties page view to change from Running to Stopping to Stopped.
7. After the Status has changed to Stopped, right click on the CA or DA customer name in the left panel to display the listing of available options again.
8. Click on the Customer's Background Tasks option again.  This should cause the check mark to the left of the option to return and cause the Status section in the properties page view to change from Stopped to Starting to Running.
9. Once the status is back to Running, close IE and log off of the server.

To restart the Enterprise Vault Accelerator Manager Service:

1. Log onto the CA or DA server using any account that has local administrator permissions (such as the Vault Service Account).
2. Launch the Services MMC snap-in.
3. Locate and select the Enterprise Vault Accelerator Manager Service.
4. Click on the Restart button in the toolbar.
5. Monitor the progress of the restart operation until the service restart has completed and the service is running again.
6. Close the Services MMC snap-in.
7. Log off of the DA or DA server.

 

NOTE: It may also be necessary to reboot the Accelerator server if restarting the Customer's Background Task and/or restarting the Enterprise Vault Accelerator Manger Service result in the same issue. In some cases, it may be necessary to wait until the cadence interval itself has been exceeded. Typically this is no more than approximately 6 hours from the time the issue is first experienced, and may vary from environment to environment. Once the cadence interval has been exceeded, Searches should run without any further intervention. Any Searches listed in an error status may be re-run.

 

Solution

Veritas has acknowledged that the above-mentioned issue is present in the version(s) of the product(s) referenced in this article. 

There are currently no plans to address this issue by way of a cumulative hotfix or service pack in the current or previous versions of the software at the present time. This issue may be resolved in a future major revision of the software at a later time, but is not scheduled for any release. If you feel this issue has a direct business impact for you and your continued use of the product, please contact your Veritas Sales representative or the Veritas Sales group to discuss these concerns.  For information on how to contact Veritas Sales, please see http://www.veritas.com. 

Etrack : 3880104