1. Support
  2. Knowledge base
  3. 100018940

Important Update: Cohesity Products Knowledge Base Articles

All Cohesity Knowledge Base Articles are now managed via the Cohesity Support Portal: https://support.cohesity.com/s/searchunify. The Knowledge Base articles available here will not reflect the latest information or may no longer be accessible.

Enterprise Vault Compliance & Discovery Accelerator searches causing indexes to fail.

Article: 100018940
Last Published: 2024-05-21
Ratings: 2 0
Product(s): Enterprise Vault

Problem

Enterprise Vault Compliance & Discovery Accelerator searches cause indexes to fail.

Error Message

In the Info column of the Compliance Accelerator or Discovery Accelerator Search listing of the index volumes searched:

Search request failed. Reason: %1 Index Id: %2 Requestor: %3 Search Query: %4 Search Arguments: %5 [0xc0041c67]
 

On the Accelerator Server:

Event Type:       Error
Event:            297
Source:           Accelerator Service Processor
Description:
APP AT - Customer ID: 2 - Error processing Vault to Search off the queue: Searching Vault Thread - worker thread for index server: 1 (0)
Vault ID: 123456
Number of tries:  5
Cause:            Search request failed.   Reason: %1   Index Id: %2   Requester: %3   Search Query: %4   Search Arguments: %5   [0xc0041c67]

V-437-297


Event Type:       Error
Event Source:     Enterprise Vault
Event ID:         7182
Description:      Index Search failed: Catastrophic failure  [0x8000ffff] Index: Internal reference: SRCI

V-437-7182


Event Type:       Error
Event Source:     Enterprise Vault
Event ID:         7231
Description:      A COM error occurred. Catastrophic failure  [0x8000ffff] Internal reference: CRSS GetIndexServer

V-437-7231


On the Enterprise Vault (Indexing) Server:

Event Type:       Error
Event Source:     Enterprise Vault
Event Category:   Index Server
Event ID:         7235
Description:      A low level indexing operation has failed
Error:            a severe error occurred; cannot continue [AV:63] Index Id: 1EF83A57578A93849B3AC814A32AB2F181110000DOMAIN.com/Volume:1234 (INDEX_VOLUME)

V-437-7235


Event Type:       Error
Event Source:     Enterprise Vault
Event Category:   Index Server
Event ID:         7271
Description:      Search request failed.
Reason:           A low level indexing operation has failed   Error:    Index Id:    [0xc0041c43] Index Id: 1EF83A57578A93849B3AC814A32AB2F181110000DOMAIN.com/VolumeSet:1234 (INDEX_VOLUME) Requestor: DOMAIN\USER

V-437-7271


Event Type:       Error
Event Source:     Enterprise Vault
Event Category:   Index Server
Event ID:         7264
Description:      Abnormal error occurred
Error:            A low level indexing operation has failed   Error:    Index Id:    [0xc0041c43] Reference: CAltaVistaSearch::NonChunkedSearch Index:     1EF83A57578A93849B3AC814A32AB2F181110000DOMAIN.com/Volume:1234 (INDEX_VOLUME) Info:      EM/h

V-437-7264


Event Type:       Error
Event Source:     Enterprise Vault
Event Category:   Index Server
Event ID:         7292
Description:      The index volume has been marked as failed. Index Volume: 1F2CD2269BFC2F94A8630D390E41931771110000DOMAIN.com/Volume:1234 (INDEX_VOLUME) Index Volume Path: C:\Indexes\1F2CD2269BFC2F94A8630D390E41931771110000DOMAIN.com_1234567
Reference:        Open
Due to errors accessing the index volume it has been marked as 'failed' to prevent further errors.  The index volume will remain inaccessible until it has been repaired. 

V-437-7292

 

Cause

There are several factors which can contribute to these types of errors/symptoms, including but not limited to the following:

  • Improper registry keys (i.e., AVSMaxLoc registry key for 32-bit index volumes set greater than 10000000 decimal).
  • Out of date or improper NIC drivers.
  • NIC Teaming.
  • NIC binding order.
  • OS patches or service packs which have made changes to TCP/IP settings.
  • TCP Chimney Offload and/or TCP/IP Offload Engine (TOE) are enabled (see Related Articles in this document for additional information).
  • I/O degradation on indexing storage device.
  • Size of Index Volume on disk.
  • Schema Type of Index Volume.
  • Total file system size larger than the OS can properly handle (32-bit OS) [i.e., between 14 to 21 TB of total disk storage].
  • Index fragmentation on 32-bit indexes.

 

Solution

In order to conduct a search, Compliance Accelerator (CA) and Discovery Accelerator (DA) must establish communications with the Indexing service(s) of the Enterprise Vault server(s) hosting the targeted index / indexes.  If these communications are degraded the above errors may be seen.

Troubleshooting Actions Checklist:

1. Check the Enterprise Vault Indexing registry values against the recommended settings below.
 
2. Check the properties of the Network Connections for Network Interface Card (NIC) teaming.
 
3. Check the Advanced properties of the Network Connections for the binding order of the NICs (NIC through which the EV traffic flows must be #1 in the binding order).
 
4. Check for any TCP/IP and / or NIC changes that may have been caused by driver, patch or Service Pack updates.
 
5. Ensure the TCP Chimney and / or TCP Offload Engine (TOE) features are disabled on all NICs through which EV traffic flows (see the Related Documents section of this article for more information on these features).
 
Important Note For Servers In A Clustered Environment Or That Use Network Interface Card Teaming

It is of vital importance to determine if a server is a member of a cluster BEFORE making any changes to the TCP Offload Engine settings described in this article. Examples include Windows Server Failover Cluster nodes and SQL Always On Availability Groups replicas. Some cluster applications require TCP Offload Engine to be enabled on each cluster node or replica for proper functionality. Disabling any TCP Offload Engine settings on cluster nodes or replicas could adversely affect network performance for cluster-aware applications and/or operating systems. As such, it is recommended not to edit any TCP Offload Engine settings for servers that are nodes or replicas in a clustered environment without first consulting the cluster application documentation. If the cluster documentation clearly confirms TCP Offload Engine settings can be changed without any negative effects, then proceed with the changes after creating a plan to roll back the changes if needed. When in doubt, do NOT make any changes to the TCP Offload Engine settings.

Similar consideration should be given to servers using Network Interface Card (NIC) teaming. Some NIC teaming applications require TCP Offload Engine to be enabled on each NIC for proper functionality. Disabling any TCP Offload Engine settings on teamed NICs could adversely affect network performance for cluster-aware applications and/or operating systems. As such, it is recommended not to edit any TCP Offload Engine settings for servers that use NIC teaming without first consulting the NIC teaming documentation. If the NIC teaming documentation clearly confirms TCP Offload Engine settings can be changed without any negative effects, then proceed with the changes after creating a plan to roll back the changes if needed. When in doubt, do NOT make any changes to the TCP Offload Engine settings.

6. Check for any I/O degradation on the indexing storage device(s).
 
7. Verify the Index volume folder size is between 2 to 3 gigabytes for 32-bit index volumes.

8. Determine the Schema Type of 32-bit index volumes by running the utility IndexServer /summary "x: \IndexVolumeFolderPath" and verify the Schema Flags option is reporting the value of 3
 (see Related Documents section of this article for more information).

9. On 32-bit operating systems, open Task Manager and view the Performance tab section for Kernel Memory (K) to note if the Paged memory amount increases beyond 150 MB with no Enterprise Vault services running (normal usage should be between 40 MB and 150 MB, depending on total file system space).  Monitor the Paged memory use to see if it grows after starting the EV services and during CA or DA searches and / or exports, without reducing in size after the CA or DA searches and / or exports complete.  Note that this Paged memory value increasing over 450 MB used will cause the errors noted above and will cause index volumes to be randomly marked as failed by CA or DA searches.
 
10. Obtain the AVTrace.log file for 32-bit index volumes for future analysis.


Workaround / System Optimization:

Note: The below changes are meant to throttle-back Compliance or Discovery Accelerator so that less of a demand is placed upon the indexes.  The desired result is that the searches can then be conducted without failing the indexes or receiving errors during the search.  These changes will also have an impact upon performance.  

In Discovery Accelerator 8.0 and above:

1.  On the CA or DA client, click on the Configuration tab.

2.  Next click on the  Settings sub-tab.

3.  Hold down the  Ctrl key and click on the words " Configuration Settings" in the banner portion of the pane.

4.  Scroll down and expand the  Search folder.

5.  Click on the Value column next to the " Maximum Number of consecutive searches on same index" option.

6.  Change the value to 200. In a 32-bit Indexing environment or a mixed 32-bit/64-bit Indexing environment, if unsatisfactory Index Search performance is still seen, this can be further incrementally reduced and Searches tested, but should not normally be set below 10; if it is determined setting this value to 1 results in the optimum Index Search performance, then it is recommended the Search criteria be modified and/or the Indexing server(s) be reviewed for available resources and overall Index health. In a 64-bit environment, if unsatisfactory Index Search performance is still seen, this can be further incrementally reduced and Searches tested, but should not normally be set below 100; if it is determined setting this value to 10 or less results in the optimum Index Search performance, then it is recommended the Search criteria be modified and/or the Indexing server(s) be reviewed for available resources and overall Index health.

7.  Click on the Value column next to the " Number of Vault search Threads" option.

8.  Change the value to: 2

9. Click on the Value column next to the  " Search Result Chunk Size" option.

10. Lower the value to any value between 500 to 20,000 (lower the number, the less memory used on the EV indexing server, but the slower the search may progress).

11.  Click the  Save button in the bottom right of the panel, then click on the OK button if a pop-up dialog window appears indicating the need to restart the Customer Background Tasks and / or Remoting.

12.  Restart the Customer Background Tasks or the Enterprise Vault Accelerator Manager Service (EVAMS) on the CA or DA server.

If the above changes still do not allow the searches to complete successfully, the below actions and registry key changes can be added to the Enterprise Vault server hosting the targeted Indexing service.

Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry.  Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and workstation be made prior to making any registry changes.


System changes:

1. Place the Enterprise Vault service account (VSA) TEMP folder on a drive with more then 20 GB of available space (40 GB or greater preferred)
(My Computer | Properties | Advanced | Environmental Variables | User variables for VSA)

2. Disable anti-virus software from scanning the VSA TEMP folder .  See Article 100017720, "Recommended list of antivirus exclusions for Veritas Enterprise Vault", in the Related Articles section of this document for more information.


Registry changes for 32-bit indexes (EV 8.x and 9.x - not needed for EV 10 and greater):

1.  Open regedit.

2.  Navigate to:
 
32-bit Operating System Registry key: HKEY_LOCAL_MACHINE\Software\KVS\Enterprise Vault\Indexing
64-bit Operating System Registry key: HKEY_LOCAL_MACHINE\Software\Wow6432Node\KVS\Enterprise Vault\Indexing
 
          Note: If the Indexing key is not present, create it.

3.  Create new DWORD value: SearchChunkSize

4.  With decimal value of:   5000    (only lower this value in extremely limited systems and under guidance of a Veritas Enterprise Vault Support personnel)

5.  Create new DWORD value: AVSMaxLoc
         See Note 2 below.
6.  With decimal value of:   500000000

7.  Create new DWORD value: SchemaType
         See Note 2 below.
8.  With decimal value of:   1
 
Note:
When these edits are completed, the Indexing registry key contents should look similar to, but may have more than, the following:
Name                              Type                       Data
(Default)                          REG_SZ               (value not set)
AVSMaxLoc                  REG_DWORD     0x1dcd6500 (500000000)
SchemaType                 REG_DWORD     0x00000001 (1)
SearchChunkSize         REG_DWORD     0x00001388 (5000)

9.  Restart the Enterprise Vault Indexing Service on the affected EV server. 

10.  Rebuild the affected index volume(s) (i.e., the index volumes that show the error in their Info column).
 
11. Restart EVAMS on the CA or DA server to trigger an immediate synchronization of the index volume changes that were made by rebuilding the index volume.
 
12. Re-run the search.

Notes:
1. As with the configuration changes in Compliance or Discovery, these changes will impact performance.
2. Will only affect new archives and rollover volumes.  In order to have the settings effect previous archives, a rebuild of the index is necessary as noted in Step 10.
3. Not having the AVSMaxLoc registry value set per the above instructions can randomly cause the latest index volume of any archive being searched to be marked as failed during a CA or DA search.  The decimal value of 500 million keeps the 32-bit index volume physical size between 2 GB and 3 GB.  This allows for a better fit of the search criteria, index volume contents and search hits into the operating system restriction of 2 GB of memory allocated to each 'IndexServer.exe' process used while searches are run.


The event errors listed in this article were found to be a result of an improper NIC binding order on the Enterprise Vault server hosting the targeted Indexing service:

1.  Open Control Panel.

2.  Open Network Connections.

3.  Click the Advanced drop down menu.

4.  Select Advanced Settings...

5.  Move the primary network interface to the top of the binding order (See Figure 1)

Figure 1
 


If it is found that the NIC bindings are in the proper order, it is recommended to investigate other possible causes as listed in the "Causes" section above.  This type of issue is inherently environmental in nature e.g.:

- Networking
- Storage
- Improper registry entries

It would prove beneficial to verify that there are no I/O problems with the storage device hosting all the indexes.  Overall, it is suggested to look at any recent changes that may have occurred to either the Accelerator or Enterprise Vault servers as this may lead to a root cause.
 
If it is found that the Paged Kernel Memory is increasing over 450 MB, the possible issue is that too much file system space is configured for the server to handle.  For example, a Windows 2003 SP2 Enterprise Edition 32-bit server with over 21 TB of SAN attached storage can cause the Paged Kernel Memory to increase to over 450 MB, which does not allow sufficient Kernel Memory to be available for other functions needed by EV indexing.  If this condition exists due to too much file system space, the resolution is to replace the EV server with a 64-bit server, provided that the existing file system space must remain.
 
For fragmented 32-bit indexes:
1. Determine the SearchID:
- Click on the Cases or Departments tab in the Client.
- In the left pane of the lower page section, click on the Case or Department with the Search for which the SearchID is to be obtained.
- Click on the Searches tab to access the listing of Searches for the specified Case or Department.
- Hover the mouse over the name of the Search.
- Note the SearchID displayed next to the 'Search:' label and the CaseID displayed next to the "Case:' label in the pop-up window.
 
2. Edit the first line of the following query to replace AA with the SearchID obtained above. Run the query against the appropriate Customer database:
 
DECLARE @SearchID nvarchar(10) SET @SearchID = AA ---Change the SearchID here SELECT tis.Name AS 'Search Name'
     , ts3.Name AS 'Search State'
     , tv.KVSVaultName AS 'Archive Name'
     , tsv.StatusID AS 'Archive Search StatusID'
     , ts4.Name AS 'Archive Search Status'
     , tv.KVSVaultEntryID
     , ISNULL(tivs.FirstItemSeqNum, -1) AS 'First Item Sequence Number'
     , tsv.Info AS 'Information'
     , tvs1.VaultServer 'Vault Server Name'
     , tvs2.Name 'Vault Store Name'
FROM tblSearchVault tsv
JOIN tblIntSearches tis ON tsv.SearchID = tis.SearchID JOIN tblStatus ts3 ON tis.StatusID = ts3.StatusID JOIN tblStatus ts4 ON tsv.StatusID = ts4.StatusID FULL OUTER JOIN tblVaults tv ON tsv.VaultID = tv.VaultID JOIN tblVaultServers tvs1 ON tv.VaultServerID = tvs1.VaultServerID JOIN tblVaultStore tvs2 ON tv.VaultStoreID = tvs2.VaultStoreID FULL OUTER JOIN tblIndexVolumeSet tivs ON tsv.IndexVolumeSetID = tivs.IndexVolumeSetID FULL OUTER JOIN tblIndexService tis2 ON tv.IndexServerID = tis2.IndexServerID WHERE tsv.SearchID = @SearchID AND tsv.Info LIKE '%c0041c67%'
ORDER BY tv.KVSVaultName, tivs.FirstItemSeqNum
 
3. Implement the compact.task defragmentation process:
3.1. For 32-bit Index Volumes on EV 10.0 and above:
- Expand Archives in the Vault Admin Console (VAC)
- Find the Archive containing the Index Volume listing the error.
- Right-click on the Archive.
- Select the  Properties option.
- Select Index Volumes tab.
- Select the  Details button.
- Look for the FirstItemSeqNum as found above in the Range column. This will be the Index Volume needing compaction.
- Note the information in the Location column of the index volume having that FirstItemSeqNum value.
- Right click the Volume with that FirstItemSeqNum value.
- Select the  Set Index Volume Offline option
- Acknowledge any prompts.
- Close all dialog boxes.
- Browse to the Index Volume folder as listed under the Location column of the index volume properties.
- In the folder, create an empty text document with the name: compact.task.
- In the VAC, close the Archive Properties window.
- Right-click on the  Indexing folder.
- Select the  Manage Indexes option.
- Select the Synchronize option to bring up the first page of the Synchronize Wizard.
- Click the Next button to access the Synchronize page.
- Click the  Add button.
- Uncheck all check boxes in the left panes except the Offline option in the Attributes: pane and the 32-bit option in the Platform: pane.
- Click the Search button.
- When the search for offline index volumes completes, a listing of all such index volumes found will appear in the upper portion of the  Search Results pane.
- Select the index volume by clicking on it.
- Click the Add button to move it to the Selected Items panel.
- Click the  OK button to return to the Synchronize page.
- Click the Next button.
- Change the default entry in the Task Name: field as appropriate (recommendation is to leave the default entry alone).
- Click the Next button.
- Click the Create Task button.
- Click the Close button.
- Back in the VAC, access the Properties page of the archive as was done at the beginning of these instructions.
- Select the  Index Volumes tab again.
- Right click on the offline index volume.
- Select the Set Index Volume Online option.
- Acknowledge any prompts.
- Close out of the dialog boxes and return to the VAC.
- Expand the  Enterprise Vault Servers folder.
- Expand the folder for the EV server hosting the archive's index.
- Select the Tasks option under the EV server name.
- Refresh the Tasks panel view if the Index Administration Task is not displayed so that this task will be displayed.
- Ensure the Index Administration Task is in a Processing ... status.
- If the task is not processing, 
     - Right-click on the  Index Administration Task.
     - Select the  Run Now option.
     - Click the Yes button then the OK button to acknowledge the task is to start processing immediately.
- Monitor the Task run
     - Right click on the Indexing folder.
     - Select the Monitor Indexing Tasks ... option.
     - Refresh the Search Results panel view periodically until the task shows as Successful (or Unsuccessful).
- When complete, recreate any Accelerator Searches and monitor the Searches.
 

 

References

Etrack : 7231 Etrack : 7264 Etrack : 2425686 Etrack : 7235 Etrack : 7271 Etrack : 7182 Etrack : 7292 Etrack : 297 UMI : V-437-7235 UMI : V-437-7231 UMI : V-437-7264 UMI : V-437-7182 UMI : V-437-7292 UMI : V-437-297 UMI : V-437-7271

Was this content helpful?

Article Languages