What can cause the index to fail

Article: 100037871
Last Published: 2014-12-09
Ratings: 1 1
Product(s): Enterprise Vault

Description

Enterprise Vault Index Admin service (IAS) has been implemented to manage the legacy indexbroker.exe and the new EVIndexAdminService.EXE, it is responsible for monitoring the health of these processes and, if required, will attempt to restart them. As part of the responsibility of this service is to monitor the index volumes for their current state, The StoragePendingWorkDiscovery checks on a regular basis whether there are any index volumes that require synchronization. This is an hourly check that only selects index volumes that have their WorkPending flag set to 1 or when the service is first started, ignores any failed flags. The following event is logged:

Log Name:      Enterprise Vault
Source:        Enterprise Vault
Date:          05/07/2011 12:00:07
Event ID:      3197
Task Category: Archive Task
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      server.LOCAL
Description:
Starting synchronization run of all mailboxes on Exchange server D700

If an error is returned from the index engine indicating a potentially corrupt index volume it will attempt a search against the index volume to confirm whether it is failed.  The 64-engine could potentially have resolved the cause of the error on its own by the time the IAS has processed the error hence this procedure before marking a healthy index volume as failed.

The frequency of checks for index volumes to process is by default set to 1 hour, this is based on the work pending flag.

  • Ignores any currently failed volumes.

The frequency of checks can be controlled using configurable settings

  • VAC -> Server Properties -> Advanced -> Indexing
    • Frequency of checks for index volumes to process
      The time interval between the indexing service’s check for index volumes to process (Hours) 


The frequency checks for failed items to process is by default set to run every 6 hours, based on the failed flag and Failed Volume Reason.

  • Includes failed volumes where the reason is one of the failed volume ID’s:
    • 1,  2,  8,  9,  13,  14,  16

This can be controlled using configurable settings

  • VAC -> Server Properties -> Advanced -> Indexing
    • Frequency of checks for failed volumes. 

The time between the indexing service’s attempts to process failed index volumes (hours).
 

Every  ten hours a Full Check occurs and gets a list of index volumes owned by the Indexing service from the Directory database (stored procedure: EnumVolumesForPendingWorkCheck).

  • Processes the a list index volumes obtained to extract the Index Volumes for the appropriate vault store database(s) and then determines if there are any pending items in the ItemsadditionstatusLog, ItemsUpdateStatusLog and ItemsDeletionStatusLog. An Index volume has pending work if there are any Journal JournalArchive, JournalDelete, JournalUpdate records in the volume’s archive and within the its sequence number range with indexcommited=0.(stored procedure: CheckIsPendingRevisions) for Index Volumes belonging to the respective Archives in the Vault Store Databases.
  • The presence of items that are neither marked successful nor poison-pill with any of the three index status log tables will instigate a notification of work to be issued.
  • Any volumes found to have pending work, are queued for submission to the relevant Index Volumes Processor on their respective servers (64bit) or IndexServer (32bit).
    The Frequency of full checks for index volumes to process by default occurs every 10 hours for a full vault store database check.  This can be controlled using configurable settings.
  • VAC -> Server Properties -> Advanced -> Indexing
    o Frequency of full checks for index volumes to process.
    The time interval between the indexing service’s full checks for index volumes to process (Hours).


An information event is raised at the ten hour interval check as follows:

Log Name:      Enterprise Vault
Source:        Enterprise Vault
Date:          04/07/2011 01:52:32
Event ID:      41365
Task Category: Index Admin Service
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      D100.server.LOCAL
Description:
Summary of the current index volume check:

Volumes synchronized: 1
Failed volumes synchronized: 0

The next synchronization checks will take place as follows:

Volumes having pending work in 0 hours and 59 minutes
Failed volumes in 3 hours and 59 minutes
All volumes in 9 hours and 59 minutes.

 

General Troubleshooting Information

View the properties of an archive, under the index tab a warning may appear that one or more index volumes are marked as failed and a reason set on the index volume.  Enterprise Vault will mark an index as ‘Failed’ when a serious error occurs while processing the index that EV cannot handle.

Clicking on the index volume shows the failed volume error reason; this can be used later when repairing the index volume.  These index volumes are unavailable until they are repaired; this is to prevent any further changes to the indexes or potentially incorrect search results being returned.

There are several reasons why the index volume may fail, it does not necessarily mean that there is a physical issue with the index and does not necessarily mean the index is corrupt, it simply may need to be updated once the underlying issue has been resolved.

The problem could be any of the following; this is identified by the volume reason ID and Error reason. The volume Reason ID can be found in the Index volume table in the Enterprise vault Directory Database.

 

Error code Error description Possible Cause Suggestions
0 General error Unexpected errors
Check that the index volume exists on disk in the correct location and that it is accessible (i.e. no network connectivity issues).
Try using Verify to check the general health of the index.
Instead of rebuilding the volume, consider restoring from backup then using Verify and Synchronize.
1 The index volume folder cannot be found
Connectivity issues.
Index files accidentally deleted from disk
Check that the index volume exists on disk in the correct location and that it is accessible (i.e. no network connectivity issues). 
Instead of rebuilding the volume, consider restoring from backup and using Synchronize
2 There are too many consecutive items that have failed
Indexing service cannot access Storage service.
Storage service cannot access archived data.
Check the Windows event log on the computer running the Storage service, which may be remote from the server running Indexing. Check connectivity between Indexing and Storage if environment is distributed.
Also check availability of the vault store database and the vault store partition. Ensure both are online and responsive.
3 There is a sequence number mismatch between the index and storage Vault store database restored from backup but EVSVR has not been run. This may indicate a DR operation not completing successfully. Consider running EVSVR
4 The index volume has been partially deleted An error occurred whilst deleting index files from disk. An error occurred while deleting an index volume from disk. Enterprise Vault should automatically retry deletion later. In the meantime, check connectivity between the Indexing server and the volume’s index location.
5 The dataset file is missing (32-bit only)
Index files accidentally deleted from disk
Connectivity issues
Check the index volume’s updates.log and avtrace.log files. Instead of rebuilding the index volume, consider restoring from backup then using Verify and Synchronize
6 The dataset file is empty (32-bit only)
Index files accidentally deleted from disk
Connectivity issues
Check the index volume’s updates.log and avtrace.log files. Instead of rebuilding the index volume, consider restoring from backup then using Verify and Synchronize
7 The dataset file is invalid (32-bit only)
Index files accidentally deleted from disk
Connectivity issues
Check the index volume’s updates.log and avtrace.log files. Instead of rebuilding the index volume, consider restoring from backup then using Verify and Synchronize
8 Unable to open the index
Intermittent connectivity issues.
Index file corruption
Check connectivity between the Indexing server and index locations. Once the connectivity issue has been resolved, use Synchronize. 
If the index files are on disk but are corrupted, then consider using Rebuild. If the environment is under load and a Rebuild is necessary, consider rebuilding just the volumes affected by corruption. Note that when a 32-bit index volume is rebuilt, the data will be ingested into a 64-bit volume. Upon successful completion of the rebuild, the 32-bit volume will be deleted
9 The checksum file is invalid (32-bit only) Index files corrupted If the contents of the index are known to be valid, consider deleting the checksum file then using Synchronize.
10 An error occurred while saving indexing data to disk (32-bit only)
Connectivity issues.
Index files corrupted
Check connectivity between the Indexing server and physical index locations.
Check the index volume’s updates.log and avtrace.log files.
If the error persists, consider rebuilding the index volume
11 An error occurred while saving dynamic attribute indexing data to disk (32-bit only)
Connectivity issues.
Index files corrupted
Check connectivity between the Indexing server and physical index locations.
Check the index volume’s updates.log and avtrace.log files.
If the error persists, consider rebuilding the index volume
12 The index compaction has failed (32-bit only)  
Check connectivity between the Indexing server and physical index locations.
Check the index volume’s updates.log and avtrace.log files.
If the error persists, consider rebuilding the index volume
13 There has been an indexing engine error
System is overloaded.
Connectivity issues
Check the Windows event log on the Indexing server for possible clues. If the error is repeated for different volumes, check connectivity between the Indexing service and index locations.
Check if the system is running low on memory.
Use Verify and Synchronize to attempt to bring the volume back to life.
14 Failed to audit items Cannot access Database server Check the database server that hosts the vault store database(s). Fix any problems there, then run Verify and Synchronize.
15 There are duplicate items in the index Data corruption Run Verify. If the problem persists, consider rebuilding the volume.
16 The state file is missing from the index Index files accidentally deleted from disk If a backup of the index exists, consider restoring it and running Synchronize. Note that if the state file is missing in the backup too, Synchronize will trigger the re-indexing of all the items already in the volume
17 There was error accessing or updating the index engine metadata
Connectivity issues.
Directory service errors
Check the index metadata location and ensure it is accessible. Restart the Indexing service. Check connectivity between the Indexing service and the Directory service. Check access to database server hosting the Directory database.
18 There are items with incorrect information in the index Data corruption Run Verify and Synchronize. If the problem persists, consider rebuilding the index volume.

 

 

Was this content helpful?