Event 40966 - Set-IndexLocationBackupMode - Failed Index Volume - Indexing engine unreachable.

Problem

Backup Exec job fails with the errors below while attempting to Set backup mode on the Index Locations.  Additionally, Event ID 40966 is logged in the Enterprise Vault event logs.

Error Message

Set-IndexLocationBackupMode : Unable to notify the indexing service on computer 'EVserver' of backup mode changes for index locations.

Event ID:  40966
Task Category:  Index Admin Service
Level: Error
Description: A program fault has raised an exception.
Exception: Failed Index Volume => Id: 3 Error: Indexing engine unreachable.

Diagnostic:
Type: System.Exception
Reference: An error occurred while invoking Void <NotifyIndexServiceBackupModeChangeEx>b__8().
Error Context : Handling index location backup mode change notification for IndexServiceEntryId 1AEC1C7046560B94BB5B469E3D91306B91710000EVserver1

Command Line: "C:\Program Files (x86)\Enterprise Vault\EVIndexAdminService.exe" -EntryID:1AEC1C7046560B94BB5B469E3D91306B91710000EVserver1
Application Domain: EVIndexAdminService.exe
Process Id: 5848 / Thread Id: 5824
Stack Trace:    at Symantec.EnterpriseVault.Indexing.Admin.EVIndexAdminUtils.InvokeHelper(Action act, String context, Boolean rethrow, Boolean shutdownCheck)
   at Symantec.EnterpriseVault.Indexing.Admin.IndexAdminCOM.ChangeIndexServiceBackupModeInternal(String indexServiceEntryId, Boolean wait)
   at Symantec.EnterpriseVault.Indexing.Admin.IndexAdminCOM.<>c__DisplayClassa.<NotifyIndexServiceBackupModeChangeEx>b__8()
   at Symantec.EnterpriseVault.Indexing.Admin.EVIndexAdminUtils.InvokeHelper(Action act, String context, Boolean rethrow, Boolean shutdownCheck)
V-437-40966
 

To validate if this is the issue a DTRACE is needed, please perform the following steps on the EV server that has generated Event ID 40966 while re-creating the issue.

• DTRACE the process EVIndexAdminService for no less than 5 minutes while re-creating the issue.
• Open the log file created and search for either of the following the strings:
   

[4228] (EVIndexAdminService) <2616> EV-H {INDEXADMINCOM.EN_US} Exception: Failed Index Volume => Id: 3 Error: Indexing engine unreachable.| Info: Diag: Type:System.Exception ST: Inner:None
[4228] (EVIndexAdminService) <2616> EV~E |Event ID: 40966 A program fault has raised an exception.|Exception: Failed Index Volume => Id: 3 Error: Indexing engine unreachable.|Diagnostic: |Type: System.Exception|Reference: An error occurred while invoking Void b__29().|Error Context : Change backup mode. IndexLocation: D:\EVIndexes\index2, BackupMode: True |Command Line: "C:\Program Files (x86)\Enterprise Vault\EVIndexAdminService.exe" -EntryID:1AEC1C7046560B94BB5B469E3D91306B91710000EVserver1|Application Domain: EVIndexAdminService.exe|Process Id: 4228|Thread Id: 2616|Stack Trace: at Symantec.EnterpriseVault.Indexing.Admin.EVIndexAdminUtils.ThrowExceptionIfNeeded(String IndexVolumesXml)| at Symantec.EnterpriseVault.Indexing.Admin.IndexAdminCOM.NotifyIVPForIndexVolumeAccessMode(String xmlIndexVolumeIDs, IndexingAccess accessMode)| at Symantec.EnterpriseVault.Indexing.Admin.IndexAdminCOM.ChangeIndexLocationBackupModeInternal(String indexLocationEntryId, Boolean inBackMode, Boolean logEvent)| at Symantec.EnterpriseVault.Indexing.Admin.EVIndexAdminUtils.InvokeHelper(Action act, String context, Boolean rethrow, Boolean shutdownCheck)

Cause

Corrupt index volume caused the Error: Indexing engine unreachable.

Solution

1. Run the SQL query below against the EnterpriseVaultDirectory database to identify the corrupt index volume (Id: 3)

Select A.ArchiveName, IRPE.IndexRootPath, IV.FolderName
From IndexVolume IV
INNER JOIN IndexRootPathEntry IRPE
ON IRPE.IndexRootPathEntryId = IV.IndexRootPathEntryId
INNER JOIN Archive A
ON A.RootIdentity = IV.RootIdentity
AND IV.RootIdentity = A.RootIdentity
Where IV.IndexVolumeIdentity = 3

Sample of the result:

ArchiveName      IndexRootPath             FolderName
TestArchive         G:\EVIndex\index3      1683A6B681B086C4AB671A9B39A0EAB8B_3

2. Rebuild the affected volume

Note: The volume will most likely show as "Normal" in the Index Volume Browser