Post upgrade Event ID 40966 is logged and Indexing tasks fail

Problem

After the upgrade to Enterprise Vault 10.0.1 (or newer releases) it is not possible to properly execute indexing upgrade, synchronize and rebuild tasks, newly archived items are not indexed; the new tasks created stay in 'inactive' state even if the 'Index Administration Task' is scheduled to run.

Error Message

Trying to stop any 'Upgrade', 'Synchronize' and 'Rebuild' tasks returns the error:

"The subtask for '<User archive>' could not be stopped: Reason: 0x8013150b"

or

Generating the following error in the event viewer:

Event ID: 40966
Task Category: Index Admin Service
Level: Error
Description: A program fault has raised an exception.

Exception: Failed to get subtask for subtask ID 17D2DB9C8E05D49F6B71ECED6CBC3E98613b00ev1
Diagnostic:
Type: System.ServiceModel.FaultException`1[[Symantec.EnterpriseVault.Indexing.Common.CommunicationFault, EVSharedManagedInterfaces, Version=10.0.3.0, Culture=neutral, PublicKeyToken=26c5e2ccf2b9267c]]
Reference: An error occurred while invoking Void <StopSubTask>b__f().
Error Context : Stopping sub task for TaskEntryId 12DD149F8BC2262478807BEF2FAE284741013a00ev1 and SubTaskEntryId 17D2DB9C8E05D49F6B71ECED6CBC3E9861013b00ev1

Command Line: "C:\Program Files(x86)\Enterprise Vault\EVIndexAdminService.exe" -EntryID:1410C9C6F53A8DE4D932D9087D28C4C841710000ev1
Application Domain: EVIndexAdminService.exe
Process Id: 2700
Thread Id: 5212
 V-437-40966
 

DTRACE Information:

To validate if this is the issue a DTRACE is needed, please perform the following steps on the EV server that has generated Event ID 40966 while re-creating the issue.

• DTRACE the process EVIndexQueryServer for no less than 5 minutes while re-creating the issue.
• Open the log file created and search for either of the following the strings:

(EVIndexQueryServer) <6140> EV-H {INDEXQUERYSERVERAPPLICATION.EN_US} Innermost exception from previous error: System.IO.FileLoadException: Retrieving the COM class factory for component with CLSID {BD7A3509-B3F3-42FC-8FFE-FE8D1CCFE013} failed due to the following error: 8007045a.|   at BuildUp_KVS.EnterpriseVault.Interop.AuditWrapperClass(IBuilderContext )|   at Microsoft.Practices.ObjectBuilder2.BuildPlanStrategy.PreBuildUp(IBuilderContext context) at Microsoft.Practices.ObjectBuilder2.StrategyChain.ExecuteBuildUp(IBuilderContext context)
 

The CLSID {BD7A3509-B3F3-42FC-8FFE-FE8D1CCFE013} corresponds to the C:\Program Files(x86)\Enterprise Vault\x64\AuditWrapper.dll; a manual AuditWrapper.dll re-registration ("regsvr32.exe C:\Program Files(x86)\Enterprise Vault\x64\AuditWrapper.dll") fails like:

Cause

The CLSID {BD7A3509-B3F3-42FC-8FFE-FE8D1CCFE013} cannot be properly initialized because the following path "HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\FIPS" has not been created during the EV binaries installation

A working Enterprise Vault (FIPS compliant DOC4820) installation must have the FIPS registry key in the following paths:

HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\FIPS

"InstallPath=C:\Program Files(x86)\Enterprise Vault\x64"

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\KVS\Enterprise Vault\FIPS

"InstallPath=C:\Program Files(x86)\Enterprise Vault\"

 
Workaround:
Manually create the missing registry path in order to facilitate the process using the text files attached at the bottom of this article.

Solution

Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry. Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and workstation be made prior to making any registry changes.

1. Download the proper version from the attachment list (FIPS_EV101.txt etc)
2. Copy the file onto each affected Enterprise Vault server (C:\temp\)
3. Edit the file in order to adjust the install path if different from c:\program files (x86)\enterprise vault\x64 (please note that the value should be like C:\\Program Files(x86)\\Enterprise Vault\\x64"
4. Change the file extension from .txt to .reg (from FIPS_EV101.txt to FIPSEV_101.reg)
5. Import the files into registry
6. Execute the batch C:\Program Files(x86)\EnterpriseVault\FileReRegister.bat (this will make sure that all the DLLs will be properly re-registered)
7. Restart all the Enterprise Vault services by restarting the Enterprise Vault Admin Service
8. Open the 'Monitor Indexes Tasks' and check if the tasks are properly running