Users may receive the message "Search request failed" when accessing Enterprise Vault Search in Enterprise Vault 11.0 or 11.0.1

Problem

In rare circumstances, users may receive the message Search request failed when they try to access Enterprise Vault Search in Enterprise Vault (EV) version 11.0 or 11.0.1.

DO NOT USE THIS ARTICLE FOR EV 12.0.0 OR GREATER

Error Message

 WINHTTP Logging shows

Frame: Number = 20589, Captured Frame Length = 117, MediaType = NetEvent
+ NetEvent:
- MicrosoftWindowsWinsockAFD: connection aborted: 2 (0x2): Process 0x0000000014FFD5B0, Endpoint 0x0000000014C7EE50, Seq 8003 (0x1F43), Reason Abortive disconnect requested on endpoint
- AFD_EVENT_ABORT: connection aborted: 2 (0x2): Process 0x0000000014FFD5B0, Endpoint 0x0000000014C7EE50, Seq 8003 (0x1F43), Reason Abortive disconnect requested on endpoint
EnterExit: 2 (0x2)
Location: 8003 (0x1F43)
+ Process: 0x0000000014FFD5B0
+ Endpoint: 0x0000000014C7EE50
Reason: Abortive disconnect requested on endpoint

Some or all of the following Event ID's below will be logged in the EV and Application event logs:

Log Name: Symantec Enterprise Vault
Source: Enterprise Vault Event
ID: 40966
Level: Error
Description: A program fault has raised an exception.
Exception: The message could not be dispatched because the service at the endpoint address
'net.tcp://localhost/EnterpriseVaultAPI/Search/SOAPSearch.svc' is unavailable for the protocol of the address.
V-437-40966

Log Name: Symantec Enterprise Vault
Source: Enterprise Vault
Event ID: 41520
Task Category: Enterprise Vault Search
Level: Error
Keywords: Classic
Description:
Unable to contact the Search API.
Reason: The message could not be dispatched because the service at the endpoint address 'net.tcp://localhost/EnterpriseVaultAPI/Search/SOAPSearch.svc' is unavailable for the protocol of the address.
V-437-41520

Log Name: Application
Source: System.ServiceModel 3.0.0.0
Event ID: 4
Task Category: WebHost
Level: Error
Keywords: Classic
User: IIS APPPOOL\EnterpriseVaultAPI
Description:
An error occurred while trying to listen for the URL '/LM/W3SVC/1/ROOT/EnterpriseVaultAPI/Search'. This worker process will be terminated. Sender Information: net.tcp Exception: System.ServiceModel.WasHosting.TcpAppDomainProtocolHandler/12892885 Process Name: System.ServiceModel.CommunicationException: The TransportManager failed to listen on the supplied URI using the NetTcpPortSharing service: .
at System.ServiceModel.Channels.SharedConnectionListener.SharedListenerProxy.Register()
at System.ServiceModel.Channels.SharedConnectionListener.SharedListenerProxy.Open(Boolean isReconnecting)
at System.ServiceModel.Channels.SharedConnectionListener.StartListen(Boolean isReconnecting)

Log Name: Application
Source: ASP.NET 2.0.50727.0
Event ID: 1334
Task Category: None
Level: Error
Keywords: Classic
Description:
An error occurred while trying to start an app domain protocol listener channel.
Exception: System.ServiceModel.CommunicationException
Message: The TransportManager failed to listen on the supplied URI using the NetTcpPortSharing service: .
StackTrace: at System.ServiceModel.Channels.SharedConnectionListener.SharedListenerProxy.Register()
at System.ServiceModel.Channels.SharedConnectionListener.SharedListenerProxy.Open(Boolean isReconnecting)
at System.ServiceModel.Channels.SharedConnectionListener.StartListen(Boolean isReconnecting)
at System.ServiceModel.Channels.SharedTcpTransportManager.OnOpenInternal(Int32 queueId, Guid token)
at System.ServiceModel.Activation.HostedTcpTransportManager.Start(Int32


Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault 
Event ID:      41177
Task Category: None
Level:         Error
Keywords:      Classic
Description:
Enterprise Vault could not authenticate a Domino user (group resolution failed). The user cannot access personal archives (access denied).
User name: CN=MYRON BOLITAR/O=VERITAS@VERITAS
Domain: VERITAS
Error: Invalid password


Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault 
Event ID:      41315
Task Category: Index Query Server
Level:         Warning
Keywords:      Classic
Description:
A search failed with error "Error in the application.".
Archive: Myron Bolitar/Veritas
Archive ID: 16974A31F52DB584CA3E3AF091DEE3A461110000evserver1.EV.LOCAL
Index volume ID: 27
User: EV@Anon\CN=Myron Bolitar/O=Veritas@Veritas@VERITAS
Maximum results: 100
Timeout (seconds): 600
Sort order: -date 


Log Name:      Symantec Enterprise Vault
Source:        Enterprise Vault 
Event ID:      41059
Task Category: None
Level:         Error
Keywords:      Classic
Description:
Task: 'Directory Service'
Unable to switch to Lotus Id file 'c:\Program Files (x86)\IBM\Notes\Data\dadmin.id' 

Solution

Note for Domino Environments:  We should check first that the new Enterprise Vault search is working by going to the Vault Admin Console, give permissions to an archive for Vault Service Account and then go into search.  If this works then the issue likely resides outside of Enterprise Vault like user ID issue or Domino SSO configuration for example.  Also, if Indexing service is on a different server other than EVDG check Enterprise Vault events on that server for 41059, 41177, and 41315 event IDs.  If you see these events the issue is with the NoteID.  The VSA account uses a NoteID file to access the Domino resources and this must be on all EV/EVDG servers.  These files are like keys that can access Domino resources.  So if any changes are made to the NoteID on one EV server the changes must be made to the NoteID on all other EV/EVDG servers.  For example, the admin changes the password of the NoteID file on the EVDG server and then changes the password in Enterprise Vault Console so that Enterprise Vault processes know the new password to use.  But the ID file located on another EV server has older password.  What will happen on that server is that Enterprise Vault will use the new password with a NoteId still configured with the old password and this will not work.  If changes to this NoteID are required, it is best to make the changes on one server and then copy and replace the NoteID file over to all other EV/EVDG servers so they are the same throughout the environment.

Note: This procedure is for Enterprise Vault 11.0 and 11.0.1 only. It must not be used with later releases of Enterprise Vault.

To resolve this issue, work through the following steps on the Enterprise Vault 11.0 or 11.0.1 server until the error is no longer reproduced:

1. In the Services Microsoft Management Console (MMC) snap-in, restart the Net.Tcp Listener Adapter.
2. In the Services MMC snap-in, restart the Net.Tcp Port Sharing Service.

3. Repair the WCF and WF components.

   On Windows Server 2008, use the ServiceModel Registration Tool ( ServiceModelReg.exe ) as follows:

   1. Open a Command Prompt window.
   2. Change to the folder %SystemRoot%\Microsoft.Net\Framework\v3.0\Windows Communication Foundation .

   3. Type the following command:

      ServiceModelReg.exe -r

      For more information on the ServiceModel Registration Tool, see the following Microsoft Knowledge Base article:

      https://msdn.microsoft.com/en-us/library/ms732012(v=vs.85).aspx

   On Windows Server 2012 , remove and add .NET Framework features as follows:
   Note: it has been found that the steps below also work on a Windows 2008 server and have resolved the issue.

   1. Start the Remove Roles and Features wizard.
   2. On the Features page, under .NET Framework 3.5 Features, clear the check boxes for HTTP Activation and Non-HTTP Activation.
   3. Click Next to the end of the wizard, and then click Remove. The features are removed.
   4. Reinstall these features using the Add Roles and Features wizard.

   Alternatively, on Windows Server 2012, you can use the Deployment Image Servicing and Management (DISM) command line tool to disable and enable the .NET Framework 3.5 features.

4. Follow the instructions in this Microsoft Knowledge Base article to repair the protocol handlers:

   https://support.microsoft.com/kb/2803161

5. Restart the Enterprise Vault server.