Review Set message 'Hotword hits may be incomplete because hotword analysis is in progress for the review set' does not change

Problem

After Accepting a Search in Enterprise Vault (EV) Compliance Accelerator (CA) with Hotword Analysis enabled, the analysis status either does not start or remains as Pending. When reviewing the items from the Search in the Review Set, the message below is displayed. The EV Event Logs on the CA server may list Event ID 41619 referencing transient errors as below. Running a DTrace of the AnalyticsIngesterTask process to capture the Hotword Analysis lists the exceptions as below.

Error Message

Warning message displayed in the Review Set:

Hotword hits may be incomplete because hotword analysis is in progress for the review set. Reapply filter to see the updated hotword hits.

EV Event Log entry on the CA server:

Event ID:    41619
Log:    Veritas Enterprise Vault
Event Type:    Information
Source:    Enterprise Vault
Description:
A transient error is preventing the execution of a SQL command. Enterprise Vault will try to run the command again.
Command: 'EVSqlConnection: DB: Data Source=.;Initial Catalog=EVConfiguration;Integrated Security=True;Max Pool Size=200;Connect Timeout=120;Load Balance Timeout=30'
Attempt: X of 40.
Last Error: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) 

V-437-41619

Exceptions/errors seen in a DTrace of the AnalyticsIngesterTask process:

[1234]    (AnalyticsIngesterTask)    <CustomerSync:5678>    EV-L    {EVSqlBase.ShouldExcludeSqlException} Error 'A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)' is transient. 'EVSqlConnection: DB: Data Source=.;Initial Catalog=EVConfiguration;Integrated Security=True;Max Pool Size=200;Connect Timeout=120;Load Balance Timeout=30' will be retried

[1234]    (AnalyticsIngesterTask)    <CustomerSync:5678>    EV-L    {EVSqlBase.HandleException} Will retry to execute 'EVSqlConnection: DB: Data Source=.;Initial Catalog=EVConfiguration;Integrated Security=True;Max Pool Size=200;Connect Timeout=120;Load Balance Timeout=30' after Xms - Attempt X

[1234]    (AnalyticsIngesterTask)    <CustomerSync:5678>    EV-L    {EVSqlBase.DoCheckConnection} Executing 'EVSqlConnection: DB: Data Source=.;Initial Catalog=EVConfiguration;Integrated Security=True;Max Pool Size=200;Connect Timeout=120;Load Balance Timeout=30' Attempt X

[1234]    (AnalyticsIngesterTask)    <CustomerSync:5678>    EV-M    {EVSqlBase.ShouldRetry} Error 'A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)'. The maximum number of '40' retries has been reached . 'EVSqlConnection: DB: Data Source=.;Initial Catalog=EVConfiguration;Integrated Security=True;Max Pool Size=200;Connect Timeout=120;Load Balance Timeout=30' will not be retried

[1234]    (AnalyticsIngesterTask)    <CustomerSync:5678>    EV-H    {EVSqlBase.LogFailureAndThrow} Failed to execute 'EVSqlConnection: DB: Data Source=.;Initial Catalog=EVConfiguration;Integrated Security=True;Max Pool Size=200;Connect Timeout=120;Load Balance Timeout=30' after Xms and 40 retries - Error Error Code: 2 / 80131904 - Severity 20 - A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)

[1234]    (AnalyticsIngesterTask)    <CustomerSync:5678>    EV-H    {-} Exception: Error Common_InitializingCustomers Info:{AcceleratorEvent} {CX} An error has occured when initializing the Customers. System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) ---> System.ComponentModel.Win32Exception (0x80004005): The system cannot find the file specified
|   at Symantec.EnterpriseVault.DatabaseAccess.EVSqlBase.Do[T](Func`1 action)
|   at Symantec.EnterpriseVault.DatabaseAccess.EVSqlBase.Do(Action action)
|   at Symantec.EnterpriseVault.DatabaseAccess.EVSqlConnection.Open()
|   at KVS.Accelerator.Configuration.Customer.GetCustomerDS(Int32 CustomerID, String VirtualDirectory, String Version, Int32 ServerID, CustomerState State)
|ClientConnectionId:00000000-0000-0000-0000-000000000000
|Error Number:2,State:0,Class:20 Diag: Type:System.Exception ST: Inner:None

[1234]    (AnalyticsIngesterTask)    <CustomerSync:5678>    EV-H    {-} Exception: Error Common_TaskModuleSynchronizingThread Info:{AcceleratorEvent} {CX} Error in  customers synchronising thread. System.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) ---> System.ComponentModel.Win32Exception (0x80004005): The system cannot find the file specified
|   at Symantec.EnterpriseVault.DatabaseAccess.EVSqlBase.Do[T](Func`1 action)
|   at Symantec.EnterpriseVault.DatabaseAccess.EVSqlBase.Do(Action action)
|   at Symantec.EnterpriseVault.DatabaseAccess.EVSqlConnection.Open()
|   at KVS.Accelerator.Configuration.Customer.GetCustomerDS(Int32 CustomerID, String VirtualDirectory, String Version, Int32 ServerID, CustomerState State)
|   at KVS.Accelerator.Configuration.Customer.GetCustomerDS(Int32 CustomerID)
|   at KVS.Accelerator.Common.TasksModuleInfo.DoUpdateCustomer()
|   at KVS.Accelerator.Common.TasksModuleInfo.UpdateCustomer()
|ClientConnectionId:00000000-0000-0000-0000-000000000000
|Error Number:2,State:0,Class:20 Diag: Type:System.Exception ST: Inner:None

[1234]    (AnalyticsIngesterTask)    <CustomerSync:5678>    EV~I    
|Event ID: 41619 A transient error is preventing the execution of a SQL command. Enterprise Vault will try to run the command again.
|Command: 'EVSqlConnection: DB: Data Source=.;Initial Catalog=EVConfiguration;Integrated Security=True;Max Pool Size=200;Connect Timeout=120;Load Balance Timeout=30'
|Attempt: X of 40.
|Last Error: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) |

Cause

The Hotword Analysis process needs to communicate with the SQL server to perform the analysis and does so using the SQL server connection as defined in the configuration files relating to Hotword Analysis. This is the default DSN line for each of the files which does not list any SQL server information and lists the default CA Configuration database:

<add key="DSNConfiguration" value="server=.;Integrated Security=true;Initial Catalog='EVConfiguration';Connection Timeout=100;Connection Lifetime=30;Max Pool Size=200" />

In the Event Log and DTrace entries above, the SQL server connection in not defined:

- This indicates no SQL server is listed (the dot indicates no entry is present): Data Source=.
- This indicates the Configuration database listed is the default name as provided when the Accelerator is installed: Initial Catalog=EVConfiguration

The issue is usually caused when moving CA to a new server and not copying over the required configuration files. As the Hotword Analysis configuration lists the default values, the processing attempts to connect to a non-existent SQL server and, thus, fails to perform the processing.

Solution

To fix the issue, the following files will need to have the correct SQL server and Configuration database information listed:

- AnalyticsConversationAnalyserTask.exe.config
- AnalyticsIngesterTask.exe.config
- AnalyticsServerApp.exe.config

1. Review the DSNConfiguration line in the AcceleratorManager.exe.config and AcceleratorService.exe.config files in the CA Server installation folder (\Program Files (x86)\Enterprise Vault Business Accelerator by default) and note the SQL server name (can include an instance and/or port number) and Configuration database information that is in use. This information will be needed on later steps.

2. Stop the Enterprise Vault Accelerator Manager Service (EVAMS) on the CA server.

3. Find the AnalyticsConversationAnalyserTask.exe.config, AnalyticsIngesterTask.exe.config and AnalyticsServerApp.exe.config files in the CA Server installation folder (\Program Files (x86)\Enterprise Vault Business Accelerator by default) that will need to have their DSNConfiguration lines edited and make a copy of them in case the changes need to be reverted.

4. Update the AnalyticsConversationAnalyserTask.exe.config, AnalyticsIngesterTask.exe.config and AnalyticsServerApp.exe.config files:

4.1. If the files are available on the previous CA server and the previous and current CA servers are the same version:

4.1.1. Copy the AnalyticsConversationAnalyserTask.exe.config, AnalyticsIngesterTask.exe.config and AnalyticsServerApp.exe.config files from the previous CA server.

4.1.2. Replace the existing files in the new CA server with the copied files from the previous CA server.

4.2. If the files are not available on the previous CA server or the previous and current CA servers are not the same version:

4.2.1. Edit the DSNConfiguration line in the AnalyticsConversationAnalyserTask.exe.config, AnalyticsIngesterTask.exe.config and AnalyticsServerApp.exe.config files on the new CA server using Notepad to add the correct SQL server name (can include an instance and/or port number) and Configuration database name as obtained above:

<add key="DSNConfiguration" value="server='<SQL_server_name>';Integrated Security=true;Initial Catalog='<Accelerator_Configuration_database_name>';Connection Timeout=100;Connection Lifetime=30;Max Pool Size=200" />

5. Start the EVAMS and monitor the EV Event Logs on the Accelerator server for any issues. If no issues seen in the Event Logs post EVAMS start, monitor the Hotword Analysis in the Review Set.