How to obtain data to investigate Compliance Accelerator Hotword Analysis issues

Description

Enterprise Vault (EV) Compliance Accelerator (CA) from version 12.5 onwards and Veritas Advanced Supervision (VAS) for CA provide a new feature called Hotword Analysis. This feature filters items in Review based on existing Hotwords, lists the counts of Hotwords found in each item and lists the Hotword statistics found in the selected item.

There may be occasions where the Hotword Analysis of an item fails. Some possible causes are:

- EV Storage Services are hung or are not running.
- CA cannot contact the EV Storage Service servers.
- Complex Hotwords or phrases, e.g. a Hotword or phrase containing a large number of wildcards, and/or a large number of Hotword hits in the item, message or attachments being processed.
- Search terms are manually entered instead of using Hotwords and/or Hotword Sets. Hotwords are prefixed by HW, Hotwords Sets are prefixed by HWS (by default). Manually entered hotwords do not have any prefixes.

Crucial to any Hotword issue investigation is to obtain the original item and the Hotwords being processed for the item for analysis. This is the most important step! In order for Hotword failures to be analysed, the item (step 3 below) and Hotwords associated to the item (step 2 below) MUST be obtained and provided to Technical Support for internal testing and analysis.

Here are the steps to obtain more information on the failed items. If there are multiple failed items, obtain the information for as many items as possible. Once all data has been gathered, please contact technical support for assistance.

Optional Preparatory Step - Run a DTrace to capture the Hotword Analysis processing

Hotword Analysis occurs after a Search has been Accepted and its items are In Review. A DTrace to capture the Hotword Analysis processing can be run, however, it must be started before Accepting a Search. As it is not known which items may fail, it may not be feasible to run DTraces for each and every Search. If a Search is known to have failed items, a new Search can be run using the previous Search's Criteria and a DTrace run before Accepting the Search in an attempt to capture the data.

Note - Be sure to save the DTrace logs in a location that has adequate available free space as the logs can be fairly large when complete. The logs are text files and should compress well to smaller sizes for uploading to technical support, if needed.

1. Create a new Search using the Search having failed items as a template. To do this:

1.1. Create a new Immediate Search in the CA Department and select the existing Search in the Based on Search dropdown list.

1.2. Review the Search Criteria and verify it is the same as the Search having failed items.

1.3. Verify the Include items already in review option is selected.

1.4. Verify the Automatically accept search results option is not selected before running the Search.

2. Run the Search and allow it to complete. Compare the results with the Search having failed items. If the results are not the same, then Reject the Search and correct any Search Criteria as needed before re-running the Search (be sure to check the Archives selected for the Search).

3. Once the Search is ready to be Accepted, prepare the DTrace as follows:

3.1. On the CA server:

3.1.1. Log on to the CA Server as the Vault Service Account.

3.1.2. Open a Command Prompt and navigate to the EV installation folder.

3.1.3. Execute: DTrace 1000000

3.1.4. At the DT> prompt execute: v

3.1.5. Note the ID number beside the AcceleratorService64 and AnalyticsIngesterTask processes and enable the processes to be traced.

3.1.5.1. If the AcceleratorService64 is listed, at the DT> prompt execute: set <process ID #> v y [enter] (For example: set 3 v y)
If the AcceleratorService64 is not listed, at the DT> prompt execute: set AcceleratorService64 v y. Then execute Yes at the prompt: AcceleratorService64 - name not registered, register? [Yes/No] (No)>.

3.1.5.2. If the AnalyticsIngesterTask is listed, at the DT> prompt execute: set <process ID #> v y (For example: set 7 v y)
If the AnalyticsIngesterTask is not listed, at the DT> prompt execute: set AnalyticsIngesterTask v y. Then execute Yes at the prompt: AnalyticsIngesterTask - name not registered, register? [Yes/No] (No)>.

3.1.6. Add an Exclude Filter to remove unneeded background cache/polling entries as follows:

3.1.6.1. At the DT prompt execute: Filter

3.1.6.2. At the DT Filter prompt execute: Exclude "SQL Back Ground tasks"

3.1.6.3. At the DT Filter prompt execute: Exclude "Cache polling thread"

3.1.6.4. At the DT Filter prompt execute: Exclude CustomerCache

3.1.6.5. At the DT Filter prompt execute: Exclude UpdateTimeOut

3.1.6.6. At the DT Filter prompt execute: exit

3.1.7. Prepare DTrace logging by executing the following after replacing c:\CAdtrace.log with a location and filename (the log must have a .log extension) for the log but do not press Enter. At the DT> prompt type: log c:\CAdtrace.log

3.2. On each EV server that has the Storage Service that services the Archives being searched:

3.2.1. Log on to the EV Server as the Vault Service Account.

3.2.2. Open a Command Prompt and navigate to the EV installation folder.

3.2.3. Execute: DTrace 1000000

3.2.4. At the DT> prompt execute: v

3.2.5. Note the ID number beside the StorageOnlineOpns process and enable the process to be traced:

3.2.5.1. If the StorageOnlineOpns is listed, at the DT> prompt execute: set <process ID #> v y (For example: set 2 v y)

3.2.5.2. If the StorageOnlineOpns is not listed, at the DT> prompt execute: set StorageOnlineOpns v y. Then execute Yes at the prompt: StorageOnlineOpns - name not registered, register? [Yes/No] (No)>

3.2.6. Prepare DTrace logging by executing the following after replacing c:\EVdtrace.log with a location and filename (the log must have a .log extension) for the log but do not press Enter. At the DT> prompt type: log c:\EVdtrace.log

4. When all DTrace logs have been prepared, start the DTrace and Accept the Search:

4.1. Press Enter at the DT> log.c:\dtrace.log prompt on the CA server and all EV Storage servers to start the DTrace.  At the DT> prompt, leave the command line window open. The DTrace is now enabled. Let DTrace run as needed to capture the issue.

4.2. Accept the CA Search.

5. Allow the Hotword Analysis processing to complete for all items in the Search, then stop the DTrace:

5.1. Once the Hotword Analysis processing has completed, go to the DTrace command line window on the CA server and each EV Storage server. At the DT> prompt execute: quit

5.2. Respond accordingly to any additional prompts in order to save any logs and exit DTrace.

Step 1 - Preview the items

When items fail Hotword Analysis, the first test is to see if the items can be successfully Previewed in the Review Set. If there are Preview errors, such as Cannot retrieve item' this will prevent Hotword Analysis from completing successfully as such errors usually indicate an issue with EV Storage Service/server(s) connectivity and/or issues with the items in storage. Review any such errors and take the necessary actions to resolve the errors. Additional error information may be logged to the EV Event Logs on the CA server and EV Storage server(s).

If the items can be Previewed, even if the Preview takes an extended amount of time to be displayed, note the date and time the item was Previewed, along with the item's ID from the Item box at the top right corner. This information can be cross-referenced in the Client Logs (see below).

Step 2 - Obtain the Hotword Analysis Error Report and Hotwords

If there are no Preview errors and errors are seen during Hotword Analysis, these errors should be logged in the Hotword Analysis Error Report. The Hotword Analysis Errors tab is located next to the Archives tab under the Search Details of a Search. To obtain the report:

1. Find the Search in the Department or Research Folder and click on the Search to select it. May need to first select the In review filter under the Filter by status facet and click Refresh at the top of the Searches pane to see the Searches that are in Review. If there are a large number of Searches and/or a large number of Departments to review, the following SQL query will provide a list of Departments and Searches where Hotword Analysis may have errors:

1.1. Open SQL Server Management Studio (SSMS) with an account having rights to query the CA Customer database, such as the Vault Service Account (VSA).

1.2. Open a new query window focused on the CA Customer database and copy the following query into the new query window:

SELECT
tc.CaseID
, tc.Name AS 'Department/Research Folder'
, CASE WHEN (tc.FolderType = 330 AND tc.Type = 101) THEN 'Case'
WHEN (tc.FolderType = 330 AND tc.Type = 102) THEN 'Department'
WHEN tc.FolderType = 331 THEN 'Folder'
WHEN tc.FolderType = 332 THEN 'Hidden Folder'
ELSE 'Other' END AS 'FolderType'
, tis.SearchID
, tis.Name
, tis.RunDate
, CASE
WHEN tis.ApplicationSearchType = -1 AND ParentSearchID IS NULL THEN ts1.Name
WHEN tis.ApplicationSearchType = -1 AND ParentSearchID IS NOT NULL THEN ts1.Name + ', Parent SearchID: ' + CONVERT(nvarchar(25), tis.ParentSearchID)
WHEN tis.ApplicationSearchType != -1 THEN ts1.Name + ', ' + ts2.Name + ', Parent SearchID: ' + CONVERT(nvarchar(25), tis.ParentSearchID)
END AS 'Search Type'
, tis.NumHits
, tis.SampleSize
, tis.SampleResultSize
, [Search_HWAnalysisStatus_Name] = ts3.Name
, [Search_HWAnalysisStatus_Status] = ts3.ResourceID
FROM tblIntSearches AS tis
JOIN tblCase AS tc ON tis.CaseID = tc.CaseID
JOIN tblStatus AS ts1 ON tis.Type = ts1.StatusID
LEFT JOIN tblStatus AS ts2 ON tis.ApplicationSearchType = ts2.StatusID
JOIN tblStatus AS ts3 ON tis.HWAnalysisStatusID = ts3.StatusID
WHERE tis.HWAnalysisStatusID IN (1327 /*Failed*/, 1328 /*In Progress with errors*/, 1329 /*Completed with errors*/)
ORDER BY tc.Name, tis.SearchID;

1.3. Execute the query. The results can be saved to a spreadsheet for easier reviewing: Right-click in the results | Select All | Right-click in the results again | Copy With Headers | Paste into a new spreadsheet.

1.4. To view the Current Hotword Analysis status and item counts for a specific Search, use one of the following options:

1.4.1. If the Search name is known, edit the Search name as indicated at the top of the following query and execute the edited query against the CA Customer database:

DECLARE @SearchName nvarchar(100) = 'Search Name'; -- Edit Search name between the single quotes here
SELECT DISTINCT
tc.CaseID
, tc.Name AS 'Department/Research Folder'
, CASE
WHEN (tc.FolderType = 330 AND tc.Type = 101) THEN 'Case'
WHEN (tc.FolderType = 330 AND tc.Type = 102) THEN 'Department'
WHEN (tc.FolderType = 331 AND tc.Type = 101) THEN 'DA Folder'
WHEN (tc.FolderType = 332 AND tc.Type = 101) THEN 'DA Hidden Folder'
WHEN (tc.FolderType = 331 AND tc.Type = 102) THEN 'CA Folder'
WHEN (tc.FolderType = 332 AND tc.Type = 102) THEN 'CA Hidden Folder'
ELSE 'Other' END AS 'FolderType'
, tis.SearchID
, tis.Name
, tis.RunDate
, tis.HWAnalysisStatusID, ts1.Name AS 'Search_HWAnalysisStatus'
, tihas.HWAnalysisStatus, ts2.Name AS 'HWAnalysisStatus'
, COUNT(tihas.DiscoveredItemID) AS 'ItemCount'
FROM tblIntSearches AS tis
JOIN tblCase AS tc ON tis.CaseID = tc.CaseID
JOIN tblDiscoveredItemToSearch AS tdits WITH (NOLOCK) ON tis.SearchID = tdits.SearchID
LEFT JOIN tblIntHotwordAnalysisStatus AS tihas WITH (NOLOCK) ON tdits.DiscoveredItemID = tihas.DiscoveredItemID
JOIN tblStatus AS ts1 ON tis.HWAnalysisStatusID = ts1.StatusID
LEFT JOIN tblStatus AS ts2 ON tihas.HWAnalysisStatus = ts2.StatusID
WHERE tis.Name LIKE '%' + @SearchName + '%'
GROUP BY tc.CaseID, tc.Name, tc.FolderType, tc.Type, tis.SearchID, tis.Name, tis.RunDate, tis.HWAnalysisStatusID, ts1.Name, tihas.HWAnalysisStatus, ts2.Name
ORDER BY tis.SearchID, tihas.HWAnalysisStatus;

1.4.2. If the SearchID is known, edit the SearchID as indicated at the top of the following query and execute the edited query against the CA Customer database:

DECLARE @SearchID int = 1; -- Edit SearchID here
SELECT DISTINCT
tc.CaseID
, tc.Name AS 'Department/Research Folder'
, CASE
WHEN (tc.FolderType = 330 AND tc.Type = 101) THEN 'Case'
WHEN (tc.FolderType = 330 AND tc.Type = 102) THEN 'Department'
WHEN (tc.FolderType = 331 AND tc.Type = 101) THEN 'DA Folder'
WHEN (tc.FolderType = 332 AND tc.Type = 101) THEN 'DA Hidden Folder'
WHEN (tc.FolderType = 331 AND tc.Type = 102) THEN 'CA Folder'
WHEN (tc.FolderType = 332 AND tc.Type = 102) THEN 'CA Hidden Folder'
ELSE 'Other' END AS 'FolderType'
, tis.SearchID
, tis.Name
, tis.RunDate
, tis.HWAnalysisStatusID, ts1.Name AS 'Search_HWAnalysisStatus'
, tihas.HWAnalysisStatus, ts2.Name AS 'HWAnalysisStatus'
, COUNT(tihas.DiscoveredItemID) AS 'ItemCount'
FROM tblIntSearches AS tis
JOIN tblCase AS tc ON tis.CaseID = tc.CaseID
JOIN tblDiscoveredItemToSearch AS tdits WITH (NOLOCK) ON tis.SearchID = tdits.SearchID
LEFT JOIN tblIntHotwordAnalysisStatus AS tihas WITH (NOLOCK) ON tdits.DiscoveredItemID = tihas.DiscoveredItemID
JOIN tblStatus AS ts1 ON tis.HWAnalysisStatusID = ts1.StatusID
LEFT JOIN tblStatus AS ts2 ON tihas.HWAnalysisStatus = ts2.StatusID
WHERE tis.SearchID = @SearchID
GROUP BY tc.CaseID, tc.Name, tc.FolderType, tc.Type, tis.SearchID, tis.Name, tis.RunDate, tis.HWAnalysisStatusID, ts1.Name, tihas.HWAnalysisStatus, ts2.Name
ORDER BY tis.SearchID, tihas.HWAnalysisStatus;

1.4.3. If the Department's CaseiD is known, such as when using the Surveillance UI where the CaseID will be listed in the URL, edit the CaseID as indicated at the top of the following query and execute the edited query against the CA Customer database - the output will list all Searches with the newest at the top:

DECLARE @CaseID int = X; -- Edit CaseID here
SELECT DISTINCT
tc.CaseID
, tc.Name AS 'Department/Research Folder'
, CASE
WHEN (tc.FolderType = 330 AND tc.Type = 101) THEN 'Case'
WHEN (tc.FolderType = 330 AND tc.Type = 102) THEN 'Department'
WHEN (tc.FolderType = 331 AND tc.Type = 101) THEN 'DA Folder'
WHEN (tc.FolderType = 332 AND tc.Type = 101) THEN 'DA Hidden Folder'
WHEN (tc.FolderType = 331 AND tc.Type = 102) THEN 'CA Folder'
WHEN (tc.FolderType = 332 AND tc.Type = 102) THEN 'CA Hidden Folder'
ELSE 'Other' END AS 'FolderType'
, tis.SearchID
, tis.Name
, tis.RunDate
, tis.HWAnalysisStatusID, ts1.Name AS 'Search_HWAnalysisStatus'
, tihas.HWAnalysisStatus, ts2.Name AS 'HWAnalysisStatus'
, COUNT(tihas.DiscoveredItemID) AS 'ItemCount'
FROM tblIntSearches AS tis
JOIN tblCase AS tc ON tis.CaseID = tc.CaseID
JOIN tblDiscoveredItemToSearch AS tdits WITH (NOLOCK) ON tis.SearchID = tdits.SearchID
LEFT JOIN tblIntHotwordAnalysisStatus AS tihas WITH (NOLOCK) ON tdits.DiscoveredItemID = tihas.DiscoveredItemID
JOIN tblStatus AS ts1 ON tis.HWAnalysisStatusID = ts1.StatusID
LEFT JOIN tblStatus AS ts2 ON tihas.HWAnalysisStatus = ts2.StatusID
WHERE tis.CaseID = @CaseID
AND (tis.StatusID NOT IN (4 /*Rejected*/, 5 /*Deleting*/, 8 /*Abandon*/, 858 /*Deleted*/)
AND tis.Type NOT IN (112 /*Schedule*/, 600 /*Guaranteed Sample*/))
GROUP BY tc.CaseID, tc.Name, tc.FolderType, tc.Type, tis.SearchID, tis.Name, tis.RunDate, tis.HWAnalysisStatusID, ts1.Name, tihas.HWAnalysisStatus, ts2.Name
ORDER BY tis.SearchID DESC, tihas.HWAnalysisStatus;

1.4.4. To list all Searches with Hotword Analysis Status Pending, execute the following query against the CA Customer database - the output will list the Searches with the newest at the top:

DECLARE @HWAnalysisStatusID int = (SELECT StatusID FROM tblStatus WHERE Type = 'SRCH_ANALYSIS_STATUS' AND Name = 'Pending');
SELECT DISTINCT
tc.CaseID
, tc.Name AS 'Department/Research Folder'
, CASE
WHEN (tc.FolderType = 330 AND tc.Type = 101) THEN 'Case'
WHEN (tc.FolderType = 330 AND tc.Type = 102) THEN 'Department'
WHEN (tc.FolderType = 331 AND tc.Type = 101) THEN 'DA Folder'
WHEN (tc.FolderType = 332 AND tc.Type = 101) THEN 'DA Hidden Folder'
WHEN (tc.FolderType = 331 AND tc.Type = 102) THEN 'CA Folder'
WHEN (tc.FolderType = 332 AND tc.Type = 102) THEN 'CA Hidden Folder'
ELSE 'Other' END AS 'FolderType'
, tis.SearchID
, tis.Name
, tis.RunDate
, tis.HWAnalysisStatusID, ts1.Name AS 'Search_HWAnalysisStatus'
, tihas.HWAnalysisStatus, ts2.Name AS 'HWAnalysisStatus'
, COUNT(tihas.DiscoveredItemID) AS 'ItemCount'
FROM tblIntSearches AS tis
JOIN tblCase AS tc ON tis.CaseID = tc.CaseID
JOIN tblDiscoveredItemToSearch AS tdits WITH (NOLOCK) ON tis.SearchID = tdits.SearchID
LEFT JOIN tblIntHotwordAnalysisStatus AS tihas WITH (NOLOCK) ON tdits.DiscoveredItemID = tihas.DiscoveredItemID
JOIN tblStatus AS ts1 ON tis.HWAnalysisStatusID = ts1.StatusID
LEFT JOIN tblStatus AS ts2 ON tihas.HWAnalysisStatus = ts2.StatusID
WHERE tis.HWAnalysisStatusID = @HWAnalysisStatusID
GROUP BY tc.CaseID, tc.Name, tc.FolderType, tc.Type, tis.SearchID, tis.Name, tis.RunDate, tis.HWAnalysisStatusID, ts1.Name, tihas.HWAnalysisStatus, ts2.Name
ORDER BY tc.CaseID, tis.SearchID DESC, tihas.HWAnalysisStatus;

2. Click on the Hotword Analysis Errors tab in the Search Details pane (next to the Archives tab).

3. Click on the Download All Errors button (at the right of the tab). Provide a location and filename for the csv file and click Save.

The report should list the DiscoveredItemIDs (ItemID) and the error information. Additionally, an item's DiscoveredItemID (ItemID) can be found by clicking on the item in the Review Set to Preview the item and then obtain the ID from the Item field at the top right of the Preview pane.

4. The list of Hotwords being processed for a specific DiscoveredItemID can be determined by editing the DiscoveredItemID in the following SQL query and running the query against the CA Customer database. The output is best viewed in a spreadsheet.

4.1. Open SQL Server Management Studio (SSMS) with an account having rights to query the CA Customer database, such as the Vault Service Account (VSA).

4.2. Open a new query window focused on the CA Customer database and copy the following query into the new query window:

DECLARE @DiscoveredItemID bigint = 1; -- Edit DiscoveredItemID here
SELECT
tihs.DiscoveredItemID, 
tidi.KVSSavesetID, 
CASE tidi.CaptureType
WHEN 1 THEN 'Search'
WHEN 2 THEN 'Random Sampling'
WHEN 3 THEN 'Ad-hoc Search'
WHEN 4 THEN 'Research Folder Search'
WHEN 6 THEN 'Guaranteed Sample Search'
WHEN 10 THEN 'SearchDuplicate'
WHEN 99 THEN 'Policy'
ELSE 'Not Defined' END AS 'Capture Type',
CASE tidi.Type
WHEN 1 THEN 'Email'
WHEN 2 THEN 'IM'
WHEN 3 THEN 'Bloomberg'
WHEN 4 THEN 'Fax'
WHEN 5 THEN 'Domino'
WHEN 6 THEN 'FSA'
WHEN 7 THEN 'SMTP'
WHEN 8 THEN 'SharePoint'
WHEN 9 THEN 'Social'
WHEN 10 THEN 'IMAP'
ELSE 'Not Defined' END AS 'Message Type', 
tihs.HotwordID,
th.WordText,
tihs.TotalOccurrences, tihs.SubjectCount, tihs.ContentCount, tihs.AttachmentHWXML,
ISNULL(CONVERT(nvarchar(25), tihas.HWAnalysisStatus), '') AS HWAnalysisStatus, ISNULL(ts1.Name, '') AS HW_Analysis_Status,  ts1.ResourceID AS HW_Analysis_Status_Description,
ISNULL(CONVERT(nvarchar(25), tihas.HWAnalysisErrorCode), '') AS HWAnalysisErrorCode, ISNULL(tihas.HWAnalysisError, '') AS HWAnalysisError, 
ISNULL(CONVERT(nvarchar(25), tihas.RetryCount), '') AS RetryCount, ISNULL(CONVERT(nvarchar(100), tihas.LastUpdated, 121), '') AS LastUpdated
FROM tblIntHotwordStatistics AS tihs (NOLOCK)
LEFT JOIN tblIntHotwordAnalysisStatus AS tihas (NOLOCK) ON tihs.DiscoveredItemID = tihas.DiscoveredItemID
LEFT JOIN tblStatus AS ts1 ON tihas.HWAnalysisStatus = ts1.StatusID
JOIN tblIntDiscoveredItems AS tidi (NOLOCK) ON tihs.DiscoveredItemID = tidi.DiscoveredItemID
JOIN tblHotword AS th (NOLOCK) ON tihs.HotwordID = th.HotwordID
WHERE tihs.DiscoveredItemID = @DiscoveredItemID
ORDER BY tihs.DiscoveredItemID, tihs.TotalOccurrences DESC, tihs.ContentCount DESC, tihs.SubjectCount DESC;

4.3. Edit the DiscoveredItemID as indicated in the query using the DiscoveredItemIDs (ItemID) from the .csv file and then execute the query.

4.4. Right-click in the results | Select All | Right-click in the results again | Copy With Headers | Paste into a new spreadsheet in Excel.

Note - Searches with failed Hotword Analysis items can be reprocessed. To do this, go to the Search containing the failed items, click on the Hotword Analysis Errors tab under Search Details, then click on Retry Failed Items. It may be best to reprocess a few Searches at a time, instead of reprocessing a large number of Searches concurrently.

Step 3 - Obtain the item's Archive information and the Original Item

Items can be viewed in the Review Set by clicking on their entries in the Item List pane or by entering the DiscoveredItemID into the Item field and pressing Enter.

1. To obtain the item's Archive information, right-click the item in the Review Set's Item List pane, click on Copy item details to clipboard, then paste that data in the clipboard into a text file. The output should contain the following information:

ID    From    Subject/Filename    Date    Status    Vault    Saveset
X    <Author>    <Subject>    <Item date>    <Item Review status>    <VaultID or ArchiveID>    <SavesetID>

Here is an example output:

ID    From    Subject/Filename    Date    Status    Vault    Saveset
1234    user1    Board Meeting Report    1/10/2022 8:30:00 AM    Unreviewed    123ABC12345A67890B12345CD1234567E1110000EV    202201104716959~202201100830000000~Z~E05A70DD824818BB6EDD9643A3356D71

In this example the DiscoveredItemID is 1234, ArchiveID is 123ABC12345A67890B12345CD1234567E1110000EV and the SavesetID is 202201104716959~202201100830000000~Z~E05A70DD824818BB6EDD9643A3356D71.

2. The original item can be downloaded from the Review Set by clicking on the View Original Item button (presuming there are no Storage Service/server issues).

Step 4 - Obtain the Client Log

The Client log contains information processed by the Client during use. The log is updated on Client startup, when certain functions are processed by the Client, and on Client shutdown. All entries use the Universal Time Coordinate (UTC) date and time. The log may be rolled over to a new file when the size reaches approximately 20 MB. The rollover files will be sequentially numbered using the naming convention CustomTraceFile_X.log.

The Client log can be found under the user's profile, as follows:

1. Click on Start | type run | press Enter to open the Run dialog box.

2. Enter the following into the Run dialog box and press Enter: %userprofile%\AppData\Local\Symantec Corporation\Enterprise Vault Accelerators\Compliance_Client\Logs\

Step 5 - Review the Event Logs for any Hotword Analysis timeouts (for CA 14.1.2 and higher, and 14.2 and higher)

If there are a large number of Hotwords being processed against an item where the item and/or its attachments contain a large number of Hotwords, there is a possibility the Hotword Analysis engine may timeout on rare occasions. When such a timeout occurs, the engine will retry the processing using a different processing method for a specific timeout period (5 seconds by default) and for a specific number of retry attempts (3 attempts by default).

Once the timeouts and retry attempts have been exceeded, the processing will fail the message and may log one or both of the following Information Event Log entries in the EV Event Logs on the CA server:

Log Name:      Veritas Enterprise Vault
Source:        Accelerator Manager 
Event ID:      600
Task Category: None
Level:         Information
Keywords:      Classic
Description:
APP Analytics - Regular expression engine was timed out while trying to match following hotwords to the content of ItemID - A: X, Y, Z

V-437-600

Log Name:      Veritas Enterprise Vault
Source:        Accelerator Manager 
Event ID:      601
Task Category: None
Level:         Information
Keywords:      Classic
Description:
APP Analytics - Regular expression engine was timed out while trying to match following hotwords to the content of attachment(ID - B) of ItemID - A: X, Y, Z

V-437-601

Event ID 600 lists the ItemID of the item or message whose content is being processed and lists the HotwordIDs of any Hotwords that have caused the Hotword Hit Analysis to exceed the timeout and retry counts. Similarly, Event ID 601 lists the ItemID of the item or message whose attachments are being processed and lists the HotwordIDs of any Hotwords that have caused the Hotword Hit Analysis to exceed the timeout and retry counts. Here is the ID legend from the above Events:

A: The (Discovered)ItemID of the item or message being processed.
B: The attachment ID of the attachment in the item or message being processed.
X, Y, Z: The HotwordID(s) of the Hotword(s) being processed.

1. Follow the steps above to obtain the item's Archive information and the original item.

2. To list the Hotwords that were being processed for the item or message:

2.1. Open SQL Server Management Studio (SSMS) with an account having rights to query the CA Customer database, such as the Vault Service Account (VSA).

2.2. Open a new query window focused on the CA Customer database and copy the following query into the new query window:

SELECT HotwordID, WordText
FROM tblHotword
WHERE HotwordID IN (
-- List HotwordIDs in a comma separated list below
1, 2, 3
-- List HotwordIDs in a comma separated list above
)
ORDER BY WordText, HotwordID;

2.3. Edit the HotwordIDs as indicated in the query using the HotwordIDs from the Event Log entry and then execute the query. The HotwordIDs should be entered in a comma-separated list. If there is only 1 HotwordID, simply enter that HotwordID.

2.4. The results will look like this, with one line per HotwordID:

HotwordID    WordText
<HotwordID>    <Hotword or phrase>
<HotwordID>    <Hotword or phrase>
<HotwordID>    <Hotword or phrase>
. . .