Compliance Accelerator is not completing the Random Sampling processing.

Problem

Compliance Accelerator is not completing the Random Sampling processing and is failing on step 6 with an 'Out Of Memory' error.

Error Message

- Error from Dtrace of the AcceleratorService on the Compliance Accelerator server:

(AcceleratorService) <Sampling Thread Num X:1234> EV-H {-} Exception: Error Sampling_Guaranteed_GuaranteSampling Info: {ACCELERATOREVENT.EN_US} {C1.EN_US} Guaranteed Sampling: Error during guaranteed sampling - aborting. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

Cause

The issue is a result of having a larger number of items available for Random Sampling than the processing can handle. Items available for Random Sampling are loaded into memory on the CA server and then processed for each Monitored Employee in each Department. This is handled by the 32-bit AcceleratorService process, which is limited to 2 GB of memory. If there are a large number of items and/or a large number of Monitored Employees per Department, the processing could require more memory than is available and/or that can be allocated, thus logging the System.OutOfMemoryException error resulting the Random Sampling process not completing. Random Sampling does not automatically recover from this condition and manual intervention is required to reset Random Sampling, as listed in this article.

There is no available method to determine what the optimum number of Monitored Employees per Department should be, nor is there a method to determine how many items can be processed by Random Sampling, as each environment is different and must be reviewed on a case-by-case basis. There are usually 3 main contributing factors:

- Too many Monitored Employees in a Department
In such a configuration, it is recommended to spread the Monitored Employees over multiple Departments which should reduce the processing load per Department.

- The Enterprise Vault Accelerator Manager Service, Background Tasks for the Customer and/or Server are either stopped or are in a stalled/hung state
If the Enterprise Vault Accelerator Manager Service or Background Tasks are manually stopped or if there is an unknown issue on the CA server that caused the tasks to go into a stalled/hung state just before the Random Sampling time, Random Sampling will not run. Once the Service/Tasks are operational, the next Random Sampling run will try to process the items but due to the number of items needing processing, the System.OutOfMemoryException error is logged and the Random Sampling process never completes. A review of the EV, Application and System Event Logs on or just before the date Random Sampling stopped may provide more details. Note that an Event Log review may not explicitly indicate any Background Task issues - the Event Logs should be reviewed for any CA issues or any issues in the Application and System Event Logs.

- SQL server operations or issues
Any SQL maintenance or other such operations can interfere with Random Sampling. It has not been observed that SQL backups have interfered with Random Sampling, but it is certainly a possibility. Additionally, any issues in general on the SQL server can affect any data processing, including Random Sampling. Once the SQL server is operational, the next Random Sampling run will try to process the items but due to the number of items needing processing, the System.OutOfMemoryException error is logged and the Random Sampling process never completes.

Solution

Currently, three possible workarounds exist to this issue to retain the Guaranteed Sampling processing instead of switching to Statistical Sampling. The first workaround is to restart the Background Tasks and lower the number of threads used for Guaranteed Sampling processing. The second workaround is to break the Department into multiple Departments with fewer Monitored Employees in each new Department.  The third workaround is to remove the items from the Sampling tables, conduct a percentage search for the date range the sampling has been missing and allow the Sampling to continue the following day.

If Workaround 1 does not resolve the issue, Workaround 2 will need to be implemented. If Workaround 2 does not resolve the issue, then a combination of Workarounds 1 and 2 may be needed. Workaround 3 may need to be included with Workaround 1 or 2 to clear the appropriate tables to allow them to be of a more manageable size.

WORKAROUND 1:

Restart the Background Tasks and lower the number of threads used for Guaranteed Sampling processing by completing the steps below.

1. Restart the Background Tasks in the EVBAAdmin administration website:
1.1. Browse to the EVBAAdmin administration website, typically http://localhost/evbaadmin, on the CA server while logged in as the Vault Service Account (VSA).
1.2. Restart the server Background Tasks:
1.2.1. Right-click the Accelerator server | Properties | De-select all Background Tasks: 'Enable Customers service', 'Enable Analytics service', 'Enable Customers tasks' | OK | The 'Enable Customers service' will automatically re-select | Click OK again and monitor the Current Status pane until 'Analytics service' and 'Customers tasks' show Stopped.
1.2.2. Then right-click the Accelerator server | Properties | Select all de-selected Background Tasks | OK | Monitor the Current Status pane until all tasks show Running.
1.2.3. Repeat for each Server.
1.3. Restart the Customers Background Tasks:
1.3.1. Expand the Accelerator server | Right-click the Accelerator Customer | Properties | De-select 'Enable Customer's tasks' | OK | Click on the Customer and monitor the Current Status pane until 'Customers tasks' shows Stopped.
1.3.2. Then right-click the Accelerator Customer | Properties | Select 'Enable Customer's tasks' | OK | Click on the Customer and monitor the Current Status pane until 'Customers tasks' shows Running.
1.3.3. Repeat for each Customer on each server.
2. Access the CA Customer via the CA Client using an account with sufficient Application level permissions, such as the VSA.
3. Go to Configuration | Settings.
4. Hold down the Ctrl key on the keyboard and click once on the Configuration Settings header at the upper left side of the window to display the hidden settings.
5. Expand Random Capture.
6. Edit the value for the 'Number of threads used for Guaranteed Sampling' Setting to 1 if not already set to 1.
7. Click outside the Value column, then click on Save to save the change.
8. Acknowledge any prompts to restart Remoteing, Background Tasks or the Enterprise Vault Accelerator Manager Service.
9. Restart the Enterprise Vault Accelerator Manager Service from the Windows Services management console.

WORKAROUND 2:

Break the largest Department or Departments into multiple Departments with fewer Monitored Employees than the original Department or Departments. For example, a Department with 400 Monitored Employees could be broken into 4 smaller Departments of 100 Monitored Employees each.

To create smaller Departments, determine how many and which Monitored Employees will be in each new Department, then create each new Department following the normal steps to create a Department.

WORKAROUND 3:

This workaround involves a reset of Random Sampling by clearing the Random Sampling tables. There are 2 steps to this workaround.

Step 1 - Reset Random Sampling

1.1. Verify the Hidden Setting 'Number of threads used for Guaranteed Sampling' is set to 1 (see Workaround 1 above).

1.2. Backup the CA Configuration and Customer databases.

1.3. Stop the Enterprise Vault Accelerator Manager Service (EVAMS) on the CA server. Then stop all Storage Services on all EV Storage servers to prevent additional items from being processed. This should be done late day to minimize the impact on daily processing.

1.4. Reset Random Sampling. Please contact Veritas Enterprise Vault Technical Support for assistance with this step.

1.5. Start the Storage Services on all EV Storage servers. Then start the EVAMS on the CA server.

Step 2 - Run Searches to find items that were not Randomly Sampled for the missed Date Range.

2.1. Determine what is acceptable by the Compliance team:

a - A randomly accepted percentage of the email for the entire Date Range. Conducting a Search over the entire Date Range would produce a randomly selected percentage overall for all mail in the Department, but could not guarantee that each day would be that exact percentage.

b - A randomly accepted percentage of the email day by day. This would require a separate Search for each day in each Department instead of one Search over the entire date range.

2.2. Conduct one or more Immediate Searches within each applicable Department covering the date range determined in Step 2 above with the appropriate Sampling Percentage as required.

How to configure a Search to emulate Random Sampling

Searches can be run to emulate Random Sampling. However, there are some caveats:
- The Search cannot emulate individual Monitoring Policy percentages for the different Message Types, nor can it emulate any Monitored Employee Policy overrides.
- The Search cannot guarantee the Monitoring Policy percentage per Monitored Employee and the percentage will apply to the Department as a whole (simulating Statistical Sampling).
- A Search can be configured for the entire Date Range of the duration when Random Sampling did not run; however, there is no guarantee the Search will find the number of items per day to match the Monitoring Policy percentages.
- A Search can be created for each day that Random Sampling did not run with the Date Range of the specific missed day(s); this will provide a more accurate emulation of Random Sampling but will require a Search to be run per missed day (can use the 'Based on Search' option to use a previous Search as a template and simply change the Date Range).

A - If each Department has different Monitoring Policy percentages, run a Search in each Department for the Date Range that Random Sampling was missed:

1. Review the Policy percentages under Properties | Monitoring for the first Department. Determine a single percentage value to be used in the Search. The Search cannot emulate individual Monitoring Policy percentages for the different Message Types, nor can it emulate any Monitored Employee Policy overrides. The Search also cannot guarantee the Monitoring Policy percentage per Monitored Employee and the percentage will apply to the Department as a whole (simulating Statistical Sampling).
2. Create a new Search in the Department and configure as follows:
2.1. Name: Enter a name for the Search.
2.2. Based on Search: <No Template>
2.3. Search Type: Immediate.
2.4. Automatically accept search results: Leave this option blank to review the hit counts before Accepting the Search; select this option to automatically Accept the Search upon completion.
2.5. Include items already in review: Leave this option blank.
2.6. Sampling | Sampling percentage: set this value to the percentage value decided upon above.
2.7. Sampling | Minimum per author: do not configure.
2.8. Sampling | Absolute limit: do not configure.
2.9. Date Range | Specific Date Range: Determine what is acceptable regarding the Date Range. A Search can be configured for the entire Date Range of the duration when Random Sampling did not run; however, there is no guarantee the Search will find the number of items per day to match the Monitoring Policy percentages. Alternately, a Search can be created for each day that Random Sampling did not run with the Date Range of the specific missed day(s); this will provide a more accurate emulation of Random Sampling but will require a Search to be run per missed day(s) (can use the 'Based on Search' option in new Searches to use a previous Search as a template and simply change the Date Range).
2.9.1. To run a single Search for the entire missed Random Sampling Date Range, set the From date as the day when Random Sampling stopped running (set the time to 00:00), then set the To date to the day before Random Sampling started running (set the time to 23:59).
2.9.2. To run multiple Searches, each Search for each day in the missed Random Sampling Date Range, set the From date as the required missed Random Sampling day (set the time to 00:00), then set the To date to the same required missed Random Sampling day (set the time to 23:59). Note multiple Searches will need to be created in the Department, one per missed Random Sample day.
2.10. Authors and recipients | Message Route: BETWEEN the Department and internal AND/OR External to organisation. Verify the Any Of operator is selected and the Department is selected.
2.11. Do not configure any other Search Criteria under Search Terms, Attachments, Miscellaneous, Policies or Intelligent Review.
2.12. Save the Search and monitor the progress. If 'Automatically accept search results' was not selected, either Accept or Reject the Search when complete; if 'Automatically accept search results' was selected, the Search results will be added to the Review Set.

B - If each Department has the same Monitoring Policy percentages, run a Search at the Application level to be run in each Department for the Date Range that Random Sampling was missed:

1. Review the Policy percentages under Properties | Monitoring for all Departments. Determine a single percentage value to be used in the Search. The Search cannot emulate individual Monitoring Policy percentages for the different Message Types, nor can it emulate any Monitored Employee Policy overrides. The Search also cannot guarantee the Monitoring Policy percentage per Monitored Employee and the percentage will apply to the Department as a whole (simulating Statistical Sampling).
2. Create a new Search under Application | Searches and configure as follows:
2.1. Name: Enter a name for the Search.
2.2. Based on Search: <No Template>
2.3. Search Type: Immediate.
2.4. Automatically accept search results: Leave this option blank to review the hit counts before Accepting the Search; select this option to automatically Accept the Search upon completion.
2.5. Include items already in review: Leave this option blank.
2.6. Sampling | Sampling percentage: set this value to the percentage value decided upon above.
2.7. Sampling | Minimum per author: do not configure.
2.8. Sampling | Absolute limit: do not configure.
2.9. Date Range | Specific Date Range: Determine what is acceptable regarding the Date Range. A Search can be configured for the entire Date Range of the duration when Random Sampling did not run; however, there is no guarantee the Search will find the number of items per day to match the Monitoring Policy percentages. Alternately, a Search can be created for each day that Random Sampling did not run with the Date Range of the specific missed day(s); this will provide a more accurate emulation of Random Sampling but will require a Search to be run per missed day(s) (can use the 'Based on Search' option in new Searches to use a previous Search as a template and simply change the Date Range).
2.9.1. To run a single Search for the entire missed Random Sampling Date Range, set the From date as the day when Random Sampling stopped running (set the time to 00:00), then set the To date to the day before Random Sampling started running (set the time to 23:59).
2.9.2. To run multiple Searches, each Search for each day in the missed Random Sampling Date Range, set the From date as the required missed Random Sampling day (set the time to 00:00), then set the To date to the same required missed Random Sampling day (set the time to 23:59). Note multiple Searches will need to be created in the Department, one per missed Random Sample day.
2.10. Authors and recipients | Message Route: BETWEEN the Department and any addressee. Verify the Any Of operator is selected and the required Departments are selected. It is not necessary to select the 'Use inheritance, automatically include new departments' option as this is an Immediate Search, not a Scheduled Search or Guaranteed Sample Search.
2.11. Do not configure any other Search Criteria under Search Terms, Attachments, Miscellaneous, Policies or Intelligent Review.
2.12. Save the Search and monitor the progress. If 'Automatically accept search results' was not selected, either Accept or Reject the Search when complete; if 'Automatically accept search results' was selected, the Search results will be added to the Review Set.