Best Practice Guide to Optimize an eDP/Clearwell to Enterprise Vault Collection Task

eDP/Clearwell Enterprise vault (EV) Collection performance and success lies primarily on the health, state and optimization of the EV environment.

Section A:  Optimization of the eDiscovery Platform environment

 

A1. Upgrade the eDP/Clearwell software version to 8.0 or higher. V8.0 has the ability to only load EV index volumes that contain items within the date range selected. Earlier versions of the software load all volumes of an Archive regardless of the requested date range search.
 

Note: This significantly increases the time to conduct a date range collection as well as lower the resource requirements against the EV indices.

 

A2. Schedule eDP/Clearwell EV Collection Tasks for non-peak EV hours and seperate each Collection Task to allow for successful completion.

The eDP/Clearwell EV Collection Task is a two fold process; Search and Content Retrieval. One EV Index Lease and one Storage lease is provided to the eDP/Clearwell appliance at a time. eDP/Clearwell will use those leases to complete a thorough search and retrieval of the items before releasing the lease to the next Collection Task in queue.

eDP/Clearwell EV Collection Tasks have a built-in sixty (60) minute timeout to obtain the EV lease before failing, therefore it is paramount to schedule multiple EV Collection Task runs far enough apart to allow each Collection Task to successful obtain the EV leases.

 

A3. Do not schedule EV Collection Tasks during EV maintenance windows (backup mode or SQL server maintenance on the EV databases).

A4. Increase Outlook memory and the Maximum Email Recipient and Attachment counts.
NOTE: You  MUST be logged into the eDP appliance as the EsaApplicationService account.

• On each eDP server, make the following registry changes:
Note: If the keys or values are not present, create them with the following values:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Messaging Subsystem
On appliances with 32Gb memory:
(DWord) SharedMemMaxSize (Value) 0x02000000
On appliances with over 32Gb memory:
(DWord) SharedMemMaxSize (Value) 0x10000000

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Messaging Subsystem\Applications\Outlook
On appliances with 32Gb memory:
(DWord) SharedMemMaxSize (Value) 0x02000000
On appliances with over 32Gb memory:
(DWord) SharedMemMaxSize (Value) 0x10000000

HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Outlook\Options\Mail
(DWord) AttachmentMax (Value) 0xFFFFFFFF
(DWord) RecipientMax (Value) 0xFFFFFFFF


Note: If using Policies to push Office registry settings, also include:

HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\15.0\Outlook\Options\Mail
(DWord) AttachmentMax (Value) 0xFFFFFFFF
(DWord) RecipientMax (Value) 0xFFFFFFFF
 

A5. Thread settings (for busy EV environments)

Note: Do not apply changes to robust EV environments

For issues concerning the EV Search portion of the Collection Task:
esa.icp.collection.evsearch.MaxThreads = 3 (default 10)
esa.icp.collection.evsearch.BatchSize = 5000 (default 50000)

For issues concerning the EV Content Retrieval portion of the Collection Task:

esa.icp.ev.content_retrieval.threads = 5 (default 15)
esa.icp.collection.ev.csvReadBatchTimeout = 600 (default 10 in secs)
esa.icp.collection.ev.content.defaultRetrieveTimeoutSecs  =  1200 (default 40 in secs)
esa.icp.ev.contentRetriever.readTimeout  = 180000 (default 60000 in millisecs)
esa.icp.ev.contentRetriever.readTimeoutDelta  = 360000
esa.icp.collection.ev.content.defaultRetrieveTimeoutThreshold  =  500000

How to modify ESA properties

a. Log into the user interface.
b. Navigate to System > Support Features
c. Choose Property Editor
d. In the text box for 'Name of property to change' enter the ESA property
e. In the text box for 'New value (leave blank to remove)' enter the new value
f. Check the box for 'Confirm change. Are you sure?*'
g. Click the Submit button

 

A6. Verify that Anti-Virus scanning has excluded: D:\MySQL\*; D:\MySQLData\*; D:\MySQLtemp\*; D:\CW\*(plus subdirectories); D:\CaseData\*; D:\CWShared\*; C:\jrockit-jdkxxx.
When Using Sound feature also exclude: C:\program Files (x86)\Nexidia\*, C:\program Files\Nexidia\* and D:\Nexidia\*  See article Required antivirus exclusions for Veritas eDiscovery Platform

 

A7. Disable TOE on the Clearwell NICs

(How to Disable TCP Chimney, TCPIP Offload Engine (TOE) or TCP Segmentation Offload (TSO))

https://www.veritas.com/docs/000094937

( Note: Run the attached batch files on the appropriate server and send in for analysis)

 

A8. Provide ample space on the Logical D hard drive to allow for scratch folder consumption.  Default installation recommends 1.5 TB.

A copy of each extracted EV item will be placed in the scratch location on the eDP/Clearwell D drive during the Content Extraction phase of the operation. Once all items have been extracted, they will be wrapped into a PST and copied to the Preservation Store. Once the PST have been scanned for integrity, the EV items will be deleted from the scratch location on the eDP/Clearwell device. 

Note: Use of the ESA property esa.icp.ev.tempDirRoot will move the temporary storage of EV items to a new location; recommend to use a locally attached drive for this location.

 

A9. Run the Performance Counters during peak usage. Do not run overnight !!!! The counters are an average and only running the Performance Counters during heavy usage, will provide an accurate assessment of resources.

(Download and install, on the eDP/Clearwell appliance, the PerformanceCounters-Clearwel.xml file from HOWTO84091 article)

 

A10. The Preservation Store (Destination Server) speed and health is important.

Hard drive spindle speeds, NIC speeds and location away (hops) from the eDP/Clearwell appliance all play a part in the optimization of performance. Direct Access Storage with spindle speeds of 15K is preferred.

B1. Increase memory to 32 GB recommended and allocate 8 CPU cores on each Enterprise Vault server.

Memory settings for VMWare
EV environments where regular eDiscovery searches are run the memory should be increased to 32GB.
Set the memory reservation to the configured size of the memory allocated to the virtual machine (32GB).

 

B2. Create a Pagefile to 1.5X installed physical memory on each Enterprise Vault server.

EV 10 Best Practice - Enterprise Vault 10 Indexing (December 2011)
https://www.veritas.com/docs/000081275

B3. Verify that all archive volumes are in a searchable condition prior to running eDP/Clearwell EV Collections. Repair any volumes necessary or note the failed volumes will result in a 'Partial Success' for the eDP/Clearwell EV Collection Task.

a. Open the Enterprise Vault Administrator Console

b. Expand the site to Indexing

c. Right click Indexing and select 'Index Volumes Browser...'

d. Select 'Extra Items', 'Failed', 'Missing Items' and 'Offline'

e. Select 'Search'

f. The results will determine which Indices need EV administrators attention prior to eDP/Clearwell access to the index.

 

B4. EV Vault Store SQL database optimization 
How to automatically back up and perform recommended maintenance for Enterprise Vault SQL databases

https://www.veritas.com/docs/000040169

Run the following SQL query against each EV Vault Store and send in the results for analysis

DBCC SHOWCONTIG (Saveset) WITH ALL_INDEXES

Note: dm_db_index_physical_stats does not contain the Extent Fragmentation, Veritas requires the results from SHOWCONTIG for a complete analysis.

Extent Fragmentation (pages out of sequence)

•Extents Scanned - (rounded up) should equal Pages Scanned / 8.

•Extent Switches - equals (Extents Scanned -1).

•Scan Density should be above 75%

•Extent Scan Fragmentation - under 50%

•Logical Scan Fragmentation - under 10%

•Avg. Page density (full) - over 90%

If any results do not fall within the listed parameters, there is fragmentation and SQL maintenance on the database should be performed.

Recommended weekly SQL maintenance

(Place the Enterprise Vault services into backup mode (read-only))

•Backup database

•Rebuild Indexes

•Update Statistics

 

B5.  SQL server memory allocation to the SQL instance

Multiple instances can reside on the same SQL box. If min and max server memory per instance is not set, an instance will attempt to control as much memory as it can obtain taking away available resources for the other instances and applications. Releasing memory from the buffer pool is slow and during the applying or releasing of legal holds, the SQL server service will begin to allocate a large quantity of available memory.

How to verify memory allocation and usage; 

Open task manager processes tab and locate sqlservr.exe and note the Mem Usage value. If the memory used exceeds seventy percent (70%) of available server memory, set the Max server memory value between 60% - 70% of available server memory, leaving the remaining 30% - 40% for extended stored procedures, COM objects, non-shared DLL's, Ex's, and MAPI components. After setting the max server memory in the instance, a reboot of the server is required to adjust the new Configuration settings into the Running settings.

Min server memory should be left at 0 unless rebooting of the SQL server is common practice. Allocating memory consumes resources and time, thus frequent rebooting of the SQL server with Min server memory setting at 0 is inefficient. To determine what value to place on min server memory, reboot the SQL server while there is a normal or below normal SQL workload (no legal hold work to be done). Open Task Manager and Min server memory at the Mem Usage value for sqlservr.exe

B 6. Follow the recommended antivirus exclusions for Enterprise Vault.   

https://www.veritas.com/docs/000032085

** Special Exclusions for EV10 onward
C:\Users\<VaultServiceAccount>\AppData\Local\Temp\
C:\inetpub\temp\appPools\EnterpriseVaultAppPool\
C:\Windows\inf\Enterprise Vault Index Query Server\
C:\Windows\TEMP\
<install_path>Program Files (x86)\Enterprise Vault\EVIndexing\data
<install_path>\Users\<VaultServiceAccount>\AppData\Local\Microsoft\Windows\Temporary Internet Files\Content.IE5

C:\EVStorageQueue ( EV11 onward)

 

B7. Vault Service Account Temporary folder size.

All transactions requests from the eDP/Clearwell Collector to the Enterprise Vault (EV) server utilize the Vault Service Account (VSA) temporary folder for temporary storage. This folder needs to be able to grow large enough to accommodate Batch processing for Index (Search) and Storage (Export) requests on the EV server. Locating the VSA temporary folders on a hard drive with at least 20 GB free space availability is recommended for all EV servers.

 

B8. Disable TOE on the Clearwell NICs

(How to Disable TCP Chimney, TCPIP Offload Engine (TOE) or TCP Segmentation Offload (TSO))

https://www.veritas.com/docs/000094937

( Note: Run the attached batch file on the appropriate server and send in for analysis)

 

B9. (For 32-bit indices only) 

Ensure the indices are approximately 2.5 GB in size. 
Original 32-bit indices were allowed to grow to approximately 20 GB before rolling over to the next volume.  This size is not efficient for eDiscovery searching using tools such as eDP/Clearwell or Discovery Accelerator.  The searches against the larger indices can cause the Enterprise Vault server to become unresponsive and possibly require a reboot to replenish the application pool memory consumed by the Indexserver service.

Note: If the indexes are over 2.5 GB in size, a rebuild of the index is required.  CAUTION:  Large 32-bit indices can take days or even weeks to rebuild.  The 32-bit index will be converted to a 64-bit index during the rebuild.  Searching against that archive will be available using only the 32-bit index until the rebuild is complete; at which time the 64-bit index will become available and the 32-bit index will be deleted.

 

B10. Upgrade to EV 12

All patches to accommodate Veritas eDiscovery products are in place in the base version of EV12.

B11. Upgrade 32-bit indices to 64-bit.

Note: The 32-bit indices will be available during the upgrade.  Once the upgrade is complete, the 64-bit index is used and the 32-bit index is deleted.

B12. Reduce StorageOnlineOPNS thread
Note: This issue occurs in all versions of EV 11 and versions of EV 12
A race condition of the StorageOnlineOpns process can exist in Enterprise Vault versions 11 and higher.  Thread contention of the StorageOnlineOpns process threads can lead to latency in Index performance and content retrieval.

Steps to remove the extra StorageOnlineOpns processes:
  a. Open the Vault Administrative Console
  b. Navigate to Site Properties > Advanced > Storage
  c. Set 'Additional StorageOnlineOpns instances' to 0 (Default is 4)
  d. Restart all Enterprise Vault (EV) services
(Note: StorageOnlineOpns is called outside of the Storage Service, which requires the start of all EV services) 


B13. (Execute after steps B12 and B13 are complete)

Repair 'Missing items' in Journal Archives

Steps to resolve 'Missing Items':
  a. Open the Vault Administrative Console
  b. Right click 'Indexing' and select 'Manage Indexes' > 'Synchronize' > Next > Add
  c. Type the Name of the Archive in the Search Criteria with 'Missing Items' as the selected attribute, then Search.
  d. Select the Archive from the Search Results pane and press Add to move the Archive to the Selected Items table.
  e. Then select  OK > Next > Next > Create Task > Close or 'Monitor Tasks'
 

Synchronization tasks can be monitored at any time by right clicking 'Indexing' > Select 'Monitor Indexing Tasks...'


B15. Run the Performance Counters during peak usage. Do not run overnight !!!! The counters are an average and only during heavy usage, will provide an accurate assessment of resources.

Note: The template is named for eDP/Clearwell, but works on the EV servers as well.

(Download and install, on the Enterprise Vault Servers, the PerformanceCounters-Clearwel.xml file from HOWTO84091 article)