Recommended steps to optimize performance on Enterprise Vault, Compliance Accelerator, Discovery Accelerator and SQL Servers

Article: 100023750
Last Published: 2023-09-19
Ratings: 20 3
Product(s): Enterprise Vault

Problem

Compliance Accelerator (CA) and / or Discovery Accelerator (DA) search and / or export performance is slower than expected.

Cause

The Enterprise Vault (EV), Compliance Accelerator (CA) and Discovery Accelerator (DA) installations are configured by default to settings that have been found to be optimized in what is considered to be common environments.  There are instances where the default optimization settings are not adequate and issues arise with CA and DA searching and exporting.  Some adjustments to the EV, CA/DA and SQL servers may be necessary.

Solution

The following settings are good starting values to optimize performance in individual environments for EV, CA/DA and SQL servers

Note: Incorrect configuration for TCP Chimney, TCPIP Offload Engine (TOE) or TCP Segmentation Offload (TSO) causes transport-level errors to be logged resulting in inaccurate hit counts for Accelerator searches. In addition to this article, Support recommends configuration of the settings recorded in 100023865.

Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly.  Great care should be taken when making changes to a Windows registry.  Registry modifications should only be carried-out by persons experienced in the use of the registry editor application.  It is also recommended that a complete backup of the registry and workstation / server be made prior to making any registry changes.
 

Optimize Enterprise Vault (EV) 32-bit Indexing (not for Enterprise Vault 10 or greater)


Create the following keys on all Enterprise Vault Indexing servers:
 
Key Location Setting
AVSMaxLoc HKLM
     \Software
               \KVS
                    \Enterprise Vault
                                     \Indexing
Value name: AVSMaxLoc

Data type: REG_SZ (Decimal)

Value data: 500000000 (500 Million)
SchemaType HKLM
     \Software
               \KVS
                    \Enterprise Vault
                                    \Indexing
Value name: SchemaType

Data type: REG_SZ (Decimal)

Value data: 1


Disclaimer: Existing Indexes have to be rebuilt while changes are effective immediately on new indexes

IMPORTANT: Rebuilding some indexes (i.e., a journal archive index with lots of index volumes) can have a significant impact on a production environment.  As such, these settings may be implemented at any time to affect current and new Index Volumes. Any index or index volume rebuild should be delayed until the time frame and impact on the production environment has been studied and sufficient time has been allotted.
 
 

Disable Opportunistic Locking on the EV, CA/DA and SQL servers (for 32-bit and 64-bit Windows Server operating systems)


For all Windows Servers prior to 2012
 
Key Location Setting
OplocksDisabled HKLM
     \System
           \CurrentControlSet
​                   \Services
                          \MRXSmb
                                   \Parameters
Value name: OplocksDisabled

Data type: REG_DWord

Value data: 1

For Windows Server 2012 / 2012 R2
 
Key Location Setting
OplocksDisabled HKLM
     \System
           \CurrentControlSet
  ​               \Services
                        \SmbMRx
                                \Parameters
Value name: OplocksDisabled

Data type: REG_DWord

Value data: 1
 
For all Windows servers:
 
Key Location Setting
EnableOplocks HKLM
     \System
             \CurrentControlSet
                     \Services
                             \LanmanServer
                                      \Parameters
Value name: EnableOplocks

Data type: REG_DWord

Value data: 0


Note: For Windows Server 2012 / 2012 R2, the following PowerShell commands can be used in place of editing the registry to set the above parameters. Running PowerShell commands may not require restarting the server upon their successful completion. Support recommends a server restart to clear any remaining in-memory settings.
  • Set-SmbClientConfiguration -OplocksDisabled $true​
     
  • Set-SmbServerConfiguration -EnableOplocks $false


Set memory management registry keys on the EV, CA/DA and SQL servers (for 32-bit Windows Server operating systems only, not for Enterprise Vault 10 or greater)

 
Key Location Setting
PoolUsageMaximum HKLM
    \System
          \CurrentControlSet
                \Control
                       \Session Manager
                             \Memory Management
Value name: PoolUsageMaximum

Data type: REG_DWORD (Decimal)

Value data: 60 *
PagedPoolSize HKLM
    \System
          \CurrentControlSet
                \Control
                       \Session Manager
                             \Memory Management
Value name: PagedPoolSize

Data type: REG_DWORD

Value data: 0xFFFFFFFF

* PageUsageMaximum: ​Default is 80 and can be set as low as 20 - this represents % used before memory management takes control
 


Set paging file on all EV, CA/DA and SQL servers for 32-bit and 64-bit Windows Server operating systems
 

  • Location: My Computer | Properties | Advanced | Performance | Advanced | Virtual Memory
    • Set initial and maximum sizes to an appropriate value (recommended in 4092 MB increments where possible) with multiple paging files across multiple drives.
    • These settings are adjusted as needed to get the total paging file size to approximately 2X the amount of physical memory on the server for 32-bit or 64-bit Windows Server, or set paging file to System managed size for 64-bit Windows Server.
       
  • Move the paging file off of the system drive (i.e., C:\-Drive) whenever possible for better system performance.
    • The system drive, by default, hosts the Windows Registry and the paging file used for memory crash dumps and to swap stale pages out of active memory.
    • Both of these are heavily used by normal Windows operations. Moving the paging file used to swap stale pages out of memory off of the system drive will allow for improved disk I/O, presuming the drive on which the paging file was moved is a physically separate drive from the system drive.​
       
  • Microsoft books online article https://learn.microsoft.com/en-US/troubleshoot/windows-client/performance/how-to-determine-the-appropriate-page-file-size-for-64-bit-versions-of-windows discusses how to determine the appropriate size for a page file on 64-bit Windows.
    • If the Windows server has sufficient physical memory to process all memory requests without having to swap stale pages to disk, there is no need to have a page file other than under certain conditions.
    • Veritas acknowledges the possibility of no need for a page file provided that performance monitoring under heavy loads indicates physical memory remains available to be used (i.e., peak memory used is always less than physical memory).
    • Careful monitoring under multiple periods of heavy loads should be performed before deciding no page file or a reduced page file from the above recommendation is needed.
 


Vault Service Account (VSA) Temporary folders recommendations:
 

  • On physical drive with a minimum 40 GB (preferably 80 GB) of available free space
     
  • Off of the system drive (i.e., C:\-Drive)
Follow these steps to change the location of the temporary folders:
  1. Log onto the EV server as the Vault Service Account (VSA)
     
  2. Stop all EV services on the server being changed
     
  3. Navigate to My Computer | Properties | Advanced | Environmental Variables | User variables for VSA
     
  4. Click the line for the TEMP variable
     
  5. Click the Edit button
     
  6. Input new folder path location in the Variable value: field (for example: D:\Temp)
     
  7. Repeat Steps D through F for the TMP variable
    Note: using the same destination folder for both the TEMP and TMP folders is permitted and recommended
     
  8. Click the OK button to save the changes
     
  9. Restart the server to clear the memory of any cached instances of the old locations and fully implement the new locations
     
Additional Considerations: 
  • Ensure the VSA has Full permissions on the folder so that it can write/delete data properly. See article 100013877.
  • The VSA TEMP folder is used heavily by normal EV processes.  The system drive is also used heavily by the Windows Registry.  Moving the VSA TEMP folder off of the system drive allows for improved disk I/O on both drives (presuming both drives are physically separate devices) as the need to access the registry and VSA TEMP folder would be spread across multiple drives.
     
  • Support recommends that the VSA TEMP directory is excluded from Anti-virus scanning.  For a complete listing of the Recommended listing of exclusions for EV refer here.
     
  • Beginning with Enterprise Vault, Compliance Accelerator and Discovery Accelerator 11.0 SP1 (11.0.1), an extra security check will be conducted at the start of the Enterprise Vault Admin Service and the Enterprise Vault Accelerator Manager Service (EVAMS) to verify the NTFS permissions set on the VSA TEMP** folder only allow authorized access. If any user or group account is found present that is not authorized, the service does start and an error is generated in the Enterprise Vault Event Log.  See the appropriate section in the EV, CA and DA installation guides, referenced here, for information related to the extra security check.
     
  • **It is important ensure the correct permissions are set on the NTFS Security Tab
    • Using Windows Explorer, right click on the folder
    • Select the Properties option
    • Click on the Security tab to review what accounts have been granted any permissions to the folder
    • Compare these permissions with those of the previous (default) location and adjust them to match if possible.
 


Optimize TCP/IP communications for 32-bit and 64-bit Windows Server operating systems
 

Important Note For Servers In A Clustered Environment Or That Use Network Interface Card Teaming

It is of vital importance to determine if a server is a member of a cluster BEFORE making any changes to the TCP Offload Engine settings described in this article. Examples include Windows Server Failover Cluster nodes and SQL Always On Availability Groups replicas. Some cluster applications require TCP Offload Engine to be enabled on each cluster node or replica for proper functionality. Disabling any TCP Offload Engine settings on cluster nodes or replicas could adversely affect network performance for cluster-aware applications and/or operating systems. As such, it is recommended not to edit any TCP Offload Engine settings for servers that are nodes or replicas in a clustered environment without first consulting the cluster application documentation. If the cluster documentation clearly confirms TCP Offload Engine settings can be changed without any negative effects, then proceed with the changes after creating a plan to roll back the changes if needed. When in doubt, do NOT make any changes to the TCP Offload Engine settings.

Similar consideration should be given to servers using Network Interface Card (NIC) teaming. Some NIC teaming applications require TCP Offload Engine to be enabled on each NIC for proper functionality. Disabling any TCP Offload Engine settings on teamed NICs could adversely affect network performance for cluster-aware applications and/or operating systems. As such, it is recommended not to edit any TCP Offload Engine settings for servers that use NIC teaming without first consulting the NIC teaming documentation. If the NIC teaming documentation clearly confirms TCP Offload Engine settings can be changed without any negative effects, then proceed with the changes after creating a plan to roll back the changes if needed. When in doubt, do NOT make any changes to the TCP Offload Engine settings.

 

  • Disable any Network Teaming on the Enterprise Vault (EV) servers running the EV Indexing Service and / or EV Storage Service, plus on the Compliance Accelerator (CA) and Discovery Accelerator (DA) server.
     
  • Disable TCP Chimney at the operating system level and all TCP Offload / Receive Side Scaling / TSOEnable settings on all Network Interface Cards on the EV, CA, DA and SQL Servers per Technical Article 100023865.
     
  • Optimize the TCP Parameters for Accelerator, Enterprise Vault and SQL servers by setting the following registry keys :
    • Note:  When increasing the MaxUserPort, Microsoft recommends that port 1434 be reserved for use by the SQL Server Browser service (sqlbrowser.exe). 

      On a Windows 2003 server, add the following registry key:
       
      Key Location Setting
      ReservedPorts HKLM
           \System
                 \CurrentControlSet
                        \Services
                               \Tcpip
      ​                              \Parameters
      Key: ReservedPorts​

      Data type: Multi-string Value

      Value data: 1434-1434
       

       
    • Additional registry keys for Windows Server 2003
       
      Key Location Setting
      MaxUserPort 
      TCPTimedWaitDelay
      MaxFreeTcbs
      ​MaxHashTableSize
      ​EnableTCPChimney
      ​EnableRSS
      EnableTCPA 
      HKLM
           \System
                 \CurrentControlSet
                        \Services
                               \Tcpip
      ​                              \Parameters
      Data type: REG_DWORD (Decimal)

      Value data:
      MaxUserPort = 64,512
      TCPTimedWaitDelay = 120
      MaxFreeTcbs = 65,536
      ​MaxHashTableSize = 16,384
      ​EnableTCPChimney = 0
      ​EnableRSS = 0
      EnableTCPA = 0

       
    • Additional registry key for Windows Server 2008
       
      Key Location Setting
      TCPTimedWaitDelay HKLM
           \System
                 \CurrentControlSet
                        \Services
                               \Tcpip
      ​                              \Parameters
      Key:  TCPTimedWaitDelay

      Data type: REG_DWORD (Decimal)

      Value data: 120
       

    Disclaimer: When making changes to a network card’s properties, the network card may reset itself internally to initialize the changes. It is possible the network card could temporarily drop its connection during this internal reset process. Please be aware of any temporary network connectivity disruptions that could result from any changes to a network card’s properties and plan the changes so as to minimize any impact on the environment (it may be required to implement any changes during a time where network activity is at a minimum).
     


    Useful Net Shell (netsh) commands:


    Useful Net Shell (netsh) commands only for Windows Server 2003 and lower:
     
    • Display the current offload settings for the TCP protocol:
      netsh int ip show offload
       
    Useful Net Shell (netsh) commands only for Windows Server 2008 and higher:
     
    • Display current global TCP settings:
      netsh int tcp show global
       
    • Display the current ports for the TCP protocol:
      netsh int ipv4 show dynamicport tcp
       
    • Display the current TCP Chimney Offload statistics for all network connections:
      netsh int tcp show chimneystats
       
    • Display the current TCP Chimney port filters:​
      netsh int tcp show chimneyports​
       
    • Display the current TCP Chimney application filters:
      netsh int tcp show chimneyapplications
       
    • Increase TCP ports:
      netsh int ipv4 set dynamicport tcp start=1500 num=63000​
      Increases user TCP ports to 63,000 TCP
      ** Note:  Some applications, such as the EV Indexing and Storage services and the CA/DA Clients, can use ports in the expanded port range.  Ports between 1500 and 8000 may need to be excluded from the expanded port range.  If a conflict is encountered (such as a service failing to start), change the starting point for the ports to a number greater than 8000.
       
    • Disable Chimney Offload (TCP Chimney):
      netsh int tcp set global chimney=disabled  
       
    • Disable Receive-Side Scaling (RSS):
      netsh int tcp set global rss=disabled  
       
    • Disable NetDMA (TCPA):​
      netsh int tcp set global netdma=disabled  


    * A restart of the server is required to implement the changes.

    Disclaimer: When making changes to a network card’s properties, the network card may reset itself internally to initialize the changes. It is possible the network card could temporarily drop its connection during this internal reset process. Please be aware of any temporary network connectivity disruptions that could result from any changes to a network card’s properties and plan the changes so as to minimize any impact on the environment (it may be required to implement any changes during a time where network activity is at a minimum).

    Note: Microsoft has identified an issue running the netsh command to set global TCP parameters on older Windows machines.  Some global parameters, such as TCPTimedWaitDelay, can be changed from their default or manually set values to 0xffffffff.  Before running the netsh command to change any global TCP parameter, it is recommended to review the following Microsoft books online article https://it is recommended to review the following Microsoft books online article https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/troubleshoot-network-performance-features-rss-netdma.  Upon completion of any netsh command to change any global TCP parameter, it is also recommended to review the TCP parameters noted in the KB article and applying the hotfix from the article if needed.
     


    CA and DA Tuning:

    1. Open the CA/DA Client
       
    2. Navigate to the System Configuration pages
       
    3. Click on the Configuration tab in the top toolbar
       
    4. Click on the Settings sub-tab in the second toolbar
       
    5. Section Name:  System

      Set a date for Enterprise Vault Oldest Archived Item Date to any date preferred (for example, January 1, 2001) - Does not require a restart of any process, service or task to become effective.

      Set a date for Enterprise Vault V5 Installation Date to the date of Enterprise Vault 5.0 installation, or any preferred date if EV5 was never installed in the environment (for example, January 1, 2001) - Does not require a restart of any process, service or task to become effective.
       
    6. Section Name:  Search
       
      Set Total Number of search results worker threads to 2 - Requires a restart of the Customer's Background Task or the Enterprise Vault Accelerator Manager Service to become effective.

      Enable Optimise searches based on oldest and youngest items by placing a check mark in the check box in the Value column of this option - Requires a restart of the Remoting process, performed by restarting the Enterprise Vault Accelerator Manager Service, to become effective.
       
    7. The setting Maximum number of consecutive searches on same index defines how many Searches a Search thread could run before switching to a different Index. The default value for this setting from 8.0 through 9.0.3 was unlimited (0). From 9.0.4 onwards, this has been changed to 500 in order to make use of improved Search queuing mechanisms which load the Index Volume(s) into memory, run the Searches against the Index Volume(s) and then unload the Index Volume(s) from memory. The default value of 500 may need to be further optimized if the Indexing environment currently consumes high resources without the added load of Accelerator Searches, and/or the Search criteria makes extensive use of wildcards. This setting is normally hidden; to view hidden settings, hold down the Ctrl key and click on the Configuration Settings banner in the Configuration | Setting tab.

      In a 32-bit Indexing environment or a mixed 32-bit/64-bit Indexing environment, it is recommended this setting be reduced to 200 and Searches tested. If unsatisfactory Index Search performance is still seen, this can be further incrementally reduced and Searches tested, but should not normally be set below 10. If it is determined setting this value to 1 results in the optimum Index Search performance, then it is recommended the Search criteria be modified and/or the Indexing server(s) be reviewed for available resources and overall Index health.

      In a 64-bit environment, it is recommended this setting be reduced to 200 and Searches tested. If unsatisfactory Index Search performance is still seen, this can be further incrementally reduced and Searches tested, but should not normally be set below 100. If it is determined setting this value to 10 or less results in the optimum Index Search performance, then it is recommended the Search criteria be modified and/or the Indexing server(s) be reviewed for available resources and overall Index health.

      If searching large Index Volumes tends to cause Index processes to fail or timeout frequently, even though there is nothing wrong with any of the Index Volumes being searched, it is also recommended to have the following settings changed to the values below:

      Set Number of Vault Search threads from 10 (default) to 4.

      Set Search Result Chunk Size (hidden setting) from 50000 (default) to 10000.

      Note: Any changes to the above settings will require the Enterprise Vault Accelerator Manager Service be restarted.
       
    8. Section Name:  Legal Holds (for DA only)
       
      Set Placing hold items Batch Size to 1,000 - Requires a restart of the Customer's Background Task or the Enterprise Vault Accelerator Manager Service to become effective.

      Set Removing hold items Batch Size to 1,000 - Requires a restart of the Customer's Background Task or the Enterprise Vault Accelerator Manager Service to become effective.

Was this content helpful?