Slow performance when backing up remote servers with the TCP Offload Engine (TOE) feature enabled on network cards and/or if Windows 2003 SP2 Scalable Networking Pack (SNP) is enabled.

  • Modified Date:
  • Article ID:000081547


Slow performance when backing up remote servers with the TCP Offload Engine (TOE) feature enabled on network cards and/or Windows 2003 SP2 Scalable Networking Pack (SNP) is enabled. Please note that most topics discussed in this article are applicable to later versions of Windows Operating System.


Windows 2003 Service Pack 2 introduced a new update called SNP (Scalable Networking Pack) . This utilizes a feature in newer network cards called TCP/IP Offload Engine (TOE ) or TCP Chimney.  Some servers that have this feature enabled experience throughput or connectivity issues when used in conjunction with certain NICs (Network Interface Cards) between the media server and remote machines.  
TCP Offload Engine is an emerging technology which is designed to offload TCP stack handling from the main system CPU to a processor built into NIC cards.  This technology is still relatively new, and when engaged, has been known to cause unstable connections.  This results in dropped sockets, dropped packets, packet reordering, packet retransmits and ultimately slow or failed backups.
There are two types of TCP offloads - stateful and stateless.
Stateful offload:
TCP Chimney is a stateful offload. TCP Chimney offload enables TCP/IP processing to be offloaded to network adapters that can handle the TCP/IP processing in hardware.
The Windows 2003 TCP Chimney setting and several Network Card (NIC) settings have been known to interfere with successful inter process communication and connectivity over TCP/IP networks.
Stateless offload:
RSS and NetDMA are stateless offloads. Where multiple CPUs reside in a single computer, the Windows networking stack limits "receive" protocol processing to a single CPU. RSS resolves this issue by enabling the packets that are received from a network adapter to be balanced across multiple CPUs.
The following has been observed in cases when this issue occurs:
  1.  Backup of Local Media Server to B2D (Backup to Disk) runs at a normal throughput rate.
  2.  Backup of Local Media Server to tape runs at a normal throughput rate.
  3.  Backup of Remote Server (One or All) to B2D runs at a normal throughput rate in most instances, but runs slower in some network configurations.
  4.  Backup of Remote server (One or All) to tape is consistently slow or loses connection all together during a backup or restore.
Note:  If this occurs only while backing up one remote server, TOE is most likely enabled on that Server's NIC.  If this is happening for all remote servers , then TOE Chimney is most likely enabled on the Media Server's NIC.  

To determine if the TOE Feature is possibly causing the performance issue, disable the feature per the following and run additional remote backups to test:
  1. Modify the network adapter (NIC) properties to disable Receive Side Scaling or the TOE Option.  Contact the Hardware Provider for specific instructions.
  2. Temporarily disable Windows 2003 SP2 Scalable Networking Pack TCP Chimney Offload feature by issuing the following command at a Windows command prompt (Start, Run, CMD). No reboot is required
Netsh int ip set chimney DISABLED
To check the current setting, observe this registry key:
Values: 1 (enabled) 0 (disabled)
Note:  This is a temporary work-around to disable the TOE Feature and will not persist after a reboot.  Also, if the command completes successfully but no improvement is seen in the performance, then it most likely is not being caused by the TOE Feature.
  1. Disable the Windows 2003 SP2 Scalable Networking Pack features (TCP Chimney Offload, Receive Side Scaling , Network Direct Memory Access) by applying the update per Microsoft KB 948496  
An update to turn off default SNP features is available for Windows Server 2003-based and Small Business Server 2003-based computers
For additional information on the TOE Feature, review the following:
How to enable or disable TCP Offload at the hardware/driver level:
In addition to what can be done through the Windows OS, individual network cards have attributes associated with their drivers which can toggle TCP offload settings from the main CPU to the processor on the NIC.
To list out which TCP offloads are enabled per network card, run this command:
Netsh int ip show offload

Here are examples of settings from 2 different NIC cards:

Example 1: Output from Netsh int ip show offload:

Offload Options for interface "Local Area Connection" with index: 10005:
TCP Transmit Checksum
IP Transmit Checksum
TCP Receive Checksum
IP Receive Checksum
TCP Large Send
In order to toggle these settings, go to Windows Device Manager > Network Adapters > (NIC) > Advanced tab
Example 1 Advanced tab settings:
Example 1 map :
TCP Transmit Checksum = Offload Transmit TCP Checksum
IP Transmit Checksum     = Offload Transmit IP Checksum
TCP Receive Checksum   = Offload Receive TCP Checksum
IP Receive Checksum      = Offload Receive IP Checksum
TCP Large Send              = Offload TCP Segmentation

Example 2 Output from Netsh int ip show offload:

Offload Options for interface "Local Area Connection" with index: 20003:
TCP Transmit Checksum
IP Transmit Checksum
TCP Receive Checksum
IP Receive Checksum
TCP Large Send
Example 2 Advanced tab settings:
Example 2 map :
TCP Transmit Checksum = Checksum Offload
IP Transmit Checksum     = Checksum Offload
TCP Receive Checksum   = Checksum Offload
IP Receive Checksum      = Checksum Offload
TCP Large Send              = Large Send Offload

Veritas Support recommends setting all these values to Off / Disabled / None  (or whatever value means "off")

Important Notes:
  1. As can be seen above, different NIC manufacturers and drivers may name these attributes differently.  Experimentation can be done to identify which settings map to which names by changing the value in the Attribute section of the NIC properties and then rerunning Netsh int ip show offload to see what changed.
  2. Modifying any of these settings will temporarily disconnect the NIC from the network.  Care should be taken when making this adjustment so as not to disrupt any critical activity.  Also, consider directly interfacing with the machine rather then using Remote Desktop Control.
  3. Modifying these settings for most NIC's does not require a reboot to engage the setting.  But if this is a great concern, check with your NIC vendor to see if a reboot is necessary, or experiment on a non-production machine before modifying a production machine.


How to observe the Offload State of a TCP connection:

The command netstat with the switch "-t"  can be used to show the Offload State of a network connection.
Example Output from the netstat -ab -t command:
Proto Local Address Foreign Address State      PID Offload State
TCP   server:epmap  server:0        LISTENING  816 InHost
Proto Local Address  Foreign Address  State       Offload State
TCP   Server:netbios name:2151        ESTABLISHED Offloaded

Flow Control Recommendations for NICs:

What is Flow Control?
"Flow Control" is a parameter that allows the user to enable or disable the receipt or transmission of PAUSE frames. PAUSE frames enable the adapter and the switch to control the transmit rate. The side that is receiving the PAUSE frame temporarily stops transmitting.
It is recommended to set Flow Control on the NIC to "Generate and Respond".  Flow Control can also be modified within the Network Card Properties > Advanced tab.  See Example 3.


Terms of use for this information are found in Legal Notices.



Did this article answer your question or resolve your issue?


Did this article save you the trouble of contacting technical support?


How can we make this article more helpful?

Email Address (Optional)