STATUS CODE 13: Backups fail with Status Code 13 "file read failed"

Article: 100016838
Last Published: 2020-05-16
Ratings: 7 11
Product(s): NetBackup & Alta Data Protection

Problem

STATUS CODE 13: Backups fail with Status Code 13 "file read failed", indicating that a read of a file or TCP socket has failed. Winsock errors 10053, 10054, and 10060 may also be seen in the bpbkar log on the client.


Winsock errors 10053, 10054, and 10060 are TCP/IP errors that occur at the networking layer of the Open Systems Interconnection (OSI) model. On both the client and server sides, applications such as NetBackup exist above the OSI network layer. The responsibility for establishing and maintaining a reliable network connection belongs to the network and system layers on each of the 2 servers.  If stable TCP/IP connections cannot be maintained, then NetBackup will be unable to continue the backup once these TCP errors are received.

Error Message

client bpbkar log file snippet:

Note: "OS Error: 100XX" is the cause of the backup failure here, but many other error codes may be present.

--------------------------------------------------------

09:21:37.151 [7860.976] <2> tar_backup_tfi::setupFileDirectives: TAR - backup filename = BACKUP G:\
12:01:51.153 [7860.976] <16> tar_tfi::processException: An Exception of type [SocketWriteException] has occured at: OS Error: 10060 (A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Expected bytes: 131072
12:01:51.153 [7860.976] <2> tar_base::V_vTarMsgW: FTL - socket write failed
12:01:51.153 [7860.976] <16> tar_backup_tfi::flush_fscp_inline_data: ERR - unable to get buffer
12:01:51.777 [7860.976] <2> tar_base::V_vTarMsgW: INF - EXIT STATUS 13: file read failed

--------------------------------------------------------

12:07:27.748 [3200.6156] <2> tar_backup_tfi::setupFileDirectives: TAR - backup filename = BACKUP H:\
12:43:37.061 [3200.6156] <16> An Exception of type [SocketWriteException] has occured at:
OS Error: 10054 (An existing connection was forcibly closed by the remote host.
Expected bytes: 131072
12:43:37.061 [3200.6156] <2> tar_base::V_vTarMsgW: FTL - socket write failed
12:43:37.061 [3200.6156] <16> tar_backup_tfi::flush_fscp_inline_data: ERR - unable to get buffer
12:43:37.061 [3200.6156] <16> bpbkar fscp_block_process: ERR - flush inline data failed
12:43:37.607 [3200.6156] <2> tar_base::V_vTarMsgW: INF - EXIT STATUS 13: file read failed
--------------------------------------------------------

Solution

The following will help prevent Status 13 errors:

1. Ensure that all the network settings throughout the environment (NICs, hubs, switches, routers, etc.) are set to full duplex, not half duplex

2. Increase the Windows client TCP packet retries to higher setting.
The following registry settings will make Windows TCP more resilient for network congestion during heavy backup loads:

Be sure to backup the registry before performing Windows Registry changes with 'regedit'

Registry Location-- 
HKey_LocalMachine\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters 

On the top bar: Click Edit> New> DWORD 32-bit value
Add the following registry values:

Value Name: TcpMaxDataRetransmissions
Data Type: DWORD 32-bit value
Once created, edit it and insert value = 15 (Decimal)

Value Name: KeepAliveTime
Data Type: DWORD 32-bit value
Once created, edit it and insert value = 750000 (Decimal)

Value Name: KeepAliveInterval
Data Type: DWORD 32-bit value
Once created, edit it and insert value = 15000 (Decimal)

Close Regedit window.

** NOTE: The registry changes require a reboot for the Windows OS to pick up the changes ** 

REFERENCE:
       TCP Keepalive Best Practices - detecting network drops and preventing idle socket timeout
       

3. Open device manager on the client server>network adapter>open the backup NIC properties>driver tab 
Note the NIC Model and Driver date. Download and install the latest NIC driver from NIC vendor website.

4. Open > NIC Advanced properties Tab. Review the list of entries containing the word 'offload' . Disable all 'offload' settings.  This will force the CPU to perform the task instead of added load on the NIC.

(Optional) - Run Windows 'netsh' command below to disable all NIC offload / chimney settings 

To view setting-
        C:\> netsh int tcp show global 
To disable the offload / chimney settings: 
        C:\> netsh int tcp set global chimney=disabled 

5. Antivirus applications can block NetBackup daemons / close the established TCP socket. Follow article for proper configuration of the Antivirus exclusion list on NetBackup servers.

General recommendations for virus scanner exclusions working with NetBackup

6. For mailbox backups, it is not recommended to run antivirus software against the mail directories due to database locking issues.  Verify if the anti-virus runs during this time frame and exclude the mail directories from being scanned if necessary.

7. For database backups, review the file system and database backups and ensure that schedules do not overlap.  Also, review any database maintenance schedules and ensure that they do not overlap with backup schedules.

If the error 13 persists, engage the network team to monitor the packet activity on the link at the time of the Windows TCP errors and resolve the issue. 
 
REFERENCE

Was this content helpful?