13: Some of the clients behind the firewall are failing with a Status 13 file read failure / 131 - Connection reset by peer

Problem

Status code: 13
  131 - Connection reset by peer
 

Error Message

bpbrm  log:  

13:00:05.827 [16042] <2> bpbrm read_backup_start: from client ClientA: read client start message
13:00:05.884 [16042] <2> bpbrm handle_backup: ADDED FILES TO DB FOR ClientA_1325703601 1 /opt/
13:05:07.496 [16042] <16> bpbrm readline: socket read failed: errno = 131 - Connection reset by peer
13:05:07.540 [16025] <2> bpbrm brm_sigcld: bpbrm child 16042 exit_status = 13, signal_status = 0
 

The socket read failure is 5 minutes after the last update to the media server

Cause

Firewall has a 300 second idle session timeout configured

One of the client backup ports may have been idle longer than 300 seconds.

The client opens a DATA port for file data to storage device (to bptm / bpdm daemon) and a NAME port for file metadata  (to bpbrm daemon).
 

bpbrm Log message about the 2 sockets opened for the backup:
13:00:05.594 [16042] <2> bpbrm start_bpcd_stat: DATA_SOCK from bpcr = 10
13:00:05.594 [16042] <2> bpbrm start_bpcd_stat: NAME_SOCK from bpcr = 11

The NAME port is usually less active (usually during large file backups) and backup will stop once it tries reading or writing to the closed port.

Solution

Extend the 300 second firewall idle session timeout to 7500 seconds or greater.

Another option is to change the TCP keepalive setting on the master and media server operating systems to a frequency that is less than the current firewall idle port timeout setting.

Yet another option if the version of NetBackup is 7.5 or greater: enable Resilient Network for the affected client. To enable from the NetBackup administration console: Host Properties of the master server > Resilient Network, click Add, add the client hostname as it appears in the backup policy client list.

....or via command-line:
• Windows: install_path\NetBackup\bin\admincmd
• UNIX: /usr/openv/netbackup/bin/admincmd
# resilient_clients status | on | off <nbuclientname>
ex. resilient_clients on mynbuclientname

Note: Specify client name (same as in policy client list), or the client IP address, or a network address. Try to refrain from enabling this for entire network addresses where too many clients (100's of clients) would now be enabled.


About WAN Resiliency
https://www.veritas.com/content/support/en_US/article.100026546

Applies To

NetBackup, any version.
- network hardware firewall
- or network load balancer hardware