13: Some of the clients behind the firewall are failing with a Status 13 file read failure / 131 - Connection reset by peer

Problem

Status code: 13
  131 - Connection reset by peer
 

Error Message

BPBRM  LOG   

13:00:05.827 [16042] <2> bpbrm read_backup_start: from client ClientA: read client start message
13:00:05.884 [16042] <2> bpbrm handle_backup: ADDED FILES TO DB FOR ClientA_1325703601 1 /opt/
13:05:07.496 [16042] <16> bpbrm readline: socket read failed: errno = 131 - Connection reset by peer
13:05:07.540 [16025] <2> bpbrm brm_sigcld: bpbrm child 16042 exit_status = 13, signal_status = 0
 

The socket read failure is 5 minutes after the last update to the media server

Cause

Firewall has a 300 second idle session timeout configured

One of the client backup ports may have been idle longer than 300 seconds.

The client opens a DATA port for file data to storage device (to bptm / bpdm daemon) and a NAME port for file metadata  (to bpbrm daemon).
 

    BPBRM Log message about the 2 sockets opened for the backup
    13:00:05.594 [16042] <2> bpbrm start_bpcd_stat: DATA_SOCK from bpcr = 10
    13:00:05.594 [16042] <2> bpbrm start_bpcd_stat: NAME_SOCK from bpcr = 11

The NAME port is usually less active (usually during large file backups) and backup will stop once it tries reading or writing to the closed port.

 

Solution

Extend the 300 second firewall idle session timeout to 7500 seconds or greater
 

Another option is to change the TCP keepalive setting on the master and media server operating systems to a frequency that is less than the current firewall idle port timeout setting.

Yet another option if the version of NetBackup is 7.5 or greater: enable Resilient Network for the affected client. To enable from the NetBackup administration console: Host Properties of the master server > Resilient Network, click Add, add the client hostname as it appears in the backup policy client list.

....or via command-line:
• Windows: install_path\NetBackup\bin\admincmd
• UNIX: /usr/openv/netbackup/bin/admincmd
# resilient_clients status | on | off <nbuclientname>
ex. resilient_clients on mynbuclientname

NOTE: can specify client name (same as in policy client list), or the client IP address, or a network address. Try to refrain from enabling this for entire network addresses where too many clients (100's of clients) would now be enabled.

Resilient Network:
http://www.symantec.com/docs/HOWTO68415

About WAN Resiliency
http://www.symantec.com/docs/TECH183552

http://www.symantec.com/business/support/resources/sites/BUSINESS/content/live/TECHNICAL_SOLUTION/183000/TECH183552/en_US/About_WAN_Resiliency.pdf
 

Applies To

NetBackup, any version.
- network hardware firewall
- or network load balancer hardware

Terms of use for this information are found in Legal Notices.