NDMP backups for Large Volumes fails with Status 40 after two hours

Problem

Large volume NDMP backups fail with Status 40 after exactly two hours, showing specific errors.

Error Message

Job details in Activity Monitor show errors:

5:33:41 PM - connecting
5:33:41 PM - connected; connect time: 00:00:00
5:33:58 PM - begin writing
6:21:21 PM - Error bpbrm(pid=2004) could not write FILE ADDED message to stderr
6:21:32 PM - Error bpbrm(pid=2004) could not write FILE ADDED message to stderr
6:21:45 PM - Error bpbrm(pid=2004) could not write FILE ADDED message to stderr    
6:21:56 PM - Error bpbrm(pid=2004) could not write FILE ADDED message to stderr    
6:22:01 PM - Error bpbrm(pid=2004) could not write FILE ADDED message to stderr    
6:22:08 PM - Error bpbrm(pid=2004) could not write FILE ADDED message to stderr    
6:22:17 PM - Error bpbrm(pid=2004) could not write FILE ADDED message to stderr    
6:22:27 PM - Error bpbrm(pid=2004) could not write FILE ADDED message to stderr    
6:22:32 PM - Error bpbrm(pid=2004) could not write FILE ADDED message to stderr

Cause

The default TCP keep alive packets from the media server were being sent every two hours, which is insufficient.

Solution

Things to check...

1. Check Firewall Settings: Ensure there is no firewall between the Filers and the NetBackup Server.
   • If a firewall exists, increase the idle socket timeout to an interval greater than 2 hours.
2. Adjust TCP Keepalive Settings: Decrease TCP keepalive settings on the Master and Media Server.
   • Confirm that the settings are less than any idle socket timeout configured within the network.
   • For more information, refer to the KB article on ‘TCP Keepalive Best Practices’ (100028680) in Related Articles below.
3. Minimize Load on bpbrm Process: Create the MAX_ENTRIES_PER_ADD touch file on each media server that backs up the NDMP device. This will update the file list less frequently.
   • Example Syntax:
     • UNIX/Linux: echo 50000 > /usr/openv/netbackup/MAX_ENTRIES_PER_ADD
     • Windows: cd <install_path\NetBackup\
       echo > 50000 MAX_ENTRIES_PER_ADD
   • Note:
     • MAX_ENTRIES_PER_ADD specifies how many files should be backed up before recording the information in the catalog database.
     • The default is 5000. The maximum value is 100000.