Backups fail with Status 25 "cannot execute cmd on client" if the client PBX port 1556 is not contactable

Problem

Backups of a 8.1 (or later) client fail with status: 25: cannot connect on socket, with a message of "cannot execute cmd on client"

Other potential symptoms include:

• bptestbpcd to the client will connect
• The same client worked with earlier versions of NetBackup
   

Error Message

Extract from Job Details:    
            
Jun 20, 2019 2:58:31 AM - Info bpbrm (pid=8928) starting bpbkar32 on client
Jun 20, 2019 3:02:36 AM - Error bpbrm (pid=8928) cannot execute cmd on client Client_Name
Jun 20, 2019 3:02:42 AM - Info bpbkar32 (pid=0) done. status: 25: cannot connect on socket
Jun 20, 2019 3:02:42 AM - end writing
cannot connect on socket  (25)

bptestbpcd output shows success to client, but is connecting via the vnetd port 13724, not the PBX port 1556:

bptestbpcd -host Client_Name
1 1 1
127.0.0.1:57273 -> 127.0.0.1:57274 PROXY 192.168.1.1:57271 -> 192.168.1.2:13724
127.0.0.1:57278 -> 127.0.0.1:57279 PROXY 192.168.1.1:57276 -> 192.168.1.2:13724

If you request bptestbpcd to make the same type of connections as used during a client backup, the third connection (which is a vnet-auth-only connection) will fail as it must be routed through PBX.

bptestbpcd -client Client_Name -auth_only
1 1 1
127.0.0.1:41447 -> 127.0.0.1:40430 PROXY 192.168.1.1:2497 -> 192.168.1.2:13724
127.0.0.1:44513 -> 127.0.0.1:53336 PROXY 192.168.1.1:2496 -> 192.168.1.2:13724
<16>bptestbpcd main: Function bpcr_new_standard_socket_rqst(kielvm2-bk) failed: 25
<16>bptestbpcd main: Function bpcr_disconnect_rqst(kielvm2-bk) failed: 25
cannot connect on socket
 

Cause

When doing a backup, the master/media first tries to connect to the client on the PBX port (1556)
For NetBackup 8.0 and earlier, if this port is not contactable, it will fall back to the legacy vnetd port (13724).

However, for NetBackup 8.1 and later, failing over to the vnetd port is no longer possible for certain types of connections, and the backup will fail with the error noted above.

The bptestbpcd output confirms the issue, as it shows the connection to the client using the vnetd port (13724) instead of the expected PBX port (1556), and the connection type required for the backup cannot succeed.

See the Network Ports Reference Guide for more details on the required ports and required network protocols TLS/SSL

NOTE: If running Sharepoint backups, while the ClientName in the policy is the Sharepoint frontend machine, backups may also need to connect to other machines in the Sharepoint farm, so connection to all machines in the farm should be checked, as the issue may be on one of those  and not the front end server itself.
 

Solution

1. On the client, check if the pbx service is running and is listening on the correct port:

Unix: /usr/openv/netbackup/bin/bpps -x

Look for /opt/VRTSpbx/bin/pbx_exchange

Windows: Open the Services panel

Look for the service "Veritas Private Branch Exchange" and check it's status.

Test on all clients for PBX connectivity:   telnet Client_name 1556

If pbx isn't' running or listening on 1556:

• Stop Netbackup services /usr/openv/netbackup/bin/goodies/netbackup stop
• Stop / start pbx  /opt/VRTSpbx/bin/vxpbx_exchanged stop  then /usr/opt/VRTSpbx/bin/vxpbx_exchanged start
• Start Netbackup services. /usr/openv/netbackup/bin/goodies/netbackup start
   

2. If pbx is running on the client and local telnet to 1556 works, check if the master/media can connect to the client on the pbx port

telnet Client_name 1556

  (If 'telnet' command is missing on Linux / NetBackup Appliance, use: 'curl -v telnet://<Server_name_or_IP>:1556'  )

If this shows the port is not open, this will need to be opened on the network or OS level.

Expected behaviour

• telnet Client_name 1556 should be able to connect to pbx
• bptestbpcd to the client should show the connection on port 1556:
• The vnet-auth-only connection will also be successful.

bptestbpcd -client Client_Name -auth_only
1 1 1
127.0.0.1:58011 -> 127.0.0.1:58012 PROXY 192.168.1.1:58009 -> 192.168.1.2:1556
127.0.0.1:58015 -> 127.0.0.1:58016 PROXY 192.168.1.1:58013 -> 192.168.1.2:1556
192.168.1.1:58016 -> 192.168.1.2:1556
 

3.Be sure that TLS/SSL protocol is permitted on port 1556 as referenced in the firewall guide in the Related Articles.

   ----OR----

4. Alternative: Enable Resilient Network for the affected remote NetBackup client which will force the NetBackup Primary (master) and media servers to communicate with the remote NetBackup client's VNETD TCP port 13724. Note: for certain types of connections, the NetBackup job may still fail. In that case, refer to steps 1-3 and ensure that the PBX TCP port 1556 is open bi-drectionally between the NetBackup servers and remote NetBackup clients. Examples of the connections that are not supported for Resilient Network are:
   ■ Clients that back up their own data (deduplication clients and SAN clients)
   ■ Granular Recovery Technology (GRT) for Exchange Server or SharePoint Server
   ■ NetBackup nbfsd process (used in a variety of GRT type jobs).
 

- To enable Reslient Network for a client via command-line from the Primary (master) server, run:

/usr/openv/netbackup/bin/admincmd/resilient_clients on clienthostnamehere

- Or to enable Resilient Network from the NB java-based admin console:

1. In the NetBackup Administration Console, expand NetBackup Management > Host Properties > Master Servers (later versions this is renamed to Primary Servers) in the left pane.
2. In the right pane, select the primary server on which to specify properties.
3. On the Actions menu, click Properties.
4. In the properties dialog box left pane, select Resilient Network.
5. In the Resilient Network dialog box, use the following buttons to manage resiliency for clients:

To add resilient settings

     - Click Add.
            The Add Resilient Network Settings dialog box appears

     - Enter a client host name, an IP address
            If you specify the client host by name, it is recommended that you use the fully qualified domain name.

     - Ensure that the Resiliency On option is selected.

     - Click Add.
           When you finish adding network settings, click Close.