STATUS CODE 25: Troubleshooting procedures for client Status 25 errors.

Problem

The master server is getting a status code 25 (cannot connect on socket) error when attempting to bring up the client host properties using the GUI or remote admin console. 

Cause

A status 25 error results from either a TCP SYN request sent to a client from the NetBackup server that was not acknowledged or the server was not resolvable so a TCP SYN request was not sent.  The majority of the causes of this issue are due to the client not listening on the bpcd or vnetd ports, the master server failing to resolve the client by hostname, or the client can not resolve the NetBackup server by its IP address.  For NetBackup clients running 6.x release, although the vnetd daemon is used for incoming connections, it is still required that bpcd perform the hostname compare to authenticate the NetBackup server.

Solution

Troubleshooting:
 

Step 1:

When troubleshooting status 25 errors on a NetBackup client, verify that the client was working prior to the issue. If it had been working try to determine what changes may have been made to the client server's OS or the network links.

Step 2:

In 6.x and above environments you can use the command- bptestbpcd to verify you can connect to both the vnetd and bpcd ports on the client server.

e.g. bptestbpcd -verbose -debug -client <client hostname>

Note: See the Related Article linked below for additional details on use of the bptestbpcd command.

Step 3:

If the master server can not connect to the client when trying to access the client host properties, below are steps you can take to further troubleshoot this issue:

[Note: It is best to rely on the Netbackup bpclntcmd commands (rather than nslookup) to test hostname and IP address resolutions. This way you can verify how Netbackup sees the servers based on the hostnames and IP addresses]

1. To test the master/media server resolution of the client server hostname run the following command:
   
   • <install path>/netbackup/bin/bpclntcmd -hn <client hostname>
2. Since reverse lookups is part of the NBU server to client connections make sure the client can also be resolved by its IP address:
   • <install path>/netbackup/bin/bpclntcmd -ip <client IP address>
3. On the client test the resolution of the NBU servers by issuing the same commands. These commands should be run against the master and all of the media servers that may be trying to backup the client server:
   • <install path>/netbackup/bin/bpclntcmd -hn <NBU server hostname>
   • <install path>/netbackup/bin/bpclntcmd -ip <NBU server IP address>
4. Verify you are able to "ping" the client's IP address from the NBU server. If this fails consult with your Network Administrator and client server System Administrator to resolve the layer 3 or IP network connectivity.
5. Double check the server's NIC's IP address and netmask to ensure they are configured correctly.

Step 4:

A very useful command to help with testing client and server resolution is the command bpclntcmd -pn when run from a NBU client or media server.

This command does the following:

1. The client retrieves the first server listed in it's server's list (this should be the master server) does a forward lookup of that hostname with 'gethostbyname call' to DNS or a host file.
2. Once that is successful the message-"expecting response from (master server hostname)". is displayed. If the forward lookup fails or the client does not have the bprd port 13720 defined (in the registry for Windows clients or in /etc/services for UNIX clients) this message is not displayed. To troubleshoot create a bplist log on the client at verbose 5 and rerun the command.
3. The client connects to the bprd port on the master server and the master server performs two functions:
   
   1. a simple reverse lookup of the incoming IP address (which returns the first hostname returned by bpclntcmd)
   2. Then  the policy database is checked for the hostname it uses when backing up the client server. (This is the second hostname returned by the command)

An example of the output: (In this example the client hostname is dotto.veritas.com and the master server is hal9000.veritas.com)

C:\Program Files\VERITAS\NetBackup\bin>bpclntcmd -pn
expecting response from server hal9000.veritas.com
dotto.veritas.com dotto.veritas.com 10.82.110.6 3412

The bplist log on the client at verbose 5 shows the attempt to resolve the master server and the connection request to the NBU server.
The bprd log on the master server shows the incoming connection and the attempt to connect to the client using the client host properties.

Step 5:

Ensure the client server has the bpcd and vnetd port in listening mode using the command netstat -a:

On Windows:

 TCP    dotto:vnetd       dotto.veritas.com:0  LISTENING
 TCP    dotto:bpcd        dotto.veritas.com:0  LISTENING

If a Windows client is not listening on the ports verify bpinetd is running on the client using task manager.

On UNIX

# netstat -a |grep bpcd
     *.bpcd               *.*                0      0 49152      0 LISTEN

#netstat -a |grep vnetd
     *.vnetd              *.*                0      0 49152      0 LISTEN

If the client is not listening verify BPCD and VNETD are defined in /etc/service and inetd:

# vi /etc/services
         bpcd 13782/tcp bpcd
         vnetd 13724/tcp vnetd

 # vi /etc/inetd.conf
         bpcd stream tcp nowait root /usr/openv/netbackup/bin/bpcd bpcd
         vnetd stream tcp nowait root /usr/openv/bin/vnetd vnetd

If the /etc/services file for bpcd shows it is using tcpd then there is a TCP wrapper on the bpcd port 13782.  This can been seen in inetd.conf as follows:

bpcd stream tcp nowait root /usr/local/bin/tcpd /usr/openv/netbackup/bin/bpcd bpcd

If a TCP wrapper is in use the hosts.alow file would have to be modified to allow connections into the Netbackup ports.

Step 6:

Locally on the client create a bpcd log and increase the verbose level to 5.

On the client server to test whether the bpcd binary is executable and will generate a log issue the following command:

UNIX:

<install path>netbackup/bin/bpcd

Windows

<install path>program files\VERITAS\netbackup\bin\bpcd

This will generate a bpcd log entry which proves the binary is executable and generates a log entry, similar to the following:

16:47:45.986 [5296.3376] <2> bpcd main: offset to GMT 21600
16:47:45.986 [5296.3376] <2> bpcd main: Got socket for input 3
16:47:46.017 [5296.3376] <2> logconnections: getsockname(3) failed: 10038
16:47:46.017 [5296.3376] <16> bpcd setup_sockopts: setsockopt 1 failed: h_errno 10038
16:47:46.017 [5296.3376] <2> bpcd main: setup_sockopts complete
16:47:46.158 [5296.3376] <2> vauth_acceptor: ..\libvlibs\vauth_comm.c.332: Function failed: 17 0x00000011
16:47:46.158 [5296.3376] <16> bpcd main: authentication failed: 17

[Note: It is normal to have it end with the <16> bpcd main: authentication failed: 17 error. ]

Then test the bpcd port locally on the client server from the command prompt.  Run this telnet test using the loopback interface:

telnet localhost 13782 or telnet 127.0.0.1 13782

This command should generate a log that looks like this:

16:49:35.352 [3336.4360] <2> bpcd main: offset to GMT 21600
16:49:35.352 [3336.4360] <2> bpcd main: Got socket for input 376
16:49:35.352 [3336.4360] <2> logconnections: BPCD ACCEPT FROM 127.0.0.1.3845 TO 127.0.0.1.13782
16:49:35.352 [3336.4360] <2> bpcd main: setup_sockopts complete
16:49:35.414 [3336.4360] <2> bpcd peer_hostname: Connection from host localhost (127.0.0.1) port 3845
16:49:35.414 [3336.4360] <2> bpcd valid_server: comparing hal9000.veritas.com and localhost
16:49:35.414 [3336.4360] <4> bpcd valid_server: localhost is not a master server
16:49:35.414 [3336.4360] <16> bpcd valid_server: localhost is not a media server either
16:49:39.189 [3336.4360] <16> bpcd main: read failed: The operation completed successfully.

[Note: It generates the <16> error because it does not understand what telnet is and also fails to authenticate the hostname 'localhost' as a NBU server.]

For Linux clients if they are missing a library file required by bpcd or vnetd you would get this type of error message-

telnet localhost bpcd
Trying 127.0.0.1...
Connected to clientname.domainname.com
Escape character is '^]'.
bpcd: error while loading shared libraries: libstdc++-libc6.2-2.so.3:
cannot open shared object file: No such file or directory

In this instance contact the OS vendor to obtain the required library file.

Step 7:

Test telnet from the master server to the client's bpcd port:

telnet [client hostname] 13782

That should generate a log entry as seen below showing the master server being accepted (comments in italics for clarity):

16:52:46.077 [1160.5436] <2> bpcd main: offset to GMT 21600
16:52:46.077 [1160.5436] <2> bpcd main: Got socket for input 400
**The client sees the incoming connection from the master server using the IP address 10.82.105.254:
16:52:46.077 [1160.5436] <2> logconnections: BPCD ACCEPT FROM 10.82.105.254.44554 TO 10.82.110.6.13782 
16:52:46.077 [1160.5436] <2> bpcd main: setup_sockopts complete
**Performs a reverse lookup of the incoming IP address and gets the hostname:
16:52:46.092 [1160.5436] <2> bpcd peer_hostname: Connection from host hal9000.veritas.com (10.82.105.254) port 44554
**Then compares the hostname to the server list on the client:
16:52:46.092 [1160.5436] <2> bpcd valid_server: comparing hal9000.veritas.com and hal9000.veritas.com
**The hostname compare succeeds:
16:52:46.092 [1160.5436] <4> bpcd valid_server: hostname comparison succeeded
16:52:49.476 [1160.5436] <16> bpcd main: read failed: The operation completed successfully.

If any of these telnet tests fails to generate a log entry then there is something outside of NBU that is preventing access to the client's port. Most likely firewall software or a TCP wrapper was placed on the ports. Check what services the client is running for Windows clients (Try Turning off MS firewall software for a quick test) or check the hosts.allow and hosts.deny files on the UNIX clients (typically are in the /etc directory e.g. in /etc/hosts.allow)

Windows 2008 clients may have the Domain Firewall, Private Firewall, and Public Firewall turned on. Either disable these firewalls in Windows firewall properties. Or create exceptions for ports 13724, 13782, and exceptions, if necessary for Netbackup processes.

For UNIX clients you can try removing the BPCD port from inetd.conf and run the command "bpcd -standalone" to see if that gets the port listening.

Note: When testing connections to bpcd note the delta time difference between the bpcd log message
e.g. "16:52:46.077 [1160.5436] <2> logconnections: BPCD ACCEPT FROM 10.82.105.254.44554 TO 10.82.110.6.13782 "
and the log message "16:52:46.092 [1160.5436] <4> bpcd valid_server: hostname comparison succeeded"

There is a hard coded bprbm timeout of 1 minute. If name resolution it exceeds that timeout then try using a host file entry to avoid DNS latency.

Step 8:

If the client had been working at NBU 5.x release but now fails after upgrading to NBU 6.x, you can try using 5.x defaults for the client connection to see if that gets the failing client working again.  To do so, from the Administration Console:

1. Expand Host Properties in the left pane
2. Select Master Server in the left pane
3. Click the master server in the right pane
4. Select the Client Attributes.
5. Add the name of the client in question if it isn't listed
6. In the Connect Options tab for the client, make the following changes:
   • BPCD connect back -> "Random port"
   • Ports ->  "Reserved port"
   • Daemon connection port -> "Daemon port only"

Note: You DO NOT have to stop and restart when making changes to the master server client attribute tab. Just simply click apply and OK to commit those changes.

If unable to resolve this issue, place a call to Veritas Technical Support for NetBackup for help in troubleshooting this issue.