STATUS CODE 58: "Can't connect to client". Backups fail with status code 58 after enabling Veritas Security Services (VxSS) and NetBackup Access Control (NBAC).

Problem
STATUS CODE 58: "Can't connect to client". Backups fail with status code 58 after enabling Veritas Security Services (VxSS) and NetBackup Access Control (NBAC).

Error Message
Media Server Log Files:

The /usr/openv/netbackup/logs/bpbrm/log.<date> file can show errors when the network connection to the client is interrupted.
<16> get_long: (1) cannot read (byte 1) from network: Interrupted system call (4)
<16> bpcr_connect: bpcd protocol error - failed to read connected message.
<16> bpbrm start_bpcd: cannot connect to nbu_client, Interrupted system call (4)
<2> bind_on_port_addr: bound to port 54702
<2> check_authentication: no authentication required
<2> bpbrm kill_child_process: start
<4> bpbrm Exit: client backup EXIT STATUS 58: can't connect to client

Client Log Files:

The /usr/openv/netbackup/logs/bpcd/log.<date> on the client can show the following if a hostname resolution problem exists.
<8> bpcd peer_hostname: gethostbyaddr failed : HOST_NOT_FOUND (1)
<16> bpcd peer_hostname: gethostbyaddr failed to return peer host, herrno = 1
<16> bpcd main: Couldn't get peer hostname
<2> bpcd main: offset to GMT 28800
<2> logconnections: BPCD ACCEPT FROM x.x.x.x.32855 TO x.x.x.x.13782

Cause

A Status 58 will occur when the server cannot connect to the client.  This can occur for standard backups when there are problems with hostname resolution or network routing.  This can also occur during catalog backups when the master server cannot make a network connection to a media server in order to read the catalog data.  This can also happen when NetBackup Access Control (NBAC) is configured incorrectly.

Solution
    • Most causes for a Status 58 are due to hostname resolution errors or network connection errors.  Page 26 of the Troubleshooting Guide has a section for "Resolving Network Communications Problems."

Verify the forward and reverse hostname lookups are working for the client.  Also verify the network path is valid for the host.  

From the media server run the command:
# telnet <client_hostname> 13782
This should connect to the client and display a blank line.  To return to the prompt, press <Enter>.  
If a "connection refused" or a "shared library error" appears this would indicate a network or operating system error is preventing a proper connection to bpcd on the client.

    • Another situation in which a status code 58 could occur is if NetBackup Access Control (NBAC) is configured incorrectly. The remainder of this article is an example of such a misconfiguration and an explanation of how to correct it.

A situation could exist where the NetBackup master server's bp.conf file could contain the following NBAC related entries:

USE_VXSS = AUTOMATIC
VXSS_NETWORK = northpeak.min.veritas.com REQUIRED
VXSS_NETWORK = dali.min.veritas.com REQUIRED
AUTHENTICATION_DOMAIN = northpeak.min.veritas.com "passwd lookups for northpeak" PASSWD northpeak 0
AUTHENTICATION_DOMAIN = min.veritas.com "NIS lookups for min.veritas.com" NIS northpeak 0
AUTHENTICATION_DOMAIN = ENTERPRISE "Windows Domain lookups for ENTERPRISE" WINDOWS ferello.min.veritas.com 0
AUTHORIZATION_SERVICE = northpeak 0
These are the typical entries that would be expected for an environment using NBAC. The basic principle behind each of these entries is as follows:

The following indicates that VxSS/NBAC should be automatically detected and used:
USE_VXSS = AUTOMATIC
The following two entries indicate that for the hosts specified, VxSS/NBAC must be used:
VXSS_NETWORK = northpeak.min.veritas.com REQUIRED
VXSS_NETWORK = dali.min.veritas.com REQUIRED
The following three entries indicate which host is performing each type of authentication:
AUTHENTICATION_DOMAIN = northpeak.min.veritas.com "passwd lookups for northpeak" PASSWD northpeak 0
AUTHENTICATION_DOMAIN = min.veritas.com "NIS lookups for min.veritas.com" NIS northpeak 0
AUTHENTICATION_DOMAIN = ENTERPRISE "Windows Domain lookups for ENTERPRISE" WINDOWS ferello.min.veritas.com 0
The following indicates which host is the authorization server:
AUTHORIZATION_SERVICE = northpeak 0

    • If the first entry described above, "USE_VXSS" is either removed or commented out from the bp.conf file, catalog backups could fail with a status code 58.

In such a situation, here are the log messages that would appear:

From the admin log on the server where the catalog backup was initiated, you would see entries similar to the following:

13:34:43.368 [8271] <2> dump_data: processing path dali:/usr/openv/netbackup/db
13:34:43.368 [8271] <2> dump_data: /usr/openv/netbackup/bin/bpbkar bpbkar -IEL -nfsok -dt 0 -ru root -nocont -no_security /usr/openv/netbackup/db (host dali)
13:34:43.369 [8271] <2> nb_getsockconnected: host=dali service=bpcd address=10.82.105.97 protocol=tcp reserved port=13782
13:34:43.370 [8271] <2> nb_bind_on_port_addr: bound to port 626
13:34:43.370 [8271] <2> nb_getsockconnected: Connect to dali on port 626
13:34:43.371 [8271] <2> logconnections: BPCD CONNECT FROM 10.82.105.72.626 TO 10.82.105.97.13782
13:34:43.375 [8271] <2> vauth_authentication_required: vauth_comm.c.793: no methods for address: no authentication required
13:34:43.379 [8271] <2> vauth_connector: vauth_comm.c.193: no methods for address: no authentication required
13:34:43.379 [8271] <2> bpcr_authenticate_connection: no authentication required
13:34:43.379 [8271] <2> nb_bind_on_port_addr: bound to port 843
13:34:43.723 [8271] <2> bpcr_connect: caller requires no VxSS, bpcd requires Vxss.
13:34:43.723 [8271] <2> bpcr_connect: Not using VxSS authentication
13:34:43.726 [8271] <2> get_long: (2) premature end of file (byte 1)
13:34:43.726 [8271] <2> bpcr_connect: bpcd protocol error - failed to read connected message.
13:34:43.727 [8271] <2> set_job_details: Sending Tfile jobid (3814)
13:34:43.728 [8271] <2> set_job_details: LOG 1094841283 16 bpbackupdb 8271 NB database backup cannot connect to dali
13:34:43.728 [8271] <2> set_job_details: Done
13:34:43.731 [8271] <16> start_command: NB database backup cannot connect to dali
To correct this situation, it is necessary to correctly configure VxSS/NBAC. It could be as simple as ensuring that both hosts in question have the same configuration options for when to use VxSS.

In the example above, re-adding the USE_VXSS = AUTOMATIC entry to the bp.conf file corrects the status code 58 problem.