Search <book_title>...

Veritas NetBackup™ Troubleshooting Guide

Last Published: 2020-09-13

Product(s): NetBackup (8.3)

Troubleshooting backup failure issues with NAT clients or NAT servers

Backup fails with the following error: bpbrm (pid=31553) cannot send mail because BPCD on host exited with status 21: socket open failed

This issue may occur because of one of the following reasons:

Media server cannot connect to the NetBackup Messaging Broker (or nbmqbroker) service.
The nbmqbroker service may not be up and running on the master server.
The NAT client is not configured to accept the reverse connection.
The client is not a NAT client.
The client is 8.1.2 or earlier.
Port configuration for the nbmqbroker service is updated.
The master server services are restarted.

Cause 1

Media server cannot connect to the nbmqbroker service.

Cause 2

The nbmqbroker service may not be up and running on the master server.

Cause 1 and Cause 2 have the same solution as follows:

Check the bpbrm logs on the media server at Install_Path/logs/bpbrm.
Check the nbmqbroker log file at:
UNIX: /usr/openv/mqbroker/logs
Windows: Install_Path/mqbroker/logs
Ensure that the nbmqbroker service is running on the master server. Use the following commands:
- Run the bpps command.
- Run the bptestbpcd -host hostname command from the master or media server and check the admin logs at Install_Path/logs/admin.

Cause 3: The NAT client or NAT server is not configured to accept the reverse connection

Do the following:

Check the subscriber logs at:
UNIX: usr/openv/logs/nbsubscriber
Windows: Install_Path/logs/nbsubscriber
Check the vnetd logs at Install_Path/logs/vnetd.
Run the bptestbpcd -host hostname command on the master or media server and check the admin logs at Install_Path/logs/admin.
Run the nbmqutil -publish -master hostname -message message_text -remoteHost hostname command.
Ensure that the ACCEPT_REVERSE_CONNECTION configuration option is set to TRUE using the nbgetconfig command.
Check the subscriber service is running on the NAT client by running the bpps command.

Cause 4: The client is not a NAT client

Do the following:

Ensure that the ENABLE_DIRECT_CONNECTION configuration option is set to TRUE on the master or media server using the nbgetconfig command.

Cause 5: The client is 8.1.2 or earlier

Do the following:

Ensure that the ENABLE_DIRECT_CONNECTION configuration option is set to TRUE on the master or media server using the nbgetconfig command.

Cause 6: Port configuration for the nbmqbroker service is updated

Do the following:

Wait until the cache is cleared.
Clear host cache on the media server using the bpclntcmd -clear_host_cache command.

Cause 7: The master server services are restarted

Do the following:

Check the subscriber service logs at:
UNIX: usr/openv/logs/nbsubscriber
Windows: Install_Path/logs/nbsubscriber
Wait until the subscriber service starts on the client.
Restart the subscriber service.

Backup fails with the following error: bpbrm (pid=9880) bpcd on host exited with status 48: client hostname could not be found

This issue may occur because of one of the following reasons:

The NAT client's host name is not mapped to it's host ID.
Host ID that is associated with the client is null or is not valid.

Do the following:

Check the bpbrm logs at Install_Path/logs/bpbrm
Check the existing host ID-to-host name mapping of the client by running the Install_Path/bin/admincmd/nbhostmgmt -li -json command on the master or media server.
If the client name is not mapped to the host ID, add a new name for the client and map it to existing host ID using the Install_Path/bin/admincmd/nbhostmgmt -add -hostid hostid -mappingname hostname command.
Clear host cache on the client using Install_Path/bin/bpclntcmd -clear_host_cache.

Backup takes too long to complete

This issue may occur because of one of the following reasons:

Client's configuration file (bp.conf file on UNIX or Windows registry) contains wrong media server entry.
The ENABLE_DATA_CHANNEL_ENCRYPTION option is not set to FALSE on the NAT host.

Cause 1: Client's configuration file contains wrong media server entry

Do the following:

Run the the Install_Path/bin/admincmd/bptestbpcd -host hostname from the master or media server and check the admin logs at Install_Path/logs/admin.
Add the media server name in the /etc/hosts file on the client.
Add the media server name in the configuration file on the client using the nbsetconfig command.

Cause 2: The ENABLE_DATA_CHANNEL_ENCRYPTION option is enabled

Do the following:

Set the ENABLE_DATA_CHANNEL_ENCRYPTION to FALSE using the nbsetconfig command.

Backup fails as the job is hung and no new job is triggered for the policy

This issue may occur because of the following reason:

The NAT host awaits an incoming message, but the nbmqbroker service has closed the client connection, and client cannot detect the closed connection.

Do the following:

Check the client logs to see if it contains the following message:

Trying to get Message from MQ Broker:[master server name]

Check the current heartbeat value that is set for the SUBSCRIBER_HEARTBEAT_TIMEOUT configuration option on the server. Use the nbgetconfig command.
Set the SUBSCRIBER_HEARTBEAT_TIMEOUT option value to minimum so that the client can detect a closed connection.
Restart the subscriber service on the client.

Backup or restore jobs fail after CLIENT_CONNECT_TIMEOUT

This issue may occur because of the following reason:

Subscriber was not able to establish the reverse connection with media server.
Message is delivered by publisher but subscriber did not receive the message.

Do the following:

Check the subscriber service logs to ensure that the subscriber service is able to connect to the PBX Transient ID.
Check the subscriber service logs to ensure that the publisher message is delivered to the subscriber.
Log message:
```
Got Message from MQ Broker:[<message>] with return:<status code> total timeout,reset:<timeout reset>
```

Status of NAT media server is down after the services are restarted

Do the following:

Run the following command on the master server:
Install_Path/bin/admincmd/bptestbpcd -host host_name
Check the logs at Install_Path/logs/admin.
Check if the media server is offline using the NetBackup Administration Console. Go to Media and Device Management > Devices > Media Servers.
If the master server service is restarted, restart the media server and wait for the media server to be online.
Check if the subscriber logs of the media server are ready to receive connection messages if the log level is set to a value greater than 1. For example:
Log message for the disconnected state: Retrying connection stopped for n seconds with attempt:m
Log message for the connected state: Successfully connected to MQ Broker: master server host with Host UUID NAT host ID