Veritas NetBackup™ Troubleshooting Guide

Last Published:
Product(s): NetBackup (8.2)
  1. Introduction
    1.  
      NetBackup logging and status code information
    2.  
      Troubleshooting a problem
    3.  
      Problem report for Technical Support
    4.  
      About gathering information for NetBackup-Java applications
  2. Troubleshooting procedures
    1.  
      About troubleshooting procedures
    2. Troubleshooting NetBackup problems
      1.  
        Verifying that all processes are running on UNIX servers
      2.  
        Verifying that all processes are running on Windows servers
    3.  
      Troubleshooting installation problems
    4.  
      Troubleshooting configuration problems
    5.  
      Device configuration problem resolution
    6.  
      Testing the master server and clients
    7.  
      Testing the media server and clients
    8.  
      Resolving network communication problems with UNIX clients
    9.  
      Resolving network communication problems with Windows clients
    10. Troubleshooting vnetd proxy connections
      1.  
        vnetd proxy connection requirements
      2.  
        Where to begin to troubleshoot vnetd proxy connections
      3.  
        Verify that the vnetd process and proxies are active
      4.  
        Verify that the host connections are proxied
      5.  
        Test the vnetd proxy connections
      6.  
        Examine the log files of the connecting and accepting processes
      7.  
        Viewing the vnetd proxy log files
    11. Troubleshooting security certificate revocation
      1.  
        Troubleshooting cloud provider's revoked SSL certificate issues
      2.  
        Troubleshooting cloud provider's CRL download issues
      3.  
        How a host's CRL affects certificate revocation troubleshooting
      4.  
        NetBackup job fails because of revoked certificate or unavailability of CRLs
      5.  
        NetBackup job fails because of apparent network error
      6.  
        NetBackup job fails because of unavailable resource
      7.  
        Master server security certificate is revoked
      8.  
        Determining a NetBackup host's certificate state
      9.  
        Troubleshooting issues with external CA-signed certificate revocation
    12.  
      About troubleshooting networks and host names
    13. Verifying host name and service entries in NetBackup
      1.  
        Example of host name and service entries on UNIX master server and client
      2.  
        Example of host name and service entries on UNIX master server and media server
      3.  
        Example of host name and service entries on UNIX PC clients
      4.  
        Example of host name and service entries on UNIX server that connects to multiple networks
    14.  
      About the bpclntcmd utility
    15.  
      Using the Host Properties window to access configuration settings
    16.  
      Resolving full disk problems
    17. Frozen media troubleshooting considerations
      1.  
        Logs for troubleshooting frozen media
      2.  
        About the conditions that cause media to freeze
    18. Troubleshooting problems with the NetBackup web services
      1.  
        Viewing NetBackup web services logs
      2.  
        Troubleshooting web service issues after external CA configuration
    19.  
      Troubleshooting problems with the NetBackup web server certificate
    20. Resolving PBX problems
      1.  
        Checking PBX installation
      2.  
        Checking that PBX is running
      3.  
        Checking that PBX is set correctly
      4.  
        Accessing the PBX logs
      5.  
        Troubleshooting PBX security
      6.  
        Determining if the PBX daemon or service is available
    21. Troubleshooting problems with validation of the remote host
      1.  
        Viewing logs pertaining to host validation
      2.  
        Enabling insecure communication with NetBackup 8.0 and earlier hosts
      3.  
        Approving pending host ID-to-host name mappings
      4.  
        Clearing host cache
    22. About troubleshooting Auto Image Replication
      1. Troubleshooting Auto Image Replication
        1.  
          Targeted AIR trusted master server operation failed in case of external certificate configuration
      2.  
        About troubleshooting automatic import jobs
    23.  
      Troubleshooting network interface card performance
    24.  
      About SERVER entries in the bp.conf file
    25.  
      About unavailable storage unit problems
    26.  
      Resolving a NetBackup Administration operations failure on Windows
    27.  
      Resolving garbled text displayed in NetBackup Administration Console on a UNIX computer
    28.  
      Unable to logon to the NetBackup Administration Console after external CA configuration
    29.  
      Troubleshooting file-based external certificate issues
    30.  
      Troubleshooting Windows certificate store issues
    31.  
      Troubleshooting backup failures
    32.  
      Troubleshooting backup failure issues with NAT clients
    33.  
      Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service
  3. Using NetBackup utilities
    1.  
      About NetBackup troubleshooting utilities
    2.  
      About the analysis utilities for NetBackup debug logs
    3.  
      About the Logging Assistant
    4.  
      About network troubleshooting utilities
    5. About the NetBackup support utility (nbsu)
      1.  
        Output from the NetBackup support utility (nbsu)
      2.  
        Example of a progress display for the NetBackup support utility (nbsu)
    6. About the NetBackup consistency check utility (NBCC)
      1.  
        Output from the NetBackup consistency check utility (NBCC)
      2.  
        Example of an NBCC progress display
    7.  
      About the NetBackup consistency check repair (NBCCR) utility
    8.  
      About the nbcplogs utility
    9. About the robotic test utilities
      1.  
        Robotic tests on UNIX
      2.  
        Robotic tests on Windows
  4. Disaster recovery
    1.  
      About disaster recovery
    2.  
      About disaster recovery requirements
    3.  
      Disaster recovery packages
    4.  
      About disaster recovery settings
    5.  
      Recommended backup practices
    6. About disk recovery procedures for UNIX and Linux
      1. About recovering the master server disk for UNIX and Linux
        1.  
          Recovering the master server when root is intact
        2.  
          Recovering the master server when the root partition is lost
      2.  
        About recovering the NetBackup media server disk for UNIX
      3.  
        Recovering the system disk on a UNIX client workstation
    7. About clustered NetBackup server recovery for UNIX and Linux
      1.  
        Replacing a failed node on a UNIX or Linux cluster
      2.  
        Recovering the entire UNIX or Linux cluster
    8. About disk recovery procedures for Windows
      1. About recovering the master server disk for Windows
        1.  
          Recovering the master server with Windows intact
        2.  
          Recovering the master server and Windows
      2.  
        About recovering the NetBackup media server disk for Windows
      3.  
        Recovering a Windows client disk
    9. About clustered NetBackup server recovery for Windows
      1.  
        Replacing a failed node on a Windows VCS cluster
      2.  
        Recovering the shared disk on a Windows VCS cluster
      3.  
        Recovering the entire Windows VCS cluster
    10.  
      Generating a certificate on a clustered master server after disaster recovery installation
    11.  
      About restoring disaster recovery package
    12.  
      About the DR_PKG_MARKER_FILE environment variable
    13.  
      Restoring disaster recovery package on Windows
    14.  
      Restoring disaster recovery package on UNIX
    15. About recovering the NetBackup catalog
      1.  
        About NetBackup catalog recovery on Windows computers
      2.  
        About NetBackup catalog recovery from disk devices
      3.  
        About NetBackup catalog recovery and symbolic links
      4. About NetBackup catalog recovery and OpsCenter
        1.  
          Specifying the NetBackup job ID number after a catalog recovery
      5.  
        NetBackup disaster recovery email example
      6. About recovering the entire NetBackup catalog
        1.  
          Recovering the entire NetBackup catalog using the Catalog Recovery Wizard
        2.  
          Recovering the entire NetBackup catalog using bprecover -wizard
      7. About recovering the NetBackup catalog image files
        1.  
          Recovering the NetBackup catalog image files using the Catalog Recovery Wizard
        2.  
          Recovering the NetBackup catalog image files using bprecover -wizard
      8. About recovering the NetBackup relational database
        1.  
          Recovering NetBackup relational database files from a backup
        2.  
          Recovering the NetBackup relational database files from staging
        3.  
          About processing the relational database in staging
      9.  
        Recovering the NetBackup catalog when NetBackup Access Control is configured
      10.  
        Recovering the NetBackup catalog from a nonprimary copy of a catalog backup
      11.  
        Recovering the NetBackup catalog without the disaster recovery file
      12.  
        Recovering a NetBackup user-directed online catalog backup from the command line
      13.  
        Restoring files from a NetBackup online catalog backup
      14.  
        Unfreezing the NetBackup online catalog recovery media
      15.  
        Steps to carry out when you see exit status 5988 during catalog recovery

Resolving network communication problems with UNIX clients

The following procedure is for resolving NetBackup communications problems, such as those associated with NetBackup status codes 25, 54, 57, and 58. This procedure consists of two variations: one for UNIX clients and another for Windows clients.

Note:

In all cases, ensure that your network configuration works correctly outside of NetBackup before trying to resolve NetBackup problems.

For UNIX clients, perform the following steps. Before you start this procedure, add the VERBOSE=5 option to the /usr/openv/netbackup/bp.conf file.

Table: Steps for resolving network communication problems with UNIX clients

Step

Action

Description

Step 1

Create debug log directories.

During communication retries, the debug logs provide detailed debug information, which can help you analyze the problem.

Create the following directories:

  • bpcd (on the master server and clients)

  • vnetd (on the master server and clients)

  • bprd (on the master server)

Use the bprd log directory to debug client to master server communication, not client to media server communication problems.

Step 2

Test a new configuration or modified configuration.

If this configuration is a new or a modified configuration, do the following:

  • Check any recent modifications to ensure that they did not introduce the problem.

  • Ensure that the client software was installed and that it supports the client operating system.

  • Check the client names, server names, and service entries in your NetBackup configuration as explained in the following topic:

    See Verifying host name and service entries in NetBackup.

    You can also use the hostname command on the client to determine the host name that the client sends with requests to the master server. Check the bprd debug log on the master server to determine what occurred when the server received the request.

Step 3

Verify name resolution.

To verify name resolution, run the following command on the master server and the media servers:

# bpclntcmd -hn client name

If the results are unexpected, review the configuration of these name resolution services: nsswitch.conf file, hosts file, ipnodes file, and resolv.conf file.

Also run the following on the client to check forward and reverse name lookup of the master server and media server that perform the backup:

# bpclntcmd -hn server name

# bpclntcmd -ip IP address of server

Step 4

Verify network connectivity.

Verify network connectivity between client and server by pinging the client from the server.

# ping clientname

Where clientname is the name of the client as configured in the NetBackup policy configuration.

For example, to ping the policy client that is named ant:

# ping ant 
    ant.nul.nul.com: 64 byte packets
    64 bytes from 199.199.199.24: icmp_seq=0. time=1. ms
    ----ant.nul.nul.com PING Statistics----
    2 packets transmitted, 2 packets received, 0% packet 
    loss round-trip (ms)  min/avg/max = 1/1/1

A successful ping verifies connectivity between the server and client. If the ping fails and ICMP is not blocked between the hosts, resolve the network problem outside of NetBackup before you proceed.

Some forms of the ping command let you ping the bpcd port on the client as in the following command:

# ping ant 1556     
    

Ping 1556 (PBX) and 13724 (vnetd) in sequence, the same sequence that NetBackup tries by default. You then know which ports are closed so that you can open them for more efficient connection tries.

Step 5

Ensure that the client listens on the correct port for the bpcd connections.

On the client, run one of the following commands (depending on platform and operating system):

netstat -a | grep bpcd
netstat -a | grep 13782
rpcinfo -p | grep 13782

Repeat for 1556 (PBX) and 13724 (vnetd). If no problems occur with the ports, the expected output is as follows:

# netstat -a | egrep '1556|PBX|13724|vnetd|13782|bpcd' | grep LISTEN
 *.1556          *.*      0      0 49152      0 LISTEN
 *.13724         *.*      0      0 49152      0 LISTEN
 *.13782         *.*      0      0 49152      0 LISTEN

LISTEN indicates that the client listens for connections on the port.

If the NetBackup processes are running correctly, the expected output is as follows:

# ps -ef | egrep 'pbx_exchange|vnetd|bpcd' | grep -v grep
root   306 1 0 Jul 18 ? 13:52 /opt/VRTSpbx/bin/pbx_exchange
root 10274 1 0 Sep 13 ?  0:11 /usr/openv/netbackup/bin/vnetd -standalone
root 10277 1 0 Sep 13 ?  0:45 /usr/openv/netbackup/bin/bpcd -standalone

Repeat the procedure on the master server(s) and media server(s), to test communication to the client.

Step 6

Connect to the client through telnet.

On the client, telnet to 1556 (PBX) and 13724 (vnetd). Check both ports to make sure that a connection is made on at least one of them. If the telnet connection succeeds, keep the connection until after you perform step 8, then terminate it with Ctrl-c.

telnet clientname 1556
telnet clientname 13724

Where clientname is the name of the client as configured in the NetBackup policy configuration.

For example,

# telnet ant vnetd
Trying 199.999.999.24 ...
Connected to ant.nul.nul.com.
Escape character is '^]'.

In this example, telnet can establish a connection to the client ant.

Repeat the procedure on the master server(s) and media server(s), to test communication to the client.

Step 7

Identify the outbound socket on the server host.

On the master server(s) and media server(s): Use the following command to identify the outbound socket that is used for the telnet command from step 6. Specify the appropriate IP address to which the server resolves the policy client. Note the source IP (10.82.105.11), the source port (45856) and the destination port (1556).

# netstat -na | grep '<client_IP_address>' | egrep '1556|13724' 
10.82.105.11.45856 10.82.104.99.1556 49152 0 49152 0 ESTABLISHED

If telnet is still connected and a socket is not displayed: Remove the port number filtering and observe the port number to which the site has mapped the service name. Check that process listens on the port number in step 5.

$ netstat -na | grep '<client_IP_address>' 
10.82.105.11.45856 10.82.104.99.1234 49152 0 49152 0 ESTABLISHED

If the socket is in a SYN_SENT state instead of an ESTABLISHED state, the server host is trying to make the connection. However, a firewall blocks the outbound TCP SYN from reaching the client host or blocks the return bound TCP SYN+ACK from reaching the server host.

Step 8

Confirm that the telnet connection reaches this client host.

On the master server(s) and media server(s), to confirm that the telnet connection reaches this client host, run the following command:

$ netstat -na | grep '<source_port>'
10.82.104.99.1556  10.82.105.11.45856  49152 0 49152 0 ESTABLISHED

One of the following conditions occurs:

  • If telnet is connected but the socket is not present: The telnet reached some other host that incorrectly shares the same IP address as the client host.

  • If the socket is in a SYN_RCVD state instead of an ESTABLISHED state, then the connection reached this client host. However, a firewall blocks the return of the TCP SYN+ACK to the server host.

Step 9

Verify communication between the client and the master server.

To verify client to master server communications, use the bpclntcmd utility. When -pn and -sv run on a NetBackup client, they initiate inquiries to the NetBackup master server (as configured in the client bp.conf file). The master server then returns information to the requesting client. More information is available about bpclntcmd.

See About the bpclntcmd utility.

The PBX, vnetd, and bprd debug logs should provide details on the nature of any remaining failure.