Veritas NetBackup™ Troubleshooting Guide

Last Published:
Product(s): NetBackup (8.2)
  1. Introduction
    1.  
      NetBackup logging and status code information
    2.  
      Troubleshooting a problem
    3.  
      Problem report for Technical Support
    4.  
      About gathering information for NetBackup-Java applications
  2. Troubleshooting procedures
    1.  
      About troubleshooting procedures
    2. Troubleshooting NetBackup problems
      1.  
        Verifying that all processes are running on UNIX servers
      2.  
        Verifying that all processes are running on Windows servers
    3.  
      Troubleshooting installation problems
    4.  
      Troubleshooting configuration problems
    5.  
      Device configuration problem resolution
    6.  
      Testing the master server and clients
    7.  
      Testing the media server and clients
    8.  
      Resolving network communication problems with UNIX clients
    9.  
      Resolving network communication problems with Windows clients
    10. Troubleshooting vnetd proxy connections
      1.  
        vnetd proxy connection requirements
      2.  
        Where to begin to troubleshoot vnetd proxy connections
      3.  
        Verify that the vnetd process and proxies are active
      4.  
        Verify that the host connections are proxied
      5.  
        Test the vnetd proxy connections
      6.  
        Examine the log files of the connecting and accepting processes
      7.  
        Viewing the vnetd proxy log files
    11. Troubleshooting security certificate revocation
      1.  
        Troubleshooting cloud provider's revoked SSL certificate issues
      2.  
        Troubleshooting cloud provider's CRL download issues
      3.  
        How a host's CRL affects certificate revocation troubleshooting
      4.  
        NetBackup job fails because of revoked certificate or unavailability of CRLs
      5.  
        NetBackup job fails because of apparent network error
      6.  
        NetBackup job fails because of unavailable resource
      7.  
        Master server security certificate is revoked
      8.  
        Determining a NetBackup host's certificate state
      9.  
        Troubleshooting issues with external CA-signed certificate revocation
    12.  
      About troubleshooting networks and host names
    13. Verifying host name and service entries in NetBackup
      1.  
        Example of host name and service entries on UNIX master server and client
      2.  
        Example of host name and service entries on UNIX master server and media server
      3.  
        Example of host name and service entries on UNIX PC clients
      4.  
        Example of host name and service entries on UNIX server that connects to multiple networks
    14.  
      About the bpclntcmd utility
    15.  
      Using the Host Properties window to access configuration settings
    16.  
      Resolving full disk problems
    17. Frozen media troubleshooting considerations
      1.  
        Logs for troubleshooting frozen media
      2.  
        About the conditions that cause media to freeze
    18. Troubleshooting problems with the NetBackup web services
      1.  
        Viewing NetBackup web services logs
      2.  
        Troubleshooting web service issues after external CA configuration
    19.  
      Troubleshooting problems with the NetBackup web server certificate
    20. Resolving PBX problems
      1.  
        Checking PBX installation
      2.  
        Checking that PBX is running
      3.  
        Checking that PBX is set correctly
      4.  
        Accessing the PBX logs
      5.  
        Troubleshooting PBX security
      6.  
        Determining if the PBX daemon or service is available
    21. Troubleshooting problems with validation of the remote host
      1.  
        Viewing logs pertaining to host validation
      2.  
        Enabling insecure communication with NetBackup 8.0 and earlier hosts
      3.  
        Approving pending host ID-to-host name mappings
      4.  
        Clearing host cache
    22. About troubleshooting Auto Image Replication
      1. Troubleshooting Auto Image Replication
        1.  
          Targeted AIR trusted master server operation failed in case of external certificate configuration
      2.  
        About troubleshooting automatic import jobs
    23.  
      Troubleshooting network interface card performance
    24.  
      About SERVER entries in the bp.conf file
    25.  
      About unavailable storage unit problems
    26.  
      Resolving a NetBackup Administration operations failure on Windows
    27.  
      Resolving garbled text displayed in NetBackup Administration Console on a UNIX computer
    28.  
      Unable to logon to the NetBackup Administration Console after external CA configuration
    29.  
      Troubleshooting file-based external certificate issues
    30.  
      Troubleshooting Windows certificate store issues
    31.  
      Troubleshooting backup failures
    32.  
      Troubleshooting backup failure issues with NAT clients
    33.  
      Troubleshooting issues with the NetBackup Messaging Broker (or nbmqbroker) service
  3. Using NetBackup utilities
    1.  
      About NetBackup troubleshooting utilities
    2.  
      About the analysis utilities for NetBackup debug logs
    3.  
      About the Logging Assistant
    4.  
      About network troubleshooting utilities
    5. About the NetBackup support utility (nbsu)
      1.  
        Output from the NetBackup support utility (nbsu)
      2.  
        Example of a progress display for the NetBackup support utility (nbsu)
    6. About the NetBackup consistency check utility (NBCC)
      1.  
        Output from the NetBackup consistency check utility (NBCC)
      2.  
        Example of an NBCC progress display
    7.  
      About the NetBackup consistency check repair (NBCCR) utility
    8.  
      About the nbcplogs utility
    9. About the robotic test utilities
      1.  
        Robotic tests on UNIX
      2.  
        Robotic tests on Windows
  4. Disaster recovery
    1.  
      About disaster recovery
    2.  
      About disaster recovery requirements
    3.  
      Disaster recovery packages
    4.  
      About disaster recovery settings
    5.  
      Recommended backup practices
    6. About disk recovery procedures for UNIX and Linux
      1. About recovering the master server disk for UNIX and Linux
        1.  
          Recovering the master server when root is intact
        2.  
          Recovering the master server when the root partition is lost
      2.  
        About recovering the NetBackup media server disk for UNIX
      3.  
        Recovering the system disk on a UNIX client workstation
    7. About clustered NetBackup server recovery for UNIX and Linux
      1.  
        Replacing a failed node on a UNIX or Linux cluster
      2.  
        Recovering the entire UNIX or Linux cluster
    8. About disk recovery procedures for Windows
      1. About recovering the master server disk for Windows
        1.  
          Recovering the master server with Windows intact
        2.  
          Recovering the master server and Windows
      2.  
        About recovering the NetBackup media server disk for Windows
      3.  
        Recovering a Windows client disk
    9. About clustered NetBackup server recovery for Windows
      1.  
        Replacing a failed node on a Windows VCS cluster
      2.  
        Recovering the shared disk on a Windows VCS cluster
      3.  
        Recovering the entire Windows VCS cluster
    10.  
      Generating a certificate on a clustered master server after disaster recovery installation
    11.  
      About restoring disaster recovery package
    12.  
      About the DR_PKG_MARKER_FILE environment variable
    13.  
      Restoring disaster recovery package on Windows
    14.  
      Restoring disaster recovery package on UNIX
    15. About recovering the NetBackup catalog
      1.  
        About NetBackup catalog recovery on Windows computers
      2.  
        About NetBackup catalog recovery from disk devices
      3.  
        About NetBackup catalog recovery and symbolic links
      4. About NetBackup catalog recovery and OpsCenter
        1.  
          Specifying the NetBackup job ID number after a catalog recovery
      5.  
        NetBackup disaster recovery email example
      6. About recovering the entire NetBackup catalog
        1.  
          Recovering the entire NetBackup catalog using the Catalog Recovery Wizard
        2.  
          Recovering the entire NetBackup catalog using bprecover -wizard
      7. About recovering the NetBackup catalog image files
        1.  
          Recovering the NetBackup catalog image files using the Catalog Recovery Wizard
        2.  
          Recovering the NetBackup catalog image files using bprecover -wizard
      8. About recovering the NetBackup relational database
        1.  
          Recovering NetBackup relational database files from a backup
        2.  
          Recovering the NetBackup relational database files from staging
        3.  
          About processing the relational database in staging
      9.  
        Recovering the NetBackup catalog when NetBackup Access Control is configured
      10.  
        Recovering the NetBackup catalog from a nonprimary copy of a catalog backup
      11.  
        Recovering the NetBackup catalog without the disaster recovery file
      12.  
        Recovering a NetBackup user-directed online catalog backup from the command line
      13.  
        Restoring files from a NetBackup online catalog backup
      14.  
        Unfreezing the NetBackup online catalog recovery media
      15.  
        Steps to carry out when you see exit status 5988 during catalog recovery

NetBackup disaster recovery email example

A catalog backup policy can send a disaster recovery email upon completion of a catalog backup. To configure a catalog backup policy, see the NetBackup Administrator's Guide, Volume I:

http://www.veritas.com/docs/DOC5332

The following is an example of a disaster recovery email after a successful catalog backup:

From:	         NetBackup@example.com
Sent:	         Thursday, January 3, 2019 05:48
To:	           NetBackup Administrator
Subject:	      NetBackup Catalog Backup successful on host
               master.example.com status 0
Attachments:	  cat_backup_1438271286_INCR
															cat_backup_1438271286_INCR.drpkg	

Server
	master.example.com

NetBackup Version
	8.1.X

Date
	4/27/2017 05:46:45 AM

Policy
	cat_backup

Catalog Backup Status
	the requested operation was successfully completed (status 0).
WARNING: External CA-signed certificates could not be backed up. 
Refer to the following article to configure external CA-signed certificates 
on the host after disaster recovery installation: 
https://www.veritas.com/support/en_US/article.100044249

DR image file: /dr/nbu_dr_file/cat_backup_1438271286_INCR

To ensure that the NetBackup catalog data is protected through
1/3/2019 10:46:45 AM, retain a copy of each attached file, and the 
media  or files listed below:

Catalog Recovery Media
 Media Server                Disk Image Path  Image File Required
 * media-server.example.com  @aaaab           cat_backup_1438267080_FULL
 * media-server.example.com  @aaaab           cat_backup_1438271206_INCR
 * media-server.example.com  @aaaab           cat_backup_1438271286_INCR

DR file written to
	/dr/nbu_dr_file/cat_backup_1438271286_INCR

DR Package file written to
 /dr/nbu_dr_file/cat_backup_1438271286_INCR.drpkg

The CA configuration at the time of catalog backup is as follows:

The master server ch12auto28 is configured to use NetBackup and external 
CA-signed certificates.
 ECA_CERT_PATH = MY\\ch12auto28.pne.ven.veritas.com
 ECA_CRL_PATH = C:\Users\Administrator\Downloads\divgrt1.crl 
 ECA_CRL_PATH_SYNC_HOURS = 1  
 ECA_CRL_REFRESH_HOURS = 24  
 ECA_CRL_CHECK = 1  
 ECA_DR_BKUP_WIN_CERT_STORE  = YES



* - Primary Media


Catalog Recovery Procedure for the Loss of an Entire Catalog

You should create a detailed disaster recovery plan to follow should 
it become necessary to restore your organization's data in the event 
of a disaster.  A checklist of required tasks can be a tremendous tool 
in assisting associates in triage.  For example, after the facility is 
safe for data to be restored, the power and data infrastructure need 
to be verified.  When these tasks are completed, the following 
scenarios will help to quickly restore the NetBackup environment, and 
in turn, restore applications and data.

Disaster Recovery Procedure using the DR Package file and DR Image File

In the event of a catastrophic failure, use the following procedure to 
rebuild the previous NetBackup environment.

Important Notes:
- If new hardware is required, make sure that the devices contain 
		drives capable of reading the media and that the drive controllers are 
		capable of mounting the drives.
- Keep the passphrase associated with the DR Package file handy. 
		This passphrase is set before the catalog backup policy configuration 
		using the NetBackup Administration Console or the nbseccmd command.

	1. Install NetBackup.
    a. The installation procedure prompts you to confirm if this is a DR 
							scenario.
						i. On the UNIX installer, you can see a prompt as "Do you want to 
									do a disaster recovery on this master server? [y,n] (y)". 
									Select "y"
      ii. On the Windows installer click the 
										"Disaster Recovery Master Server" button.
    b. The installation procedure prompts you for the master server's DR 
							Package 
							(refer to the /dr/nbu_dr_file/cat_backup_1438271286_INCR.drpkg 
							mentioned earlier).
       Make sure that the Master Server can access the attached DR package 
							file.
    c. Type the passphrase associated with the Master Server's DR Package, 
							when prompted.
      i. The installer validates the DR package using that passphrase
      ii. In case of errors in validation, the installer aborts the 
										operation. To work around the issue, refer to the following 
										article: http://www.veritas.com/docs/000125933

	2. Configure the devices necessary to read the media listed above.
	3. Inventory the media.
	4. Make sure that the master server can access the attached DR image file.
	5. Start the NetBackup Recovery Wizard from the NetBackup Administration
	   Console.  Or, start the wizard from a command line by entering
		  bprecover -wizard.

WARNING: CRLs are not backed as part of the DR package backup. 
Refer to the following article to manually add the CRLs:
https://www.veritas.com/support/en_US/article.100044250


Disaster Recovery Procedure without the DR Image File
NOTE: ONLY ATTEMPT THIS AS A LAST RESORT If you do not have the 
attachment included with this email, use the following instructions to 
recover your catalog. (If using OpenStorage disk pools, refer to the 
Shared Storage Guide to configure the disk pools instead of step 2 and 
3 below ):

	1.  Install NetBackup.
	2.  Configure the devices necessary to read the media listed above.
	3.  Inventory the media.
	4.  Run
		    To recover from copy 1:
		    bpimport -create_db_info -stype AdvancedDisk -dp dp-advdisk
      -dv /storage/advdisk
	5.  Run:  
		   cat_export -client client1.example.com
	6.  Go to the following directory to find the DR image file
	    cat_backup_1438271286_INCR:
	    /usr/openv/netbackup/db.export/images/master.example.com/1438000000
	7.  Open cat_backup_1438271286_INCR file and find the BACKUP_ID
	    (for example: master.example.com_1438271286).
	8.  Run:  
		   bpimport [-server name] -backupid master.example.com_1438271286
	9.  Run:
		   bprestore -T -w [-L progress_log] -C master.example.com -t 35 
     -p cat_backup -X -s 1438271286 -e 1438271286 /
	10. Run the BAR user interface to restore the remaining image database 
	    if the DR image is a result of an incremental backup.
	11. To recover the NetBackup relational database, run:
		   bprecover -r -nbdb
	12. Stop and Start NetBackup.
	13. Configure the devices if any device has changed since the last
     backup.
	14. To make sure the volume information is updated, inventory the
     media to update the NetBackup database.