NetBackup™ Troubleshooting Guide

NetBackup disaster recovery email example

A catalog backup policy can send a disaster recovery email upon completion of a catalog backup. To configure a catalog backup policy, see the NetBackup Administrator's Guide, Volume I:

http://www.veritas.com/docs/DOC5332

The following is an example of a disaster recovery email after a successful catalog backup:

From:	         NetBackup@example.com
Sent:	         Tuesday, June 13, 2023 04:42
To:	           NetBackup Administrator
Subject:	      NetBackup Catalog Backup successful on host
               primary.example.com status 0
Attachments:	  cat_1686692545_FULL.drpkg	

Server
	primary.example.com

NetBackup Version
	10.3

Date
	6/13/2023 04:42:20 PM

Policy
	cat

Catalog Backup Status
	the requested operation was successfully completed (status 0).


DR image file: /usr/openv/cat_1686692545_FULL

To ensure that the NetBackup catalog data is protected through
Tue 13 Jun 2023 04:42:20 PM CDT, retain a copy of each attached file, and 
the media or files listed below:

Catalog Recovery Media
 Media Server                Disk Image Path  Image File Required
 * media-server.example.com  @aaaab           cat_1686692540_FULL
 * media-server.example.com  @aaaab           cat_1686692545_FULL
 * media-server.example.com  @aaaab           cat_1686692545_FULL

DR file written to
	/usr/openv/cat_1686692545_FULL

DR Package file written to
 /usr/openv/cat_1686692545_FULL.drpkg

The CA configuration at the time of catalog backup is as follows:


The primary server primary.example.com is configured to use NetBackup certificates.

 ECA_CRL_PATH_SYNC_HOURS = 1  
 ECA_CRL_REFRESH_HOURS = 24  
 ECA_CRL_CHECK = LEAF  



The primary server is configured to use service account: root

The primary server is configured to run with FIPS mode set to: DISABLE


* - Primary Media


Catalog Recovery Procedure for the Loss of an Entire Catalog

You should create a detailed disaster recovery plan to follow should it become 
necessary to restore your organization's data in the event of a disaster.  A 
checklist of required tasks can be a tremendous tool in assisting associates in 
triage.  For example, after the facility is safe for data to be restored, the 
power and data infrastructure need to be verified.  When these tasks are 
completed, the following scenarios will help to quickly restore the NetBackup 
environment, and in turn, restore applications and data.

Disaster Recovery Procedure using the DR Package file and DR Image File

In the event of a catastrophic failure, use the following procedure to rebuild 
the previous NetBackup environment.

Important Notes:
- If new hardware is required, make sure that the devices contain drives 
  capable of reading the media and that the drive controllers are capable of 
  mounting the drives.
- Keep the passphrase associated with the DR Package file handy. This 
  passphrase is set before the catalog backup policy configuration using the 
  NetBackup web UI or the nbseccmd command.
- If the catalog backup is encrypted using keys from an External KMS, configure 
  the External KMS in NetBackup after the installation completes and before 
  starting recovery. See the NetBackup Security and Encryption Guide for 
  information on how to configure an external KMS. 
  http://www.veritas.com/docs/DOC5332
- If this catalog backup is encrypted using a keys from the NetBackup KMS, 
  configure the NetBackup KMS and restore the required keys after the installation 
  completes and before starting recovery. See the NetBackup Security and 
  Encryption Guide for information on how to backup and restore keys from the 
  NetBackup KMS. http://www.veritas.com/docs/DOC5332

	1. Install NetBackup.
    a. The installation procedure prompts you to confirm if this is a DR 
							scenario.
						i. On the UNIX installer, you can see a prompt as "Are you currently 
         performing a disaster recovery of a primary server? [y,n] (y)". 
									Select "y"
      ii. On the Windows installer click the "Disaster Recovery Primary 
         Server" button.
    b. The installation procedure prompts you for the primary server's DR 
							Package 
							(refer to the /usr/openv/cat_1686692545_FULL.drpkg mentioned earlier).
       Make sure that the primary server can access the attached DR package 
							file.
    c. Type the passphrase associated with the DR Package, 
							when prompted.
      i. The installer validates the DR package using the passphrase.
      ii. In case of errors in validation, the installer aborts the 
										operation. To work around the issue, refer to the following 
										article: http://www.veritas.com/docs/100033743
      iii. If the external CA-signed certificates could not be backed up,
           configure the certificates on the host.
           Refer to the following article: 
           http://www.veritas.com/docs/100044249
	2. Configure the devices necessary to read the media listed above.
	3. Inventory the media.
	4. Make sure that the primary server can access the attached DR image file.
	5. Start the NetBackup Recovery Wizard from the NetBackup web UI.  Or, start 
    the wizard from a command line by entering bprecover -wizard.

Disaster Recovery Procedure without the DR Image File
NOTE: ONLY ATTEMPT THIS AS A LAST RESORT 
If you do not have the attachment included with this email, use the following 
instructions to recover your catalog. (If using OpenStorage disk pools, refer 
to the Shared Storage Guide to configure the disk pools instead of step 2 and 
3 below ):

	1. Install NetBackup.
 2. Run:
    Configure certificates for the media server that is associated with this
    catalog recovery by running the below commands on that host:
        nbcertcmd -getCACertificate
        nbcertcmd -getCertificate -force
	3.  Configure the devices necessary to read the media listed above.
	4.  Inventory the media.
	5.  Run
		    To recover from copy 1:
		    bpimport -create_db_info [-server name] -id /
	6.  Run:  
		   cat_export -client client1.example.com
	7.  Go to the following directory to find the DR image file
	    cat_backup_1686692545_FULL:
	    /usr/openv/netbackup/db.export/images/primary.example.com/1686000000
	8.  Open cat_backup_1686692545_FULL file and find the BACKUP_ID
	    (for example: primary.example.com_1686692545).
	9.  Run:  
		   bpimport [-server name] -backupid primary.example.com_1686692545
	10.  Run:
		   bprestore -T -w [-L progress_log] -C primary.example.com -t 35 
     -p cat_backup -X -s 1686692545 -e 1686692545 /
	11. Run the NetBackup web UI to restore the remaining image database 
	    if the DR image is a result of an incremental backup.
	12. To recover the NetBackup relational database, run:
		   bprecover -r -nbdb
	13. Stop and start NetBackup.
	14. Run:
     Re-configure the certificates on the primary server and the media server,
     because the database is restored to a previous point in time.
         Run the following set of commands on the primary server:
             nbcertcmd -getCACertificate -force
             nbcertcmd -createToken -reissue -host <primary/media>
             name <>
             nbcertcmd -getCertificate -token <> -force
         Run the following set of commands on the media server that is
         associated with this catalog recovery:
             nbcertcmd -getCACertificate -force
             nbcertcmd -getCertificate -force
 15. Configure the devices if any device has changed since the last
     backup.
	16. To make sure the volume information is updated, inventory the
     media to update the NetBackup database.