Getting your environment up and running after a disaster when the disaster recovery package passphrase is lost

Problem

A disaster recovery package is created during each catalog backup. The package is encrypted with the passphrase that you set.

The disaster recovery package contains the following information:

• Security certificates and private keys of the master server and the NetBackup CA (Certificate Authority)
• Information about the hosts in the domain
• Security settings

You need to provide the same encryption passphrase when you install NetBackup in the disaster recovery mode on the master server after a disaster. If you fail to provide the appropriate passphrase, the NetBackup identity cannot be recovered and you need to manually deploy host ID-based certificates on all NetBackup hosts.

Additional information about passphrases can be found in the NetBackup Security and Encryption guide, and also from article 100040029 (About NetBackup 8.1 catalog passphrases).

Solution

If the master server and the media server are configured to use NetBackup CA-signed certificates (host ID-based certificates) only

To get your NetBackup environment up and running after a disaster when the disaster recovery passphrase is lost, carry out the following procedure:

1. Install NetBackup in the non-disaster recovery mode as the passphrase is lost. In this case, the master server receives a new Certificate Authority (CA) certificate and host ID-based certificate.
2. Add the media server (associated with the catalog backup that you want to restore) to the master server to make it visible in the Host Properties node in the NetBackup Administration Console.

• Add the media server using the NetBackup Administration Console:
  • Go to Host Properties > Master Servers > Servers and add the media server.
  • Restart the NetBackup services.
• Add the media server using the command-line interface. Run the following command:
  • nbemmcmd -addhost -machinename <media_server> -machinetype media -masterserver <master_server> -operatingsystem <OS> -netbackupversion <netbackup_media_server_version>

3. Manually deploy a host ID-based certificate on the media server that is associated with the catalog backup that you want to restore. Run the following commands:

• nbcertcmd -getCAcertificate
• nbcertcmd -getCertificate -force

• nbcertcmd -getCAcertificate
• bpnbat -login -loginType WEB
• nbcertcmd -createToken -name <token_name>
• nbcertcmd -getCertificate -token

4. Clear the cache from the master server and the media server using the following command:

• bpclntcmd -clear_whitelist_cache

5. Deploy host name-based certificates on the media server if the catalog backup is on the media server. To deploy the host name-based certificate, run the following command on the master server.

• bpnbaz -ProvisionCert <media_server_name>

6. Restart the NetBackup services on the media server where the host name-based certificate is deployed.
7. Perform catalog recovery. Restart NetBackup services on the master server when the recover is completed.
8. Refresh the certificate on the master server because the catalog recovery has brought the old database back.

• If your environment is not clustered, run the following commands:
• bpnbat -login -loginType WEB
• nbcertcmd -createtoken -name <reissue_token_name> -reissue -host <host_name>
• nbcertcmd -getCertificate -token <token_from_previous_command> -force
• nbcertcmd -getCertificate -force -host <host_name>
• If your environment is clustered, run the following commands:
• bpnbat -login -loginType WEB
• nbcertcmd -createtoken -name <reissue_token_name> -reissue -host <cluster_virtual_name>
• nbcertcmd -getCertificate -token <token_from_previous_command> -force -cluster
• nbcertcmd -getCertificate -force -host <host_name>

9. To refresh the certificate on the media server, run the following commands:

• nbcertcmd -getCAcerificate
• nbcertcmd -getCertificate -force

• nbcertcmd -getCAcertificate
• bpnbat -login -loginType WEB
• nbcertcmd -createToken -name <token_name>
• nbcertcmd -getCertificate -token

10. To Clear the cache from the master server and the media server, run the following command:

• bpclntcmd -clear_whitelist_cache

11. Restart the NetBackup services on the media server.
12. To deploy new host ID-based certificates on the remaining hosts (clients and media servers), run the following commands:

• nbcertcmd -getCAcertificate
• nbcertcmd -getCertificate -force
• If the certificate deployment level on the master server is set to Very High, you must create an authorization token before you deploy a certificate. Run the following commands:
• nbcertcmd -getCAcertificate
• nbcertcmd -createToken -name <token_name>
• nbcertcmd -getCertificate -token

13. Deploy the host name-based certificates on all media servers and clients. To deploy the host name-based certificates, run the following command on the master server.

• bpnbaz -ProvisionCert -AllMediaServers -AllClients

13. Set a new Disaster Recovery Passphrase.  See the NetBackup Security And Encryption Guide for information on how to set the passphrase.  Once completed, run a catalog backup.
14. Verify whether the normal backups that you have previously created run successfully or not.

Note: NetBackup services need to be restarted on the media servers and clients where the host name-based certificates are deployed.

If the master server is configured to use both NetBackup CA-signed certificates and external CA-signed certificates and media server is configured to use external CA-signed certificates

Do the following:

1. Install NetBackup on the master server. A NetBackup certificate (host ID-based certificate) is deployed during installation.
2. Once the master server is configured to use NetBackup certificate, carry out the following steps to recover external certificate configuration:
   1. In case the external certificates are not impacted during the disaster
      1. Issue NetBackup certificate from the new master server to the hosts in the domain which are configured to use NetBackup certificate.
      2. Ensure CRL settings are done in bp.conf and CRLs are downloaded if CDP is not configured on each host.
      3. Re-enroll the external certificate for the master server
      4. Clear the host cache on each host.
      5. Perform catalog recovery from the media server where the catalog backup images reside.
      6. Clear the host cache on the media server. This enables communication between the master server and the media server using the external certificate and catalog recovery can be initiated.
      7. Enroll the external certificate for the media server.
         nbcertcmd -enrollCertificate
      8. Update the NetBackup web server to use the newly-issued certificate
         configureWebServerCerts -addExternalCert -all -certPath <certificate_path> -privateKeyPath <private_key_path> -trustStorePath <trust_store_path>
      9. Update the path to the new certificate and rest of the external certificate configuration options.
      10. Place this certificate and the external CA trust certificate in a location accessible by the master server (in case of Windows, they can also be placed in the Windows certificate store).
      11. Issue a new certificate from the existing external CA for the master server
      12. In case the external CA itself is lost in the disaster
          1. Setup a new external CA.
          2. Issue a new certificate from the new CA  - only for the master and the media server.
          3. Place this certificate and the external CA trust certificate in a location accessible by the server (in case of Windows, they can also be placed in the Windows certificate store).
          4. Update the path to the new certificate and rest of external configuration options.
          5. Update the NetBackup web server to use a certificate issued by new external CA.
             configureWebServerCerts -addExternalCert -all -certPath <certificate_path> -privateKeyPath <private_key_path> -trustStorePath <trust_store_path>
          6. Configure the external certificate on the master server and enroll media server certificates with the master server.
             nbcertcmd -enrollCertificate
          7. This enables communication between the master and the media servers using the external certificate and catalog recovery can be initiated.
          8. Perform catalog recovery from the media server where the catalog backup images reside.
          9. Once the catalog recovery is performed, enroll a new external certificate for the master and the media servers.
          10. Issue a new external certificate for all hosts in the domain as required and the configuration and enrollment of the same can be done on each host of the domain.
          11. Issue a NetBackup certificate from the new master server to the hosts in the domain that are configured to use NetBackup certificates.
          12. Clear the host cache on each host.
          13. Ensure that the CRL configuration options are specified and the CRLs are downloaded if CDP is not configured on each host.
          14. Remove the older external certificate from other NetBackup hosts of the domain.
          15. Remove the old external root certificate from the trust stores of the hosts.

Note: Hosts which have NBCA enabled will need their certificates to be renewed, as a new NBCA with fresh installation of Master is configured.

Disclaimer:
External certificate authority (CA) support is added in NetBackup 8.1.2.1, which is a limited release. For more information on the external CA support in NetBackup, contact the Veritas Technical Support team.