NetBackup catalog recovery fails with STATUS 13, if the previous EMM database used an alternative staging directory other than the default /usr/openv/db/staging

Problem

NetBackup catalog recovery fails with STATUS 13, if the previous EMM database used an alternative staging directory other than default /usr/openv/db/staging.

If the NetBackup Enterprise Media Manager (EMM) database is configured as part of a shared resource (clustered) or the default staging location (/usr/openv/db/staging) is not used, then a recovery of the database could fail with STATUS 13.

During the recovery of the EMM database the database files are placed into a staging location (where they were backed up from). This is the correct behavior, however, when recovering the EMM database if the staging location backed up was not the default location, it must be defined before the catalog recovery is started (using the "/usr/openv/db/bin/nbdb_admin -vxdbms_nb_staging <new_staging_area> " command). If the changes are not performed prior to the recovery, the NetBackup recovery will assume the default location of /usr/openv/db/staging will be used. See the resolution below for more details.

The problems are not related to the original backup or configurations held in the /usr/openv/netbackup/bp.conf, these are all correctly defining the new/alternative/final location of the EMM database (for example: /opt/VRTSnbu/db/data). The bp.conf does not define the staging directory location.

The same problems could occur in a non-clustered environment if the staging area is altered from the default location (/usr/openv/db/staging).The actual database files could be in the default location of /usr/openv/db/data but the staging could be held at /new_directory/staging. If the recovery is started before defining the correct staging location the recovery will fail with STATUS 13.

Error Message

ADMIN log - Restores use a Non-Default location of /opt/VRTSnbu/db/staging/

10:20:52.781 [27530] <4> sendRequest: waiting for restore complete status
10:20:52.781 [27530] <4> sendRequest: calling get_bprdESwto(): sockfd=<6>, timeout=<0>
10:21:05.232 [27530] <4> sendRequest: RESTORE status: <0> <the requested operation was successfully completed>
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: VxBSABeginFileListRestore - SUCCESS
10:21:05.232 [27530] <4> VxBSAGetFirstFileListObject: INF - entering VxBSAGetFirstFileListObject.
10:21:05.232 [27530] <4> VxBSAGetFirstFileListObject: INF - Returning Object pathName: /opt/VRTSnbu/db/staging/EMM_DATA.db status: 6
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: VxBSAGetFirstFileListObject - SUCCESS
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: #RESTORED VXF1216799015 /opt/VRTSnbu/db/staging/EMM_DATA.db
10:21:05.232 [27530] <4> VxBSAGetNextFileListObject: INF - entering VxBSAGetNextFileListObject.
10:21:05.232 [27530] <4> VxBSAGetNextFileListObject: INF - Returning Object pathName: /opt/VRTSnbu/db/staging/EMM_INDEX.db status: 6
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: VxBSAGetNextFileListObject - SUCCESS
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: #RESTORED VXF1216799015 /opt/VRTSnbu/db/staging/EMM_INDEX.db
10:21:05.232 [27530] <4> VxBSAGetNextFileListObject: INF - entering VxBSAGetNextFileListObject.
10:21:05.232 [27530] <4> VxBSAGetNextFileListObject: INF - Returning Object pathName: /opt/VRTSnbu/db/staging/NBDB.db status: 6
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: VxBSAGetNextFileListObject - SUCCESS
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: #RESTORED VXF1216799015 /opt/VRTSnbu/db/staging/NBDB.db
10:21:05.232 [27530] <4> VxBSAGetNextFileListObject: INF - entering VxBSAGetNextFileListObject.
10:21:05.232 [27530] <4> VxBSAGetNextFileListObject: INF - Returning Object pathName: /opt/VRTSnbu/db/staging/NBDB.log.1 status: 6
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: VxBSAGetNextFileListObject - SUCCESS
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: #RESTORED VXF1216799015 /opt/VRTSnbu/db/staging/NBDB.log.1
10:21:05.232 [27530] <4> VxBSAGetNextFileListObject: INF - entering VxBSAGetNextFileListObject.
10:21:05.232 [27530] <4> VxBSAGetNextFileListObject: INF - Returning Object pathName: /opt/VRTSnbu/db/staging/databases.conf status: 6
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: VxBSAGetNextFileListObject - SUCCESS
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: #RESTORED VXF1216799015 /opt/VRTSnbu/db/staging/databases.conf
10:21:05.232 [27530] <4> VxBSAGetNextFileListObject: INF - entering VxBSAGetNextFileListObject.
10:21:05.232 [27530] <4> VxBSAGetNextFileListObject: INF - Returning Object pathName: /opt/VRTSnbu/db/staging/server.conf status: 6
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: VxBSAGetNextFileListObject - SUCCESS
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: #RESTORED VXF1216799015 /opt/VRTSnbu/db/staging/server.conf
10:21:05.232 [27530] <4> VxBSAGetNextFileListObject: INF - entering VxBSAGetNextFileListObject.
10:21:05.232 [27530] <4> VxBSAGetNextFileListObject: INF - Returning Object pathName: /opt/VRTSnbu/db/staging/vxdbms.conf status: 6
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: VxBSAGetNextFileListObject - SUCCESS
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: #RESTORED VXF1216799015 /opt/VRTSnbu/db/staging/vxdbms.conf
10:21:05.232 [27530] <4> VxBSAGetNextFileListObject: INF - entering VxBSAGetNextFileListObject.
10:21:05.232 [27530] <4> VxBSAGetNextFileListObject: INF - Returning Object pathName: NULL
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: VxBSAGetNextFileListObject - SUCCESS
10:21:05.232 [27530] <4> ASA_XBSA_file_restore: Exit with status 0.

...but during "do_nbdb_recovery" step, NetBackup looks for the restored files under the local file system on /usr/openv/db/staging and immediately after we have the STATUS 13.

10:21:10.236 [27530] <16> ReadVxDBMS_ConfFile: Can't open configuration file: /usr/openv/db/staging/vxdbms.conf
10:21:10.236 [27530] <16> do_nbdb_recovery: No database information exists in vxdbms.conf from staging directory: /usr/openv/db/staging
10:21:10.236 [27530] <16> do_nbdb_recovery: No database data directory information exists in vxdbms.conf from staging directory: /usr/openv/db/staging
10:21:10.236 [27530] <16> do_nbdb_recovery: No database index directory information exists in vxdbms.conf from staging directory: /usr/openv/db/staging
10:21:10.236 [27530] <16> do_nbdb_recovery: No database transaction log directory information exists in vxdbms.conf from staging directory: /usr/openv/db/staging
10:21:10.236 [27530] <16> bprecover: ERR - Failed to recover NBDB on main (13)

Solution

There is a command /usr/openv/db/bin/nbdb_admin (-vxdbms_nb_staging <new_staging_area> ), that allows for the staging area to be defined differently to the default. This update is placed in the vxdbms.conf file as a new entry. The command must be run during the recover process at the time when the NetBackup documentation states: If you have used the nbdb_move command to relocate the NetBackup database files.... 

Note: The Related Knowledge Base Articles below (100017664) has a section that states, "This creates a temporary database in the default location. If the database has been moved or the environment is clustered.... ". At this point, the /usr/openv/db/bin/nbdb_admin -vxdbms_nb_staging <new_staging_area> command needs to be run.

After the move of the database files is successful:

# /usr/openv/db/bin/nbdb_admin -vxdbms_nb_staging <new_staging_area>
VXDBMS_NB_STAGING entry in vxdbms.conf updated successfully.

Contents of vxdbms.conf should show the following:

VXDBMS_NB_SERVER = NB_<name>
VXDBMS_NB_PORT = 13785
VXDBMS_NB_DATABASE = NBDB
VXDBMS_NB_DATA = <new_path>
VXDBMS_NB_INDEX = <new_path>
VXDBMS_NB_TLOG = <new_path>
VXDBMS_NB_MLOG = <new_path>
VXDBMS_NB_STAGING = <new_path>    <<<<<<--------NEW ENTRY !
VXDBMS_NB_PASSWORD = c7ed54df2e136026158fb2b4aebc433a0a2b54bd2e4

Workaround:

If "nbdb_admin -vxdbms_nb_staging" is not used:

Creating a symbolic link from /usr/openv/db/staging to /opt/VRTSnbu/db/staging (or staging directory related to an alternate location) before the recovery is started will enable the restore to complete without errors.