Duplications of Granular (GRT) Backups from disk to tape gives error 191

Problem

Duplications from disk to tape fail with 191 error, which is associated with a Database System Error (220) in the BPDBM log.

Issues only occur for Granular Recovery Technology (GRT) backups that are being duplicated

Status 191
Status 220

 

Error Message

Detailed status in the Activity Monitor:

07/11/2013 11:25:28 - begin Duplicate
07/11/2013 11:25:30 - requesting resource hcart-robot-tld-10
07/11/2013 11:25:30 - requesting resource @aaaaV
07/11/2013 11:25:39 - granted resource  G42488
07/11/2013 11:25:39 - granted resource  HP.Ultrium4-SCSI.010.8
07/11/2013 11:25:39 - granted resource  hcart-robot-tld-10
07/11/2013 11:25:39 - granted resource:MediaID=@aaaaV;DiskVolume=PureDiskVolume;DiskPool=MSDP;Path=PureDiskVolume;StorageServer=MEDIA_SERVER;MediaServer=MEDIA_SERVER
07/11/2013 11:27:15 - Info bpdbm (pid=23439) cataloguing file number 10000 path /System State/Active Directory/CN=Configuration/CN=DisplaySpecifiers/CN=419/CN=LS-Display/
07/11/2013 11:28:32 - Info bpdbm (pid=23439) cataloguing file number 20000 path /System State/Active Directory/DC=staff/OU=EMC Celerra/OU=Computers/CN=B-staff/sAMAccountName
07/11/2013 11:35:33 - Error bpduplicate (pid=23428) db_IMAGE() failed: database system error (220)
07/11/2013 11:35:33 - Error bpduplicate (pid=23428) Status = no images were successfully processed.
07/11/2013 11:35:33 - Error bpduplicate (pid=23428) Duplicate of backupid CLIENT_1372924921 failed, database system error (220).
07/11/2013 11:35:33 - Error bpduplicate (pid=23428) Status = no images were successfully processed.
07/11/2013 11:35:33 - end Duplicate; elapsed time 0:10:05
no images were successfully processed  (191)

admin log: 

11:25:39.105 [23428] <2> logconnections: BPDBM CONNECT FROM 168.192.1.2.52139 TO 168.192.1.2.1556 fd = 5
11:25:39.130 [23428] <2> db_end: Need to collect reply
11:35:33.010 [23428] <2> db_end: no DONE from db_getreply(): database system error
11:35:33.014 [23428] <16> build_catalog_for_virtual_browse_images: db_IMAGE() failed: database system error (220)

bpdbm log:

11:25:42.242 [23439] <8> do_pbx_service: [vnet_connect.c:2108] via PBX VNETD CONNECT FROM 168.192.1.2.52149 TO 10.4.1.3.1556 fd = 31
11:25:42.680 [23439] <2> dbm_open_client_connection: client socket from bpcr = 30
11:25:42.880 [23439] <2> bpcr_get_version_rqst: bpcd version: 07100000
11:26:12.686 [23439] <2> bpcr_get_string_pentry: filenum:0 length:0 blknum:0 in_image:23 raw_partition_sz:0 gigabytes:0 device_num:0 plength:48 path:/System State/Active Directory/CN=Configuration/ dlength:41 data:16832 root root 0 0 1372818621 1249908865

<SNIP>

11:29:52.363 [23439] <2> bpcr_get_string_pentry: filenum:0 length:0 blknum:0 in_image:23 raw_partition_sz:0 gigabytes:0 device_num:0 plength:63 path:/System State/Active Directory/DC=staff/OU=Resources/OU=Groups/ dlength:41 data:16832 root root 0 0 1259070918 1249909557
11:34:52.373 [23439] <16> bpcr_get_string_pentry: bpcd protocol error - failed to read file entry, 62
11:34:52.374 [23439] <16> extend_list_add_files: get_file_rec() error: -1 image: /usr/openv/netbackup/db/images/CLIENT/1372000000/02_AD_NA_HOSTS_1372924921_INCR.f nfiles: 0
11:34:52.382 [23439] <2> dbm_close_client_connection: sending disconnect request to bpcd
11:35:32.956 [23439] <2> bpcr_disconnect_rqst: bpcr received the following message:
11:35:32.956 [23439] <2> bpcr_disconnect_rqst: bpcr received the following message:
11:35:33.010 [23439] <2> process_request: request complete: exit status 220 database system error; query type: 248

Cause

During the Duplication of a GRT backup a BPDBM process is tasked with generating the Full Files File ( .f file) in the NetBackup Images Catalog. This occurs because a Full Files file ( .F file) is not created during the GRT backup process. To perform this task the BPDBM process makes a connection to the Client via BPCD and requests that the Client mounts the GRT backup image. The Client then enumerates the objects within the backup on disk, and sends the Metadata back to BPDBM so that it can generate the Full Files File ( .f file) for the backup image. However in the BPDBM code there is a hardcoded BPCD_TIMEOUT value that is set to 5 minutes (300 seconds). If any of the Enumeration tasks on the Client take longer than 5 minutes to return the requested Metadata to the BPDBM process, then BPDBM exits with the "Database System Error (220)"

Solution

In 7.5 a setting was added on a Master Server to NetBackup that allows a user to set a value for BPCD_TIMEOUT manually.
The new setting is a bp.conf entry on UNIX Master Servers, or a Registry entry on Windows Master Servers.

To set a new BPDBM BPCD_TIMEOUT value:

On a UNIX Master Server:

- Edit the bp.conf file and add the entry :

IGNORE_BPCD_TIMEOUT = <timeout_value_in_seconds>

For example :

IGNORE_BPCD_TIMEOUT = 1800


On a Windows Master Server:

- Open the registry by running regedit

Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry. Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and workstation be made prior to making any registry changes.

- Navigate to HKLM\Software\Veritas\NetBackup\CurrentVersion\Config

- Add a REG_DWORD value to the above registry path and enter the Name :

      IGNORE_BPCD_TIMEOUT

- Modify the new IGNORE_BPCD_TIMEOUT registry value and set the Base value to Decimal

- Then input the new BPCD TIMEOUT setting that is required, in Seconds in the "Value data" field
 
The IGNORE_BPCD_TIMEOUT entry will be the new value that BPDBM will use for the BPCD_TIMEOUT setting, in seconds

Applies To

Master and Media Server :  NetBackup 7.5.0.5
Client :                              NetBackup 7.1.0.4