NetBackup Image Cleanup job fails with Status 88 "OpenStorage WORM lock error"

Problem

NetBackup Image Cleanup job fails to clean up WORM Images.

Error Message

OpenStorage WORM lock error.  (88)

Cause

The media server bpdm log reports:

Aug 26, 2021 10:30:01 AM - Info nbdelete (pid=6952) Setting OST Worm locks on images. Media Server: ##### Media: @####
Aug 26, 2021 10:30:01 AM - requesting resource  @####
Aug 26, 2021 10:30:01 AM - granted resource  MediaID=@####;DiskVolume=<NAME>-1;DiskPool=<NAME>-1;Path=<PATH>;StorageServer=<NAME>;MediaServer=<NAME>
Aug 26, 2021 10:30:01 AM - Info bpdm (pid=7057) started
Aug 26, 2021 10:30:01 AM - started process bpdm (pid=7057)
Aug 26, 2021 10:30:03 AM - Critical bpdm (pid=7057) Failed to set WORM immutable and indelible lock for image: <BACKUP_ID> with status: 2060064 WORM invalid min
OpenStorage WORM lock error.  (88)
 

In addition to the status code 2060064 “WORM invalid min” error, there may be other reasons that setting WORM properties is failing such as 2060065 “invalid maximum”, 2060037 “access not allowed”, among others.

Regardless of the reason for the failed lock, NetBackup cleanup jobs will continue to retry the lock operation until the image expiration time has elapsed, or until storage configuration has changed such that the lock operation succeeds (for example changing minimum/maximum WORM lock configuration).

NetBackup reports on two times which are relevant for WORM, both of which can be seen from bpimagelist output:

Expiration Time:  Wed 06 Oct 2021 05:07:46 PM CD (1633558066)
Copy WORM Unlock Time: Wed 06 Oct 2021 05:07:52 PM CD (1633558072)

if ExpirationTime > Copy WORM Unlock Time

In this condition, NetBackup will try to extend the Copy WORM Unlock Time so that it will be equal to or greater than the ExpirationTime.  This is the typical condition when cleanup jobs fail with status 88.

If ExpirationTime < Copy WORM Unlock Time

NetBackup will not do anything about this condition, as it cannot shorten the WORM unlock time on storage.  In normal operation, the UnlockTime will be slightly ahead of the ExpirationTime.  NetBackup will not attempt to automatically cleanup such an image until both times are elapsed.

Copy WORM Unlock Time with time “INFINITY” is a special case.  INFINITY means that NetBackup has never verified the WORM Unlock Time on storage.  This condition goes with

Copy 1 Flags:            0xa WORM_LOCK_PENDING
Copy WORM Unlock Time:           INFINITY       (32535212400)

Java UI will show this time as “December 31, 3000 05:00 PM”
Such an image will be removed from NetBackup catalog once its ExpirationTime has elapsed.

Solution

By default, WORM copies cannot be removed from the NetBackup catalog until after their expireTime and WormUnlockTime have elapsed.

The simplest option to fix this condition is to use bpexpdate and set a new expiration time which is within the storage allowed “MinimumLockDuration” and “MaximumLockDuration” values.

First, find the min and max settings on the Logical Storage Unit (LSU), for example by running the command "nbdevquery -listdv", or looking at the disk volumes and disk pools in the Java GUI.

Calculate an acceptable WormUnlockTime as follows:

time now + allowed duration,

where allowed duration is between MinimumLockDuration and MaximumLockDuration for the disk volume.

NOTE: MinimumLockDuration and MaximumLockDuration are storage-side configurations and cannot be modified from NetBackup.

NOTE: After changing MinimumLockDuration and MaximumLockDuration values on storage, you must use nbdevconfig -updatests, -updatedp, and -updatedv in order for NetBackup to be aware of these new values.

% nbdevquery -listdv -stype PureDisk -U  | grep 'WORM Lock'
WORM Lock Min Time         : 54000
WORM Lock Max Time         : 10000000
% date
Wed Oct  6 17:44:26 CDT 2021

Set the new expiration time:

%bpexpdate -d “10/07/2021 08:44:27” -backupid <the_backup_id> -extend_worm_locks

In the above example, 15 hours is the minimum allowed lock duration for the disk volume.  The expiration of the image was set to be 15 hours + 1 second in the future.  NetBackup will change both the ExpirationTime and WORM Unlock Time to the same value “08:44:27”.

Another option is to use bpexpdate with the -nodelete option to remove the image from the NetBackup catalog.  It will then be the responsibility of the storage admin to remove the images from storage at a later time once the WORM lock is elapsed.  Please consult with the storage vendor on the best method to remove the images.  This option should only be used if the images are no longer needed for recovery operations. 

The final, and most extreme option is to use bpexpdate with the -try_expire_worm_copy option to allow a WORM copy to be removed from the NetBackup catalog.  Even with the -try_expire_worm_copy option, the storage may prevent actual deletion of the copy.   If the storage device continues to enforce the copy's indelible time, the fragments remain in NetBackup's deletion worklist in which case bpexpdate -nodelete may be the only option.

NOTE: The -try_expire_worm_copy option should be used with caution, and usually only after a storage administrator has removed the WORM locks from images. If in doubt, contact Veritas technical support, reference this article and supply a 'bpimagelist -backupid <backupID_goes_here> -L' for the backupID in question.

Location of commands bpimagelist, nbdevquery, bpexpdate:
On UNIX/Linux:
/usr/openv/netbackup/bin/admincmd/

Or Windows:
<install_path>\NetBackup\bin\admincmd\

bpexpdate -backupid <backupID_goes_here>  -d 0 -copy <copy_number_goes_here> -try_expire_worm_copy