Data integrity issue in Lazy Delete feature for NetBackup when writing to S3 storage without immutable storage enabled

Problem

A data integrity issue has been identified in the lazy delete feature of NetBackup 11.0 and 11.0.0.1. This issue may result in unintended deletion of backup images under specific conditions when writing to S3 storage without immutability (WORM) enabled when using MSDP-C or Cloudscale. 

Affected Versions

• NetBackup 11.0
• NetBackup 11.0.0.1

Error Message

• If the S3 storage is WORM enabled, an error message in the job details can be seen as in the following example: 

Failed to set WORM immutable and indelible lock for image: sample.com_1759505446_C1_F18, indelible interval: 604800 seconds, retry: 0, status: 2060018file not found 

• No error messages are related to this issue if the S3 storage is not WORM enabled. 

Cause

If lazy delete is enabled and one of the three scenarios occurs all fragments of the image which have been written are set to a lazy delete state rather than just the incomplete or empty fragments. After the lazy delete retention has expired, these fragments will be deleted. 

1. Checkpoint-enabled backup failure: When writing to S3 storage, if a backup which has checkpointing enabled fails, then NetBackup attempts to delete the incomplete fragment and retry from that fragment. The issue may occur during fragment deletion, resulting in the image associated with that fragment being deleted. 

2. Duplication Job Failure:  When duplicating an image to S3 storage, if the duplication job fails, then NetBackup attempts to delete the fragments which have been written and starts a new job to retry the duplication.  The issue may occur during fragment deletion, resulting in the image associated with that fragment being deleted, either before or after the retried duplication job succeeds. If the duplication source image has been affected by the lazy delete feature, the duplication would be affected too. 

3. Empty fragment creation in backup: when writing to S3 storage is extremely rare. An empty fragment is only created when the size of the backup data lands exactly on the boundary of the fragment size. By default, the fragment size is 50GB, and the empty fragment is only created if the backup size is an exact multiple of 50GB (e.g. 50GB, 100GB, 150GB). 

Solution

• Immediately apply the following EEBs (Emergency Engineering Binaries) to the media servers configured with MSDP-C :
  • For NetBackup 11.0: MSDP EEB bundle 4196284 v4 or above
  • For NetBackup 11.0.0.1: MSDP EEB bundle 4207692 v1 or above

If EEB application is not immediately feasible, disable lazy delete on affected media servers using MSDP-C:

1. Create a credentials file (msdpc-creds.txt) with the following content:

MSDPC_ACCESS_KEY=<access_key>
MSDPC_SECRET_KEY=<secret_key>
MSDPC_REGION=<region>
MSDPC_PROVIDER=<provider>
MSDPC_ENDPOINT=<endpoint>

Where: 

• <access_key>: AWS access key or Azure storage account name
• <secret_key>: is the AWS secret key associated with the access key or the Azure storage account access key 
• <region>: Cloud region is the AWS or Azure region associated with the MSDPC bucket. Example (e.g., "us-east-1", "eastus")
• <provider>: Cloud provider ("amazon", "azure", etc.) the cloud provider.  Run /usr/openv/pdde/pdcr/bin/msdpcldutil platform list to see a full list of providers. 
• <endpoint>: Endpoint URL if applicable, set the endpoint. For azure this would be (e.g., for Azure: https://your_storage_account.blob.core.windows.net/)

See more information about setting MSDPC variables for msdpcldutil in the NetBackup™ Commands Reference Guide

2. Run the following command to disable lazy delete:

/usr/openv/pdde/pdcr/bin/msdpcldutil lazy-delete disable --credfile <cred-file-path> --bucket <bucket-name>

• <cred-file-path>: is the full path to "msdpc-creds.txt" created in step 1.
• <bucket-name>: is the name of the cloud bucket.

For more details on setting MSDPC variables and using msdpcldutil, refer to the NetBackup Commands Reference Guide