Duplication jobs to CloudCatalyst may complete with status 0, but if cache fills up, dataloss could result

Severity

Possible Data Loss

Description

Duplication jobs to the cloud using NetBackup CloudCatalyst will complete with status 0, but if the CloudCatalyst cache volume becomes full, data loss can result.

Versions Affected

NetBackup 8.1 EEB 3958410 v11 (or below)

NetBackup 8.1.1 EEB 3956754 v12 (or below)

NetBackup 8.1.2 EEB 3958437 v9 (or below)

NetBackup 8.2 EEB 3981837 v4 (or below)

Cause

In very busy systems or systems with slower upload speeds cache volume may become full.

Cache eviction may run normally and free up space in cache before the high watermark is reached, but may not be able to release space fast enough to keep up with the incoming volume.

Cache eviction starts at the midpoint between low and high watermark (recommended 50/90, so cache eviction would start at 70.   The user's settings for this is in the esfs.json file)

Normally, cache eviction would run and free up space before the high watermark is hit, but in very busy systems or systems with slower upload speeds, it may not be able to do so.

If the cache volume becomes full while running those versions, there is a possibility that duplication jobs will trigger the creation of 0-byte container .bin or .bhd files under /msdpc/cache/userdata/data directory. CloudCatalyst will write an "__EMPTYFILE__" string to these 0-byte container files and will upload them to the cloud storage. 

Action Required

Upgrade the CloudCatalyst binaries to the following HotFix versions or higher.

8.1   EEB 3958410 v12 (or above)

8.1.1 EEB 3956754 v13 (or above)

8.1.2 EEB 3958437 v10 (or above)

8.2   EEB 3981837 v5 (or above)

The fixes contained in these HotFix EEB Bundles will change the CloudCatalyst behavior. When the fix is installed, CloudCatalyst will stop writing "__EMPTYFILE__" string into 0-byte container .bin or .bhd files, therefore the upload request will fail and trigger the duplication job to fail. 

CloudCatalyst will pause container writes to the cache volume if the high watermark is reached. This will help prevent the cache from filling up causing the creation of 0-byte .bin and .bhd container files or other errors indicating the storage server DOWN status.

Note: A High watermark of more than 90 is NOT recommended to help avoid disk full errors. Build-Your-Own (BYO) configurations with relatively small cache size may need a high watermark lower than 90 to avoid disk full errors.

To Obtain the latest CloudCatalyst Hotfix EEB bundles, please log into the DownLoad Center and obtain the appropriate bundle based on the version installed.

NetBackup 8.1 UPD159549

NetBackup 8.1.1 UPD437757

NetBackup 8.1.2 UPD893444

NetBackup 8.2 UPD772994

Any CloudCatalyst installation with the affected versions and the aforementioned scenarios, apply the latest CloudCatalyst EEB bundle which is now available for this issue in the current version(s) of the product(s) mentioned. Refer to the EEB links under Related Downloads to obtain the bundles needed to resolve the issue.

The following command may be used to determine if the 14 byte files reside on the Cloud volume.

# find /msdpc/storage/data/  -type f -size 14c |  tee -a /tmp-space/CC_hostname/cloud14.list

If these files exist and contain __EMPTYFILE__ inside, contact NetBackup Support to run the msdpcheck utlliity to determine if there are affected images.