NetBackup Abs: CloudCatalyst to Azure - Failed uploads reported as successful

NetBackup Abs: CloudCatalyst to Azure - Failed uploads reported as successful

Article: 100045595
Last Published: 2019-09-30
Ratings: 0 0
Product(s): NetBackup

Severity

Data Loss

Description

Certain error conditions could cause failed uploads to be reported as successful resulting in a potential data loss. These error conditions can be exposed by, but are not limited to, running CloudCatalyst on servers with relatively small amounts of memory.

Error:

In order to have data loss, an error code would be returned for only one file out of the set of files for a NetBackup image being uploaded via the cloud connector.  Consider these two files within the same minute on this CloudCatalyst server:

Note: The following example shows CloudCatalyst configured with Azure Cloud, but this problem is not Azure specific.

File 1, for which the upload failed:

/data/117/119843.bin    (/data/117/119843.dbin in esfs_storage log due to substitution for the . to .d)

  1. The upload error is caused by failure to allocate memory.
  2. Cloud connector interface to close the image (stspi_close_image) returns an error 2060002, Unable to upload partial blob status.
  3. Error code 2060002 is returned back as the status of the upload, so the upload fails.

esfs_storage log:

[04/17/2019 16:08:48.181319 18702] cloud_storage.cpp:83(logm) ERROR - "ESP Azure: CAzureManager::UploadBlob: Failed to allocate memory. Error:[std::bad_alloc].

Returning with status:[2060002]"

[04/17/2019 16:08:48.181546 18702] cloud_storage.cpp:83(logm) ERROR - "ESP Azure: stspi_close_image: Unable to upload partial blob status : [2060002]"

[04/17/2019 16:08:48.300016 18702] cloud_storage.cpp:83(logm) ERROR - "ESP metering: Failed to close image, return code = 2060002"

[04/17/2019 16:08:48.300164 18702] cloud_storage.cpp:83(logm) ERROR - "ESP [throttling_close_image_v7]fail to call sts_close_image, error code = 2060002"

[04/17/2019 16:08:48.300207 18702] cloud_storage.cpp:83(logm) ERROR - "ESP gateway: Failed to close image, return code = 2060002"

[04/17/2019 16:08:48.300256 18702] esp_worker.cpp:326(handle_put_request) INFO - "PUT image (<path>/data/117/119843.dbin) took 1482065 microseconds; 67001298 bytes xfer'd."

NetBackup detected/reported that the upload to cloud failed:

esfs_monitor log:

[04/17/2019 16:08:48.300715 5693] esfs_monitor.cpp:1213(cloud_status_update) ERROR - "file /data/117/119843.bin failed with 2060002 error, size 67001298, upload scn 658. fields in db will not change"

However for File 2, the upload also failed (at about the same time, different thread):

/data/178/182335.bin   (/data/178/182335.dbin dbin in esfs_storage log due to substitution for the . to .d)

  1. The upload error is caused by failure to allocate memory.
  2. Cloud connector interface to close the image (stspi_close_image) returns an error 2060002, Unable to upload partial blob status.
  3. Error code 2060002 is overwritten by a successful status, returned from a subsequent call to delete the the partial/incomplete uploaded image.
esfs_storage log: [04/17/2019 16:08:45.098867 18708] cloud_storage.cpp:83(logm) ERROR - "ESP Azure: CAzureManager::UploadBlob: Failed to allocate memory. Error:[std::bad_alloc]. Returning with status:[2060002]" [04/17/2019 16:08:45.098960 18708] cloud_storage.cpp:83(logm) ERROR - "ESP Azure: stspi_close_image: Unable to upload partial blob status : [2060002]" [04/17/2019 16:08:45.247257 18708] cloud_storage.cpp:83(logm) ERROR - "ESP metering: Failed to close image, return code = 2060002" [04/17/2019 16:08:45.247434 18708] cloud_storage.cpp:83(logm) ERROR - "ESP [throttling_close_image_v7]fail to call sts_close_image, error code = 2060002" [04/17/2019 16:08:45.247513 18708] cloud_storage.cpp:83(logm) ERROR - "ESP gateway: Failed to close image, return code = 2060002" [04/17/2019 16:08:45.247775 18708] cloud_storage.cpp:65(logm) INFO - "ESP CsscSlaveHelper::CsscSlaveHelper_GetCapath setting CA Path : /usr/openv/var/webtruststore/cacert.pem" [04/17/2019 16:08:45.247831 18708] cloud_storage.cpp:65(logm) INFO - "ESP CsscSlaveHelper::CsscSlaveHelper_GetCertAndKeyPathsByName hostname is <media_server>" Delete the partial/incomplete uploaded image from the cloud: [04/17/2019 16:08:48.975901 18708] cloud_storage.cpp:65(logm) INFO - "ESP Azure: CAzureManager::DeleteAllImageObjects: Deleted IMAGE_PROPERTIES [<path>/data/178/182335.dbin/META_IMAGE_PROPERTIES]" [04/17/2019 16:08:48.975973 18708] cloud_storage.cpp:65(logm) INFO - "ESP Azure: stspi_delete_image_v10: Image <path>/data/178/182335.dbin deleted" [04/17/2019 16:08:48.976062 18708] esp_worker.cpp:326(handle_put_request) INFO - "PUT image (<path>/data/178/182335.dbin) took 5384555 microseconds; 67074963 bytes xfer'd." esfs_monitor log reports a successful upload to cloud: [04/17/2019 16:08:48.976764 5693] esfs_monitor.cpp:1243(cloud_status_update) INFO - "/data/178/182335.bin uploaded successfully, set size in cloud 67074963, upload scn 2016, scn 2016 in db"

Cause:

NetBackup reports that the file was uploaded successfully when it was not.

During an upload of a file to the cloud, an error is returned that there was not enough memory to get allocated.

The cloud connector calls an interface to close the image and receives an error that there was a partial or incomplete upload.

This is followed by a delete of the partial or incomplete image, overwriting the previously received error from closing the image with the successful status received when deleting the image.

Solution:

Install the latest version of the NetBackup Cloud Catalyst Engineering Binary Bundle.  The fix for this issue is included in the the following CC bundle versions and higher:
8.1 3958410 v7
8.1.1 3956754 v7
8.1.2 3958437 v6

This issue does not affect NetBackup 8.2+ because it doesn't use the cloud connector sts interface any more to upload data to the cloud.

Contact NetBackup Support to run msdpcheck to see what other images may have been affected by this problem.

Action Required"

Install the NBU Cloud Catalyst EEB Bundle associated with the version of NetBackup that is installed.

8.1 3958410 v7
8.1.1 3956754 v7
8.1.2 3958437 v6

This EEB is available through the Veritas Download Center

https://www.veritas.com/content/support/en_US/downloads

Was this content helpful?