Unable to configure MSDP Cloud disk pool when there is no connection from the storage server to the cloud provider's default region.

Problem

The error occurs when adding an MSDP Cloud disk pool and selecting cloud buckets located in a non-default location.

This article will use, as an example, Amazon Web Services (AWS) for which the default region is us-east-1

Note: The problem can occur for other cloud providers as well.

Use the wizard to add a disk pool in the NetBackup WebUI (Storage -> Disk pools -> Add)

Select a region for the bucket other than the default. For example, eu-west-3:

Cloud buckets are successfully fetched and are correctly displayed in the region that was chosen above.

After selecting a bucket and clicking Next, "cannot connect on socket. getHostSession failed" error is received.

Error Message

Cannot connect on socket

Error details:

- getHostSession failed

Increase logging for the following processes:

1. Turn up the logging for Open Cloud Storage Daemon (OCSD) to 3

Edit file: <msdp storage path>/etc/pdregistry.cfg 

Change:

[Symantec/PureDisk/OpenCloudStorageDaemon]
 loglevel=1

To:

loglevel=3

Then:

restart ocsd if it is running

To stop ocsd:

/usr/openv/pdde/pdconfigure/pdde stop ocsd

To start ocsd:

/usr/openv/pdde/pdconfigure/pdde start ocsd

2. Change logging for oid 364 to DebugLevel 6:

/usr/openv/netbackup/bin/vxlogcfg -p nb -o 364 -a -s DebugLevel=6 -s DiagnosticLevel=6

Retry the configuration, and collect the logs that cover the configuration attempt from these directories
/usr/openv/logs/nbrmms
Prior to NetBackup 9.x:       /usr/openv/netbackup/logs/esfs_storage

Starting with NetBackup 9.x:  /usr/openv/netbackup/logs/ocsd_storage

Review esfs_storage log (located under /usr/openv/netbackup/logs/esfs_storage).

Search for:"Unable to get bucket location" error, it shows a connection attempt to the default region us-east-1:

{"level":"error","bucket":"<bucket_name>","error":"RequestError: send request failed\ncaused by: Get \"https://<bucket_name>.s3.dualstack.us-east-1.amazonaws.com/?location=\": dial tcp 52.217.98.238:443: connect: connection timed out","default region name":"us-east-1","time":"2021-02-05T10:38:29.036692542+01:00","message":"Unable to get bucket location"}

To confirm the errors in the wizard, review oid 364 log (logged to nbrmms log file) .

At first it shows that the list of buckets was retried successfully:

03/05/2021 14:50:55.722 [Debug] NB 51216 364 PID:2606 TID:139841434003200 File ID:222 [No context] 5 [STS Service] amazon: retrieved bucket list successfully, HTTP code 200.

03/05/2021 14:50:55.722 [Debug] NB 51216 364 PID:2606 TID:139841434003200 File ID:222 [No context] 5 [STS Service] amazon_raw:tmpListBucketAlias_1614952253994: No. of buckets <number of buckets>

03/05/2021 14:50:55.722 [Debug] NB 51216 364 PID:2606 TID:139841434003200 File ID:222 [No context] 5 [STS Service] amazon: Setting bucket name- <bucket_name>

After that start getting errors listing properties of the object:

03/05/2021 14:52:55.113 [Debug] NB 51216 364 PID:2606 TID:139841434003200 File ID:222 [No context] 5 [STS Service] amazon: Entered AmzRestRequest::validateHeaders

03/05/2021 14:52:55.113 [Debug] NB 51216 364 PID:2606 TID:139841434003200 File ID:222 [No context] 1 [STS Service] amazon: Unable to get HEAD properties of bucket <bucket_name>, HTTP code: 403, Response:

Followed by "failed to get storage properties" error after all the retries are exhausted:

03/05/2021 14:54:54.364 [Debug] NB 51216 364 PID:2606 TID:139841434003200 File ID:222 [No context] 1 [STS Service] amazon_raw:tmpListBucketAlias_1614952253994: Error: failed to get storage properties for bucket <bucket_name> (rv = 2060037)

03/05/2021 14:54:54.364 [Debug] NB 51216 364 PID:2606 TID:139841434003200 File ID:222 [No context] 5 [STS Service] metering: Failed to get lsu prop by name, error code = 2060037

After collecting the logs for nbrmms and esfs_storage:

Revert the change to the pdregistry.cfg fle for the logging. Restart ocsd.

Reduce logging for oid 364:

/usr/openv/netbackup/bin/vxlogcfg -p nb -o 364 -r -s DebugLevel=6 -s DiagnosticLevel=6

Cause

Open Cloud Storage Daemon (OCSD) needs to verify the location of the buckets as the location can be changed after the initial configuration. In order to do that, OCSD always attempts to connect to the default region when determining the region of the bucket. For this step to succeed, there needs to be a connection between the storage server and the cloud provider's default region.

If there is no connection from the storage server to the cloud provider's default region, OCSD will fail to verify the bucket location and the creation of the disk pool will fail.

The following steps can further confirm that the issue is in the connectivity and not with the cloud provider's ability to list object's properties:

1. As oid 364 log indicated an error getting HEAD properties of the bucket, confirm the ability to list properties of objects using the AWS CLI by running “aws s3api head-object” command.

• Copy a single object to the bucket using the following cp command:

               aws s3 cp <file_name>  s3://<bucket_name>

• List properties of the copied object

2. Determine if there is a connection from the storage server to the default region's endpoint

• Review the esfs_storage log to determine the https address:

               Get \"https://<bucket_name>.s3.dualstack.us-east-1.amazonaws.com/?location=\":

• ping that address

               ping <bucket_name>.s3.dualstack.us-east-1.amazonaws.com

If there is no connection to the cloud provider's default region and the connection cannot be established, proceed to the steps outlined in the next section.

Solution

Create a new cloud instance starting with "aa" for the desired non-default region, which would serve as an alias to the existing region and should be used instead.

The name of the new cloud instance needs to start with an "aa" for the Web UI to use it.

Provide only one service endpoint (the URL of the entry point for an AWS web service) corresponding to the correct region for the new cloud instance.

/usr/openv/netbackup/bin/admincmd/csconfig cldinstance -a -in aaa_amazon_eu-west-3 -pt amazon -sh s3.dualstack.eu-west-3.amazonaws.com
Successfully added cloud instance: aaa_amazon_eu-west-3

/usr/openv/netbackup/bin/admincmd/csconfig cldinstance -ar -in aaa_amazon_eu-west-3 -lc eu-west-3 -rn "eu-west-3(custom)" -sh s3.dualstack.eu-west-3.amazonaws.com
Region(s) eu-west-3 added to cloud instance aaa_amazon_eu-west-3 successfully

After this instance is created, in WebUI please select the newly created service host and region.

Now OCSD won’t use us-east1 region and service endpoint, allowing the configuration to work as expected.