NetBackup Snapshot Manager Backup from Snapshot operation is failing with error "The security token included in the request is invalid."

Problem

NetBackup Snapshot Manager Backup from Snapshot operation is failing with error The security token included in the request is invalid.

Error Message

Job Details:
MM/DD/YYYY HH:MM:SS - Warning bpbrm(pid=1406) from client aws-ec2-<region>-<instance_id>: WRN - The security token included in the request is invalid.
MM/DD/YYYY HH:MM:SS - Critical bpbrm(pid=1406) from client aws-ec2-<region>-<instance_id>: FTL - cleanup() failed status 11

MM/DD/YYYY HH:MM:SS - Info bpbkar(pid=0) done. status: 11: system call failed
MM/DD/YYYY HH:MM:SS - Error nbpem(pid=5716) duplicate exited with status 11 (system call failed)

VxMS:
MM/DD/YYYY HH:MM:SS : log_callback::0 <ERROR> : :CAwsCredentialManager::getInstanceRole:TID139722004649856:Failed to get IMDSv2 token will use IMDSv1
MM/DD/YYYY HH:MM:SS : log_callback::0 <ERROR> : :awsRetry exhausted for error 28. Tried 10 times.

<ERROR> : : CAwsJsonParser::getMessageString:TID139722004649856:AWS json:{"message":"The security token included in the request is invalid."}

Cause

If the AWS instance of Snapshot Manager has been launched with a default hop limit of 1, it causes the AWS API calls to IMDSv2 metadata service to time out.

Solution

• Check the existing HttpPutResponseHopLimit configured by running the following AWS CLI command:
  • # aws ec2 describe-instances --instance-id <instance_id_of_NSM_host> --region <region>
  • Response: Reservations -> Instances -> MetadataOptions -> HttpPutResponseHopLimit
• If IMDSv2 metadata service has been enabled on the AWS instance where Snapshot Manager resides, then it must be reconfigured to use HttpPutResponseHopLimit > 1
  • # aws ec2 modify-instance-metadata-options --instance-id <instance_id_of_NSM_host> --region <region> --http-put-response-hop-limit 2