unable to configure any AWS plugins for both source account and cross account. error comes in discovery stage. The status is “Failed”,

Problem

unable to configure any AWS plugins for both source account and cross account. 
error comes in discovery stage. The status is "Failed".

Error Message

Cross Account Error

An invalid API request is encountered.
Error Details:
?the input server is not a valid sharedservices server?
@@@@@@@@@@@@@@@@

Source account

The plug-in instance already exists in the cloudpoint server
Attribute errors
Msg: The plugin configuration for aws already exists
errMsg: The plugin configuration for aws already exists
httpstatuscode: 409
type: error

Example Environment:

• NBU 9.1 NBU Master server on RHEL 8.4 
• CloudPoint server on RHEL 8.3

Cause

NBU has 4 configurations showing.

"Plugin_ID": "SourceAccount", "Config_ID": "aws.xxxxxx-2xxxxxx8-xxxxxx0", 
"Plugin_ID": "aws.xxxxxx-9xxxxxx72-xxxxxx58", "Config_ID": "aws.7xxxxxxxxxxxxxxxxxx8", 
"Plugin_ID": "aws.exxxxxxxxxxxxxxxxxx0a", "Config_ID": "aws.exxxxxxxxxxxxxxxxxxxxxxxx0a", 
"Plugin_ID": "infra-non-prod", "Config_ID": "aws.1xxxxxxxxxxxxxxxxxxxxxxxxc7",

The first 3 configurations are good and we can see the agent running on CP for each of them.

aws.9xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0 ->>> flexsnap-agent.fxxxxxxxxxxxxxxxxxxxxxxxx3 
aws.7xxxxxxxxxxxxxxxxxxxxxxxx8 ->>> flexsnap-agent.2xxxxxxxxxxxxxxxxxxxxxxxx8 
aws.e7xxxxxxxxxxxxxxxxxxxxxxxx0a ->>> flexsnap-agent.2xxxxxxxxxxxxxxxxxxxxxxxx1c 

The stale 4th configuration does not have any presence in logs and no agent is running for that in CP, but the configuration does exist in CP data base:

aws.1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxc7 {"_id":{"$oid":"61xxxxxxxxxxxxxxxxxxb5"},"agentid":
"agent.4bxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxf6",
"hostname":"flexsnap-agent.4bxxxxxxxxxxxxxxxxxxxxxxxxf6",
"osName":"linux","onHost":false,"lastMessage":1632839921,"pluginConfig":
{"aws":[{"configId":"aws.1dxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx6c7","pollInterval":{"unit":"hours","value":1},"status":"failed","errmsg":
"An error occurred (UnauthorizedOperation) when calling the DescribeNetworkInterfaces operation: 
You are not authorized to perform this operation.","configuration":
{"targetrole":"MCCVeritasNetbackupCloudpointCrossAccountRole","targetaccount":"06xxxxxx34","configtype":"iamrole","regions":["ap-southeast-1"],
"iamconfigtype":"cross_account","endpoints":[{"name":"ap-southeast-1","location":"Singapore","country":"Singapore","continent":"Asia Pacific",
"endpoint":"ap-southeast-1.amazonaws.com"}],"accountId":"065xxxxxx34"},"templateSelector":{"schemaVersion":2,"location":"aws"},"configHash":"98xxxxxxxxxxxxxxxxxxbffda0",
"taskid":"dcxxxxxxxxxxxx3a-xxxxxxb64","discovered_time":1.6328376589438376e+09}]},"containerRunning":true,"status":"online","agentType":"child","parentId":
"agent.096xxxxxxxxxxxxa8","hidden":true,"srcids":["aws-06xxxxxx34-ap-southeast-1"]}

We are seeing that all the 3 function configuration fails in discovery with below error related to permission; user to fix this issue else the discovery would keep on failing on cloud point.

- AWSPlugin.assets: failed to get EC2 instances: An error occurred (UnauthorizedOperation) 
when calling the DescribeNetworkInterfaces operation: You are not authorized to perform this operation.
2021-09-14T15:11:52.302769924+00:00 stdout F"
2021-09-14T15:11:52.302769924+00:00 stdout F File ""aws/aws.py"", line 6323, in assets""
2021-09-14T15:11:52.302769924+00:00 stdout F objs.append(AWSInstanceObj(self, i, ec2,""
2021-09-14T15:11:52.302769924+00:00 stdout F File ""aws/aws.py"", line 2750, in __init__"
2021-09-14T15:11:52.302769924+00:00 stdout F network_security_group = ec2.NetworkInterface("
2021-09-14T15:11:52.302769924+00:00 stdout F File ""/tmp/cloudpoint/libs/aws/lib/boto3/resources/factory.py"", line 339, in property_loader"
2021-09-14T15:11:52.302769924+00:00 stdout F self.load()"
2021-09-14T15:11:52.302769924+00:00 stdout F File ""/tmp/cloudpoint/libs/aws/lib/boto3/resources/factory.py"", line 505, in do_action""
2021-09-14T15:11:52.302769924+00:00 stdout F response = action(self, *args, **kwargs)""
2021-09-14T15:11:52.302769924+00:00 stdout F File ""/tmp/cloudpoint/libs/aws/lib/boto3/resources/action.py"", line 83, in __call__""
2021-09-14T15:11:52.302769924+00:00 stdout F response = getattr(parent.meta.client, operation_name)(*args, **params)""
2021-09-14T15:11:52.302769924+00:00 stdout F File ""aws/aws.py"", line 5961, in wrapper""
2021-09-14T15:11:52.302769924+00:00 stdout F return getattr(self._client, name)(*args, **kwargs)""
2021-09-14T15:11:52.302769924+00:00 stdout F File ""/tmp/cloudpoint/libs/aws/lib/botocore/client.py"", line 357, in _api_call""
2021-09-14T15:11:52.302769924+00:00 stdout F return self._make_api_call(operation_name, kwargs)""
2021-09-14T15:11:52.302769924+00:00 stdout F File ""/tmp/cloudpoint/libs/aws/lib/botocore/client.py"", line 676, in _make_api_call""
2021-09-14T15:11:52.302769924+00:00 stdout F raise error_class(parsed_response, operation_name)"
2021-09-14T15:11:52.302769924+00:00 stdout F An error occurred (UnauthorizedOperation) when calling the DescribeNetworkInterfaces operation: You are not authorized to perform this operation.
2021-09-14T15:11:52.302851332+00:00 stdout F 11:52 flexsnap-agent.faxxxxxxx63 flexsnap-agent-agent.facxxxxxxxxx63[8] Poll detect_asset_changes@3600secs flexsnap.plugins.aws: INFO - AWSPlugin.sources: SourceId aws-77xxxxxx85-ap-southeast-1

We see permission related to DescribeNetworkInterfaces missing.

{"name":"configtype","singlevalue":"iamrole"},{"name":"regions","multivalue":["ap-southeast-1"]},{"name":"targetrole","singlevalue":
"MCCVeritasNetbackupCloudpointCrossAccountRole"},{"name":"iamconfigtype","singlevalue":"cross_account"}],"configId":
"aws.099xxxxxxxxdd","isDisable":false,"cloudpointHostname":"ccam2p282","status":"failed","errMsg":
"An error occurred (UnauthorizedOperation) when calling the DescribeNetworkInterfaces operation: You are not authorized to perform this operation.",
"configSchemaVersion":2,"deploymentLocation":"aws"},"links":{"self":{"href":"/config/snapshotproviders/configuredplugins/aws/instances/perimeter"}}},
53:com.netbackup.discovery.cloud.service.CloudPluginInfo,50,
51216,495,495,36248187,1632997692055,3827,248,16:349F4D3C78C66B63,112::getPluginInfoUsingAPICall() - 
CPServer:ccam2p282 configid:aws.099xxxxxxxxx88cdd state:failed,53:com.netbackup.discovery.cloud.service.CloudPluginInfo,40,
51216,495,495,36248188,1632997692055,3827,248,16:349F4D3C78C66B63,60::createDiscoveryEvent() - Cloud Configid Discovery:perimeter,63:com.netbackup.discovery.cloud.service.CloudDiscoveryServiceImpl,40,
51216,495,495,36248189,1632997692055,3827,248,16:349F4D3C78C66B63,57::createDiscoveryEvent() - Invalid discovery config:failed,63:com.netbackup.discovery.cloud.service.CloudDiscoveryServiceImpl,10,
51216,495,495,36248190,1632997692056,3827,248,16:349F4D3C78C66B63,103:Request URL - [POST]

Solution

The configuration is in failed state and due to this the discovery operation marked fail.

If we see the erroMsg we will see the error related to permission.

Here is the new set of permission in effect in 9.1, different than 9.0:

https://www.veritas.com/content/support/en_US/doc/140789355-148057836-0/v141441905-148057836

Once this new set of permissions is provided, this issue should all be resolved.