Sharepoint or OneDrive scans fail for few monitored site collections or accounts

Problem

For Veritas Data Insight (DI), the Sharepoint Online and OneDrive scans fail for few monitored site collections or accounts with Exit Code: -1

Error Message

From commd logs: 

Wed Jun 14 19:18:25 UTC 2023 Full scan for Cloud Account \\OneDrive\ctrfs09@vtas.local\ failed (args: [\\OneDrive\ctrfs09@vtas.local, Cloud Account, false]) (took: 20901538ms) (Details: Exit code: -1; Scan Job Failed) (Obj: SHARE:\\OneDrive\ctrfs09@vtas.local\)

From connector_service logs:

2023-07-12 02:59:32 {"message":"{ConnectorID=0c31569f-3547-4f1e-8090-687c67a5a9e9, JobID=3784b889-0dd9-41f7-ab05-73aaf86974b6} Errored response is 429 TOO MANY REQUESTS","level":"ERROR"

2023-07-12 02:59:32 {"message":"{ConnectorID=0c31569f-3547-4f1e-8090-687c67a5a9e9, JobID=3784b889-0dd9-41f7-ab05-73aaf86974b6} API https://disme-my.sharepoint.com/personal/ctrfs09_vtas_local/_api/$batch got THROTTLED","level":"ERROR"

2023-06-14 13:32:01 {"message":"{ConnectorID=2aedaed0-c123-4ca2-99bb-fe3926cd06a9, JobID=f2770f2a-a39c-4ce8-943e-444b53af0a9e} Putting Thread on sleep for 206000 milliseconds","level":"ERROR"

2023-06-14 13:35:27 {"message":"{ConnectorID=2aedaed0-c123-4ca2-99bb-fe3926cd06a9, JobID=f2770f2a-a39c-4ce8-943e-444b53af0a9e} Switching Application because throttling occured","level":"INFO"

2023-06-14 13:35:27 {"message":"{ConnectorID=2aedaed0-c123-4ca2-99bb-fe3926cd06a9, JobID=f2770f2a-a39c-4ce8-943e-444b53af0a9e} Switching from application d65480ed-9833-46fd-a215-7b17f30bca63","level"

2023-06-14 13:35:27 {"message":"{ConnectorID=2aedaed0-c123-4ca2-99bb-fe3926cd06a9, JobID=f2770f2a-a39c-4ce8-943e-444b53af0a9e} Cannot switch application because single application with client id d65480ed-9833-46fd-a215-7b17f30bca63 was used","level":"ERROR"

Cause

• There is a performance issue on the Collector node which is used for scanning, in which case there will be CPU/Memory related warnings for the collector node.
• There are too many requests for the Sharepoint online OR Onedrive registered cloud application, in which case, the cloud application tries to load balance or throttle. 

Since there is a single application registered in Azure, it does not find additional applications to throttle with and eventually fails due to high load. 

Solution

Register more than one application for the Sharepoint Online OR Onedrive accounts added to DI. 

The DI admin can register two or more applications and test the scan until throttling is no longer an issue.

For more information, please review the section:To create and register an application with Microsoft in the Veritas Data Insight Administrator's Guide