CloudCatalyst duplications fail with status 191 and esfs_storage log shows "Request data header too old"

Problem

Duplications (or backups) to a CloudCatalyst system failing with 191 error and the cache becomes full and is not automatically cleared out as data is not able to be uploaded to the backend cloud storage.

Error Message

esfs_storage log:

[03/25/2020 16:23:34.071937 285555] cloud_storage.cpp:83(logm) ERROR - "ESP Azure: CAzureRestRequest::ValidateCredentials:Operation failed with HTTP error code:[403]. "
[03/25/2020 16:23:34.072013 285555] cloud_storage.cpp:83(logm) ERROR - "ESP Azure: CAzureApi::ValidateCredentials: Request failed. Error code:[7], HTTP status code:[403]."
[03/25/2020 16:23:34.072290 285555] cloud_storage.cpp:83(logm) ERROR - "ESP Azure: CAzureException::CAzureException: xml Message : [<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:66301b34-801e-0071-11bb-02a77c000000
Time:2020-03-25T15:40:39.5648836Z</Message><AuthenticationErrorDetail>Request date header too old: 'Wed, 25 Mar 2020 15:23:34 GMT'</AuthenticationErrorDetail></Error>]"
[03/25/2020 16:23:34.072473 285555] cloud_storage.cpp:83(logm) ERROR - "ESP Azure: CAzureManager::validateCredentials: Failed to validate credentials. Error message:[AuthenticationFailed]. Error Message Detail:[Request date header too old: 'Wed, 25 Mar 2020 15:23:34 GMT'], HTTP Error code:[403]. Returning with status:[2060029]"

Cause

There is a time difference between the Cloud Catalyst system and the Cloud provider (in this case Azure) and the cloud provider is rejecting the request as it believes it is too old.

Solution

The solution is to adjust the time on the CloudCatalyst system to be correct ideally by using NTP to keep it in sync with the cloud provider