Unexpected reingestion of Exchange Online data during incremental backup runs in Cohesity Alta SaaS Protection

Problem

Some customers using Cohesity Alta SaaS Protection (CASP) to back up Exchange Online workloads may have experienced an unexpected increase in storage space being consumed. Also, the duration of incremental backups would have been higher than normal. 

Error Message

N/A.

Cause

March-April 2024 

Microsoft made a change intended to address an unrelated event on their side.  However, it had the consequence of updating the last modified date of items in users’ Exchange Online mailboxes along with some properties which are not visible to users. Veritas uses a Microsoft supported and suggested method, namely the SyncFolderItems method from the EWS API for identifying incremental changes between backups. This method flagged all these items as changed, which caused a new version of the items to get backed up in CASP, which resulted in an unexpected increase in storage space consumed. 

Here is a note from Microsoft about this occurrence: 

August-September 2024

Some customers started reporting an unexpected increase in the duration of backups along with an increase in the amount of Exchange data being backed up after August 16. After verifying that CASP is functioning on expected lines, Veritas has been working actively with Microsoft to troubleshoot the issue.  

Microsoft has acknowledged that this issue is unrelated to the previous one.  This has been published as service incident EX870973. All impacted customers are expected to be notified of this incident through their Service Health dashboard. Here is the text from the notification: 

As mentioned in the note, Microsoft plans to roll out a fix to all the tenants by September 20, after which the data included in backups should be reduced considerably. Even during this incident, CASP has been backing up items which are identified as changed by SyncFolderItems EWS API. 

January-February 2025

CASP has experienced a substantial rise in incremental backups for Microsoft Exchange from mid-January to the end of February. 

CASP uses the Microsoft Exchange SyncFolderItems operation to monitor changes, including new items, deletions, and updates to existing items, whether they are generated by the system or initiated by users. Once these changes are detected, CASP captures and backs up the modifications to ensure data integrity and availability.

Solution

Customers should allow the backups to run as per their schedule. Given that CASP has been backing up the items marked as modified by Microsoft-supported and suggested methods, customers can take some actions to decrease their storage size in CASP.  See the below ways to reduce storage:
 

1. Purchase Additional Storage Licenses to cover the current consumption levels.
2. Run Deletion Policies to reduce data below the current entitlement.
3. Using Version Control Settings on the Exchange Stor(s) to gradually reduce storage over time, by keeping only the most relevant backed up versions and removing the older, redundant ones based on the version control settings. 
    

Do not enable version pruning on the Stor if the following are applicable to your organization:

• WORM retention policies are enabled in CASP for the Exchange Stor(s), as this will prevent item versions from being removed via the pruning process.
• Have a compliance policy that requires all data to be retained indefinitely.  
• Do not want to unintentionally lose any data at all. 
  • NOTE: While Exchange doesn't have item versions, it allows edits of items, such as saving a draft of an email or updating a calendar invite.  These updates are captured as new item versions in CASP, and if pruning is enabled on the Exchange Stor(s) then these items will be removed. 

To enable the pruning policy on the Exchange Stor(s) do the following:

1. Login to the CASP Admin Portal.
2. Go to Administration -> Storage -> StorSite Region -> Stor.
3. On the General tab, go to Version Control Settings, select Enable this feature. 
   
   
    
4. Put '1' for each pruning setting as seen below.  This configuration will sparse out the item versions stored in CASP, eventually ensuring that there is only a single item version for each item backed up.  It will take time for the policy evaluation process to analyze and actually prune the versions.  
   
   
    
5. Select Save at the top. 
   
   NOTE: Once the Version Control Settings are saved on the Stor it cannot be disabled.  Additionally, the default settings can only be edited at the time of enabling version pruning.  If there are any questions, please contact Cohesity Support.