How and when to reclaim space from within a Deduplication Storage or Deduplication enabled cloud storage in Backup Exec.

Article: 100008765
Last Published: 2023-03-06
Ratings: 1 3
Product(s): Backup Exec

Problem

Longer retension, add hoc Backups may sometime lead to over use of Deduplication enabled storage (cloud/local deduplication storage) . This may fill the storage and cause Backups to queue or fail with storage errors.

Backup Exec 21 known issue (fixed in Backup Exec 22)
Backup Exec 21.4/22 known issue with OST media deletion when set (on shared dedupe storage) expires.
(Fixed in Backup Exec 22.1)

Solution
 

SECTION 1 -  RECLAIM SPACE FROM LOCAL DEDUPLICATION STORAGE.
If a cloud storage is configured to use deduplication then the commands would differ.
SECTION 2 -  RECLAIM SPACE FROM DEDUPLICATION ENABLED CLOUD STORAGE.
(Note: Deduplication enabled cloud storage is a feature introduced from Backup Exec version 21.3)

Step 1 -3  helps decide if manual intervention is needed to reclaim space.

Step 1 -

Check Storage Statistics by running the following command from command prompt ( run as administrator ) , change path to BE Install path which by default is C:\Program Files\Symantec or Veritas\Backup Exec\

crstats.exe --verbose --convert-size

Step 2 - 

Review the Use Rate section. For the Backups to work efficiently the use rate should be less than 95 percent. At 95 percent it is recommended to stop the Backup and first reclaim some space

Step 3 -

Review the Catalog Logical Size section as well in the above command output. This is the front end data that is backed up and stored in the Deduplication Storage Folder.

It is possible that the Catalog Logical Size may show more size compared with what shows up when all Backup sets inside the Deduplication Storage ( from BE UI --> Storage Tab --> Dedupe Storage --> Backup Set) are summed up. To calculate the total size of backup sets in Backup sets section, simply select all (Ctrl A) in the space, right click copy and past it on a  spreadsheet to sum the size (sets in TBs and GBs can be considered for a rough calculation. The sets in MBs can be omitted unless that brings in a huge difference). If the UI shows less and Catalog Logical Size shows more, then an Inventory and Catalog of the Deduplication Storage is recommended with use storage based catalogs selected from BE Global Settings --> Catalog.

 

Step 4 -

To reclaim the space manually, Backup sets can be expired from the Dedupe Storage --> Backup Set section. A right click Expire shows if a set has dependent sets. Carefully Expire the sets which are not needed. It triggers a deletion of media ( in the case of Deduplication storage, segments within Deduplication Storage folder. Under no scenario should any file be deleted from within Deduplication Storage drive.) and catalog of the Backup set. To confirm successful expiration, check the Audit Log from BE console --> Configuration and Settings --> Audit Log --> Select Backup Set Retention category and review the recent OST deletions. The Backup set should also vanish from the BE UI.

If a Deduplication Storage is holding 500 TB ( Front-end Data) in 40 TB of Storage, 50 TB of front end data expiry will probably free up 5 TB ( very rough figures if 10:1 Deduplication Ratio is considered) within Deduplication Storage.
Note : Deduplication Storage stores unique chunks which are referenced with many Backup Sets and a deletion of one set may not mean that the stored chunk is removed. If 10 sets are referring to a particular chunk then only after those sets are all deleted will the chunk be marked for deletion as it would then have zero reference to it. The Deduplication Drive may not show the space release to the operating system as it will be controlled by Backup Exec Deduplication Storage.

Step 5 -

An expiry or expiration of Backup sets within Deduplication Storage lead to Reference Deletions. These form Tlogs in DeduplicationStorage\Queue folder. To commit these instantly run the following command two times otherwise they would get processed when the daily Queue Processing task runs within Deduplication Storage.

crcontrol.exe --processqueue

This processing of tlogs is called "Queue Processing". To check the status of the command that was run two times use the following command :

crcontrol.exe --processqueueinfo

When Busy and Pending show no, then it means the command completed.

 

Repeat Step 1 and 2 to review the space usage. If the space reclaimed is not enough for newer Backups, the Dedupe drive should be extended ( if no more backups can be expired ) or more Backup sets need to be expired. Make sure to review Step 3 as well.

 

RECLAIM SPACE FROM DEDUPLICATION ENABLED CLOUD STORAGE
 

Step 1 -

Get the datastorage ID of the deduplication enabled cloud storage.

C:\Program Files\Veritas\Backup Exec>pddecfg -a listcloudlsu
dsid,     lsuname,       storageId,         CachePath
 3,   AzCloudDedupeStore, azure.com/azdedupe/AzCloudDedupeStore, T:\BackupExecDeduplicationStorageFolder\data\ds_3
   
Note: DSID is the first column of the command output. Ensure correct <dsid> data storage ID is used in the command in order to get the correct output. 

Step 2 -

 The statistics of the storage can be calculated by below commands 

C:\Program Files\Veritas\Backup Exec>crstats.exe --cloud-dsid <dsid>
C:\Program Files\Veritas\Backup Exec>crcontrol.exe --clouddsstat <dsid>

Step 3 -

If Backup sets are expired manually to reclaim space, execute processqueue command two times. 
If not run manually, it would run by default as per the schedule set within Backup Exec. 
Note: The processqueue command helps to update the changes at that instant and hence the stats would 
start to show accurately once the processqueueinfo command show Busy and Pending as NO.

C:\Program Files\Veritas\Backup Exec>crcontrol.exe --processqueue --dsid <dsid>
C:\Program Files\Veritas\Backup Exec>crcontrol.exe --processqueue --dsid <dsid>
C:\Program Files\Veritas\Backup Exec>crcontrol.exe --processqueueinfo --dsid <dsid>
Busy   : no
Pending: no
Note: The Busy and Pending may show Yes initially. When the task is processed, both will show as NO.

Step 4 - 

Rerun the crcontrol.exe --clouddsstat <dsid> or crstats.exe --cloud-dsid <dsid> command to 
check the updated statitics for deduplication enabled cloud storage.

 

Was this content helpful?