After expiring all images for a deduplication disk pool pointing to PureDisk or NetBackup Media Server Deduplication Option, 'crcontrol --dsstat' still shows a large amount of used container space.

Problem

After expiring all images, verifying with "Images On Disk" report that no images exist, verifying that the mini-catalog no longer references any backups, 'crcontrol --dsstat' still shows a hundreds of GB or 1 or more TB of 'Space used within containers'. While the value for 'Space needs compaction' is very small.
 

Error Message

************ Data Store statistics ************
Data storage Size Used Avail Use%
3.6T 596.4G 3.1T 16%
Number of containers : 12669
Average container size : 269357042 bytes (256.88MB)
Space allocated for containers : 3412484375861 bytes (3.10TB)
Space used within containers : 615302318429 bytes (573.04GB)
Space available within containers: 2797182057432 bytes (2.54TB)
Space needs compaction : 0 bytes (0.00MB)
Records marked for compaction : 0
Active records : 4779407
Total records : 4779407

Cause

Appears to be a storage leak, or at least data leftover after cleaning.

Solution

Issue resolved itself. This is normal, expected behavior. It can take days to see all the expired data released from the dedup containers after expiring all images written to a Media Server Deduplication disk pool. This is due to optimizations in the code provide the best possible performance for every day dedup activity like 'add some data', 'delete some data'. In other words the code is more optimized for daily normal behavior and not for mass expirations/deletions of data. During expiration and compaction, all containers must be scanned and compared against the cache after which expired data is only marked for compaction but not actually released. Successive compaction operations eventually free up space.

Simply continue to monitor via 'crcontrol --dsstat' over the course of a week.

Applies To

5220, version 2.0.1 with Media Server Deduplication Option (MSDP) configured.

Or MSDP on a non-appliance platform, any supported operating system.