NetBackup 8.3.0.1+ backups to MSDP in a MSDP Cloud configuration fail with status 129 'Disk storage unit is full' yet adequate MSDP storage space appears to exist

Article: 100050653
Last Published: 2021-06-16
Ratings: 0 0
Product(s): NetBackup & Alta Data Protection

Problem

NetBackup 8.3.0.1+ backups to MSDP in a MSDP Cloud configuration fail with status 129 'Disk storage unit is full' yet more than adequate MSDP storage space exists as seen in the 'df' and 'crcontrol --dsstat' command outputs:

 

# df -h /msdp/data/dp1/pdvol
Filesystem                    Size  Used Avail Use% Mounted on
/dev/vx/dsk/nbuapp/pdvol      448G   32G  414G   8% /msdp/data/dp1/pdvol

 

# /usr/openv/pdde/pdcr/bin/crcontrol --dsstat
********** Data Store statistics **********
Data storage      Raw    Size   Used   Avail  Use%
                 444.7G 425.8G  31.6G 394.2G   8%

Number of containers             : 211
Average container size           : 111163092 bytes (106.01MB)
Space allocated for containers   : 23455412514 bytes (21.84GB)
Reserved space                   : 20315194368 bytes (18.92GB)
Reserved space percentage        : 4.3%

 

Error Message

status 129 'Disk storage unit is full'

In the OS (operating system) /var/log/messages log, spoold reports the following error to the kernel:

Jun 15 08:15:36 nbmsdp1 spoold[12604]: WARNING [140393204950848]: -1: Partition containing containers at /msdp/data/dp1/pdvol/data(...): 448 GiB, 413.47 GiB free (92.29%) of which 395.94 GiB is required as margin for temporary files (88.38%) (warning threshold exceeded).#012The Content Router is starting to run low on disk space. Data removal, queue processing and/or garbage collection is advised.
 

Cause

In this particular environment, 5 cloud Logical Storage Units (LSU) existed.

# /usr/openv/pdde/pdcr/bin/pddecfg -a listcloudlsu
dsid, lsuname, storageId, CachePath
3, msdp_cloud_vol1, nbmsdp1_msdp_cloud_vol1/bkl-netbackup-test/msdp_cloud_vol1, /msdp/data/dp1/pdvol/data/ds_3
4, msdp_cloud_vol2, nbmsdp1_msdp_cloud_vol2/bkl-netbackup-test/msdp_cloud_vol2, /msdp/data/dp1/pdvol/data/ds_4
5, msdp_cloud_vol3, nbmsdp1_msdp_cloud_vol3/bkl-netbackup-test-air/msdp_cloud_vol3, /msdp/data/dp1/pdvol/data/ds_5
6, test, nbmsdp1_test/004-netbackup-virtual-appliance-backups/test, /msdp/data/dp1/pdvol/data/ds_6
7, dp_msdp_cloud_nbmsdp1, nbmsdp1_dp_msdp_cloud_nbmsdp1/004-netbackup-virtual-appliance-backups/dp_msdp_cloud_nbmsdp1, /msdp/data/dp1/pdvol/data/ds_7



And the 'crcontrol --dsstat --verbosecloud' command or 'crcontrol --dsstat 2' command output reveals 377GB of 'Cache' is used which is the total amount of space reserved by all the configured cloud LSU's. This space contributes to the overall used MSDP space which is why the backups to the MSDP disk pool failed with status 129.

# /usr/openv/pdde/pdcr/bin/crcontrol --dsstat --verbosecloud

********** Data Store statistics **********
Data storage Raw Size Used Avail Cache Use%
444.7G 425.8G 31.3G 394.6G 377.0G 8%

Number of containers : 211
Average container size : 111163092 bytes (106.01MB)
Space allocated for containers : 23455412514 bytes (21.84GB)
Reserved space : 20315194368 bytes (18.92GB)
Reserved space percentage : 4.3%


Each cloud LSU configured for MSDP will reserve 1 TB of space based upon the combined total of the <dedup_path>/etc/puredisk/contentrouter.cfg default CloudDataCacheSize, CloudMapCacheSize, CloudMetaCacheSize, and CloudUploadCacheSize parameters.

In the aforementioned environment, the contentrouter.cfg *CacheSize values had been changed to the following which combined and then multiplied by 5 (the total number of cloud LSUs) totals 385GB and accounts for the 'Cache' size reserved:
CloudDataCacheSize=50GiB
CloudMapCacheSize=5GiB
CloudMetaCacheSize=10GiB
CloudUploadCacheSize=12GiB
 

Solution

Increase the local MSDP volume filesystem space for the 'data' partition (preferred); or reduce the CloudDataCacheSize, CloudMapCacheSize, CloudMetaCacheSize, and CloudUploadCacheSize values in the <dedupe_path>/etc/puredisk/cloud.json file. Then restart the NetBackup Deduplication Engine (spoold) process.

/usr/openv/pdde/pdconfigure/etc/init.d/RedHat/pdservice restart spoold

Was this content helpful?