Content Router Compaction Failing

Article: 100006281
Last Published: 2012-07-28
Ratings: 0 0
Product(s):

Problem

Unable to do backups or in-bound replications to the PureDisk storage pool with and out of space error despite system reporting several terabytes of space available.

 

# /opt/pdcr/bin/crcontrol --dsstat

************ Data Store statistics ************
Data storage       Size   Used  Avail Use%
                  7.0T   1.9T   5.1T  28%
Number of containers             : 26676
Average container size           : 267927025 bytes (255.52MB)
Space allocated for containers   : 7147221326288 bytes (6.50TB)
Space used within containers     : 7143221038089 bytes (6.50TB)
Space available within containers: 4000288199 bytes (3.73GB)
Space needs compaction           : 5042506643390 bytes (4.59TB)
Records marked for compaction    : 56114820
Active records                   : 24170344
Total records                    : 80285164

The issue is that the space available consists of 2 parts: space which is availble to be used now and the space which needs to be compacted within the datastore containers before it can be used. From the above output we an see that 4.59TB is waiting to be compacted.

Normally the amount to be compacted should be at most a few 100gb even on large StoragePools. Compaction is an internal Content Router process which checks every 20 seconds (by default) if any of the datastore containers required compaction. To check that compaction is enabled execute "/opt/pdcr/bin/crcontrol --compactstate" and you should see the following output:

# /opt/pdcr/bin/crcontrol --compactstate
Data store compaction: ON, DeleteSpaceThreshold: 30%, CompactLBound: 4MB
Compaction busy: No

To manually start a compaction run execute "/opt/pdcr/bin/crcontrol --compactstart" and check the status in the /Storage/log/spoold/spoold.log.

Error Message

September 07 15:41:23 INFO [1076382016]: COMPACT: dcOutPlaceUpdate dcid 17397
September 07 15:41:23 ERR [1076382016]: 25002: dcOutPlaceUpdate: failed to get stat of /Storage/data/journal/0.bhd (No such file or directory)
September 07 15:41:23 ERR [1076382016]: 25002: dcCompactContainer: dcOutPlaceUpdate failed: no such object)

Cause

One or more of the intermediatary files which compaction uses was missing and causing the failure.

Solution

The solution is to shutdown the PureDisk Content Router process (when no backups or other activities are running) and recreate the files:

# /etc/init.d/puredisk stop pdcr
# rm /Storage/data/journal/*.bhd
# rm /Storage/data/journal/*.bin
# rm /Storage/data/journal/compactd.spare
# touch /Storage/data/journal/0.bin
# touch /Storage/data/journal/0.bhd
# touch /Storage/data/journal/compactd.spare
# chmod 750 /Storage/data/journal/0.*
# chmod 750 /Storage/data/journal/compactd.spare
# /etc/init.d/puredisk start pdcr

Once the Content Router process has reloaded you should be able to start compaction "/opt/pdcr/bin/crcontrol --compactstart" and monitor the results in /Storage/log/spoold/spoold.log.

Also running "/opt/pdcr/bin/crcontrol --dsstat" will show that the Space needing compaction is reducing:

# /opt/pdcr/bin/crcontrol --dsstat

************ Data Store statistics ************
Data storage       Size   Used  Avail Use%
                  7.0T   1.9T   5.1T  28%
Number of containers             : 26676
Average container size           : 267916962 bytes (255.51MB)
Space allocated for containers   : 7146952890832 bytes (6.50TB)
Space used within containers     : 6915591089046 bytes (6.29TB)
Space available within containers: 231361801786 bytes (215.47GB)
Space needs compaction           : 4814876694347 bytes (4.38TB)
Records marked for compaction    : 52321584
Active records                   : 24170344
Total records                    : 76491928

 


Applies To

PureDisk 6.6.1.2

Was this content helpful?