Space thresholds exceeded warning even when space looks ok.

Problem

Errors relating to Thresholds being exceeded may be seen causing backups to fail. Checking the space on the content router (df) appears to show that there is enough space and that no thresholds have been exceeded.

Error Message

Errors can be seen in the the /Storage/log/spoold/spoold.log that a space threshold has been reached

Partition containing database at /Storage/databases: 399.98 GiB, 178.20 GiB free (44.55%) of which 158.71 GiB is required as margin for temporary files (39.68%) (low space threshold exceeded).
Previous state: 399.98 GiB, 178.35 GiB free (44.59%) of which 158.71 GiB is required as margin for temporary files (39.68%) (warning threshold exceeded).
The Content Router has insufficient disk space to accept new data. No new data will be accepted until more disk space becomes available. Rerouting, data removal and/or garbage collection is needed urgently.

Cause

Checking the content router appears to show that space is not a problem in /Storage/data

# df -kl /Storage /Storage/data
Filesystem               1K-blocks   Used       Available  Use%  Mounted on
/dev/vx/dsk/dg1/Storage  419407576   206280786  199815802  51%   /Storage
/dev/vx/dsk/dg2/data     8589366256  6703831136 1870805048 79%   /Storage/data

# /opt/pdcr/bin/crcontrol --dsstat
************ Data Store statistics ************
Data storage       Size   Used  Avail Use%
                   8.0T   6.0T   2.0T  75%
Average container size           : 216240196 bytes (206.22MB)
Space allocated for containers   : 6824108107654 bytes (6.21TB)
Space used within containers     : 6586051908465 bytes (5.99TB)
Space available within containers: 238056199189 bytes (221.71GB)
Space needs compaction           : 32459024007 bytes (30.23GB)
Records marked for compaction    : 330617
Active records                   : 69218097
Total records                    : 69548714

Checking the threshold settings in the /etc/puredisk/contentrouter.cfg file these are set to the following

WarningSpaceThreshold=20%
LowSpaceThreshold=5%
VeryLowSpaceThreshold=3%

With the content router only 75% utilised this should be well within these thresholds.

We can see from the df output above that /Storage is approximately 400GB in size and /Storage/data is 8TB. This is below the minimum system requirements. Puredisk requires that /Storage be at least 15% of the size of /Storage/data in order to have space to create temporary files during backups.

The thresholds used for calculating space for tempory files in /Storage are the same thresholds that the content router uses in /etc/puredisk/contentrouter.cfg, which can be confusing.

Looking at the error It showed there was 44.59% free space in /Storage/databases but it needed 39.68% for temporary files, so that is very near the 5% that is set for the LowSpaceThreshold.

Solution

Ensure that the content router is correctly configured with /Storage having enough space in relation to the amount of /Storage/data. We recommend that /Storage be 15% of the total size of /Storage/data.

As a workaround the thresholds in /etc/puredisk/contentrouter.cfg can be changed (via the PureDisk GUI) but this must be done with caution as reducing them too much can have knock on effects with space being used up.