Windows System State backup fails with Status 69 if any mounted volume's capacity is greater than 63 TB

Problem

Windows System State backup fails with status 50 / status 69 if any of the client's mounted volume capacities are greater than 63 TB, even if the volume is empty, excluded from the backup, or if the volume is hidden.

Error Message

Job details shows:

Error bpbrm(pid=xxxx) from client <client-name>: ERR - Unable to backup System State or Shadow Copy. Please check the state of VSS and associated Writers.INF - EXIT STATUS 69: invalid filelist specification

Job may end with status:

client process aborted  (50)

Verbose client bpbkar log shows:

12:50:11.426 PM: [3256.9020] <2> ov_log::V_GlobalLog: INF - ERROR: Disc \\?\Volume{cc0aedaa-a407-11e0-b2ab-78e7d190c93a}\ is greater than defined threshold of 63 TB
12:50:11.426 PM: [3256.9020] <2> ov_log::V_GlobalLog: INF - EXIT VssSnapshotVolume::CheckForUnsupportedDiscs() return=[0x8004230c VSS_E_VOLUME_NOT_SUPPORTED]!
12:50:11.426 PM: [3256.9020] <2> ov_log::V_GlobalLog: INF - EXIT VssSnapshotVolume::Initialize() return=[0x8004230c VSS_E_VOLUME_NOT_SUPPORTED]!
...
12:51:29.066 PM: [3256.9020] <2> ov_log::V_GlobalLog: INF - Status E_FAIL (0x80004005) returned getting Backup Coordinator when initializing shadow copy
12:51:29.066 PM: [3256.9020] <2> ov_log::V_GlobalLog: INF - AD:Status FS_SNAP_SHOT_COORDINATOR_FAILURE (0xE000FEC3) attaching 'Shadow?Copy?Components' in SystemState::InitializeShadowCopy:368
12:51:29.066 PM: [3256.9020] <2> ov_log::V_GlobalLog: INF - AD:Status FS_SNAP_SHOT_COORDINATOR_FAILURE (0xE000FEC3) initializing Shadow Copy in SystemState::AttachToDLE:710
12:51:29.066 PM: [3256.9020] <2> ov_log::V_GlobalLog: ERR - BEDS_AttachToDLE():FS_AttachToDLE() DeviceName:'System?State' BackupReason:0x400 Failed! (0xE000FEC3:A failure occurred accessing the SnapShot coordinator.
12:51:29.066 PM: [3256.9020] <2> ov_log::V_GlobalLog: INF - v_beds::V_FindFirst() EXIT Name: bRC:false LastError:0xFFFFFEC3:0xa000fec3 (2684419779)

Cause

Mounted volumes, both visible and hidden on the Windows server are probed and discovered to have a capacity greater than the default setting of 63 TB as configured in the NetBackup BEDS registry settings.

Troubleshooting

Verbose client bpbkar logs are required:

• On the Windows client, create directory: install path\NetBackup\logs\bpbkar
• Enable verbose log settings on client through client Backup Archive and Restore GUI
• Start > Programs > NetBackup > Backup Archive and Restore >
• Click "File" on the top bar of the GUI window > NetBackup Client Properties >
• Troubleshooting tab > set General=2, Verbose=5, click OK.

Solution

To resolve this issue, on the client, modify a NetBackup BEDS setting in the Windows registry key: HKLM\SOFTWARE\Veritas\NetBackup\BEDS\Engine\Misc\BescLargeDiscBlock to be greater than the capacity of the largest volume on the server.

To find all volumes and their sizes on a server, open a Command Prompt window and use the OS diskpart command:

Example:

C:\> diskpart
DISKPART> list volume
   < Disk outputs here >
DISKPART> exit

For example, if the server's largest volume's capacity is 99 TB, then the registry value should be set to 110 decimal or more. The mathematics relates to 1024 bytes per kilobyte. Hence, it is recommended to set a value at least 10% larger than the size of the disk.

This configuration key 'BescLargeDiscBlock' should exist by default, however If this key does not exist already, it will be necessary to create it.

HKLM\SOFTWARE\Veritas\NetBackup\BEDS\Engine\Misc

BescLargeDiscBlock          REG_DWORD
Default Value = 63 decimal   /   3f  hex

Note that in some cases a reboot may be necessary after performing the registry edit in order for the change to completely take effect.

• Additional information: It has been observed that the sum total of the size of all volumes needs to be entered rather than the size of the largest single volume on some systems. If the backups continue to fail after entering the value for the largest volume, try changing the value to the number representing the size of all volumes added together.
• Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry. Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and machine be made prior to making any registry changes.
• 3rd Party Microsoft Reference: Usability limit for Volume Shadow Copy Service - Windows Server | Microsoft Docs