Under rare circumstances, intermittently NetBackup (NBU) Agent fails to clear the backup mode for random Enterprise Vault (EV) index locations.

Problem

Under rare circumstances, intermittently NetBackup (NBU) Agent fails to clear the backup mode for random Enterprise Vault (EV) index locations.

Error Message

In NetBackup BPResolver Logs, it can be observed that PowerShell process gets timed out:

04:30:26.855 [20796.18564] <2> debuglog: <2> DebugLog(): INF - CPSCmdLetHandler::ExecutePowerShellCommand - entering

04:30:26.855 [20796.18564] <2> debuglog: <2> DebugLog(): INF - CPSCmdLetHandler::ExecutePowerShellCommand - PS Command: ("C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe" -PSConsole "D:\Program Files (x86)\Enterprise Vault\evshell.psc1" -command "& { Set-IndexLocationBackupMode -EntryId 16F7C208DBDAC6B42B41C194D526141BC1810000evmb ; $error | out-file 'C:\Program Files\VERITAS\\NetBackup\logs\beds\EV-Backup_083026.log' }")

04:50:26.714 [20796.18564] <2> debuglog: <2> DebugLog(): INF - CPSCmdLetHandler::ExecutePowerShellCommand - PowerShell process TIMEOUT exit. Timeout = 1200000 MilliSeconds

04:50:26.714 [20796.18564] <2> debuglog: <2> DebugLog(): INF - CPSCmdLetHandler::ExecutePowerShellCommand - exiting 0x102.

04:50:26.714 [20796.18564] <2> debuglog: <2> DebugLog(): INF - CPSCmdLetHandler::ModifyBackupModeStatus - Failed to execute backup mode powershell command. Error 0x102

In Enterprise Vault Dtrace log, it can be observed that when putting the indexes in backup mode EV does not receive the acknowledgement from the indexing engine for random index volumes:

928799  04:31:10.366      [16876]               (EVIndexVolumesProcessor)         <43628>              EV-H            {IndexVolumesManager} Access mode [BackupMode] has not been applied to index volume [1AAB098E9B30EDD40B4ADF85C3B40029A1110000evmb_32470]. The operation may still be pending.

Cause

Under rare conditions, the EV thread which performs the changes to the index volume backup mode goes into a blocked state. The blocked thread causes the NBU agent process PS command executer to halt, and eventually timeout.

NBU agent considers it as a failure and tries to put the Index Volume in back mode again. This causes the Index Volume to go into backup mode two times and its reference count increases accordingly.

At the time of clearing the backup mode, NBU agent calls Clear-IndexLocationBackupMode that removes the Index Volume from backup mode only once, and the reference count is reduced by one. The Index Volume remains in backup mode as the reference count does not become zero. This happens intermittently with random Index Volumes.

Solution

Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry. Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry be made prior to making any registry changes

In such situations, the backup mode reference count can be overwritten by using the switch -ForceClearBackupMode with the PowerShell command Clear-IndexLocationBackupMode. This PowerShell switch is now configurable via below registry key:

Name: ForceClearBackupMode

Location: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\KVS\Enterprise Vault\Indexing

Type: REG_DWORD

Set ForceClearBackupMode to 1 to enable and 0 to disable.

Note: This registry key can only take effect on EV versions 14.3.1 and later.