Enterprise Vault (EV) Vault Stores or Index Locations stay in Backup Mode even though PowerShell scripts cleared backup mode successfully

Problem

When using the PowerShell Scripts to clear Backup Mode the command says it successfully cleared but the Vault Stores and/or Index Locations still stay in Backup Mode.

The events that signal Backup Mode being cleared from Storage Management and Index Admin Service are not reported in the Veritas Enterprise Vault Event Log. In the case of the Indexing Service the event that sets Backup Mode will be reported even when the call to clear it is being made.

Error Message

The following dtrace logs demonstrated the cause of the issue:-

(StorageManagement)    <8576>    EV:L    CStorageManage::CheckIfAllVaultStoresNeedBackupModeTransition: Information: VaultStoreID=[16308951FE698EF4B8A976AB12AA20B511210000EV], VSName: [Mailbox Vault Store] state=[2], RefCount:[3]
(StorageManagement)    <8576>    EV:L    CStorageManage::IsStorageServiceIdValidForThisServer: entry
(StorageManagement)    <8576>    EV:L    CStorageManage::IsStorageServiceIdValidForThisServer: exit
(StorageManagement)    <8576>    EV:L    CStorageManage::CheckIfAllVaultStoresNeedBackupModeTransition: Information: No need to notify storage file watch for the vault store . VaultStoreID=[16308951FE698EF4B8A976AB12AA20B511210000EV]
91    05:17:07.686     [8892]    (StorageManagement)    <8576>    EV:L    {CStorageManage::CheckIfAllVaultStoresNeedBackupModeTransition} (Exit) Status: [Success]
 

Cause

Enterprise Vault keeps track of Backup Mode when the PowerShell scripts are used in its database. When a request is sent to set Backup Mode a count goes up by one. When the request to clear it the subsequently goes down by one. If two scripts are called to set Backup Mode but one completes before the other the request to clear,  it will decrease the count but since it has not reached 0 the Vault Store and/or Index Location will stay in Backup Mode. This is by design to allow both scripts to complete and the operations to be performed while the system is in Backup Mode.

Solution

There are two options to clear the Backup Mode completely by forcibly zeroing the value of RefCount (as shown above in the dtrace logs) from the Enterprise Vault database.

Option 1:

Clear the Backup Mode from the Vault Administration Console (VAC). 

Option 2:

Run the Powershell commands with the following additional parameter:-

 -ForceClearBackupMode 1 

e.g.

 Clear-VaultStoreBackupmode -Name 'Vault_Store_Group_Name' -EVServerName EV_Server_Name -EVObjectType 'VaultStoreGroup' -ForceClearBackupMode 1 

 To suppress the prompt from the command above, another parameter can be used as below:-

 -SuppressConfirm

e.g.

 Clear-VaultStoreBackupmode -Name 'Vault_Store_Group_Name' -EVServerName EV_Server_Name -EVObjectType 'VaultStoreGroup' -ForceClearBackupMode 1 -SuppressConfirm