How to recover an archive after it has been deleted in a large Enterprise Vault site.

Description

There are instances when an archive has been deleted by the administrator and it needs to be restored. This document covers how to restore an archive after it has been deleted in Enterprise Vault (EV) for Microsoft Exchange in a large Enterprise Vault site with more than one Enterprise Vault server running the Storage Service.  If the site has only one Storage Service running in the environment, refer to https://www.veritas.com/docs/000047365

Before performing the following recovery procedures confirm only the Enteprise Vault data has been backed up and the registry.  Do not backup the system disks on the Enterprise Vault servers.

The procedures described in this section require backups of the following Enterprise Vault SQL databases:

• EnterpriseVaultDirectory
• EnterpriseVaultMonitoring
• EnterpriseVaultAudit
• FSA Reporting databases - if using FSA Reporting
• Fingerprint databases
• Vault store databases

Also backup the following Enterprise Vault data:

• Vault store partitions
• Index locations

Use these procedures when a recovering one Enterprise Vault server, or to recover multiple servers.

To recover each server, identify the Enterprise Vault services that were running before the disaster occurred. If unsure which Enterprise Vault services were running on each server, run the SQL script ServiceLocations.sql, which is installed in the Enterprise Vault installation folder, for example: C:\Program Files (x86)\Enterprise Vault.

Note: Before running the script, the Enterprise Vault Directory database must be restored.

Recovery procedure 1: Installing software on the servers

1. All the data relating to your previous Enterprise Vault installation needs to be recovered onto the new servers. For each server that has failed, a new computer must be set up. Ideally, set up each computer with the same name as the original computer that it is replacing.

Note: If this is not possible, the recovery steps provide addition information to accommodate a change in computer name.

2. Build each new system, starting with the installation of Windows and then all the prerequisites for Enterprise Vault. Refer to the Enterprise Vault documentation for the prerequisite software that must be installed on each computer.

3. Install Enterprise Vault on the server.

Please note the following:

Install Enterprise Vault on each new server, into the same folder as on the original server.

Install the same version of Enterprise Vault that is used in the current environment.

Do not run the Enterprise Vault Configuration wizard at the end of completing the installation of the Enterprise Vault software.

Recovery procedure 2: Restoring Enterprise Vault databases

1. Restore the following Enterprise Vault SQL databases:

• EnterpriseVaultDirectory
• EnterpriseVaultMonitoring
• EnterpriseVaultAudit
• FSA Reporting databases - if using FSA Reporting
• Fingerprint databases
• Vault store databases

If the EnterpriseVaultMonitoring or the FSA Reporting databases have been restored to a SQL server other than the one that previously hosted them, the Directory database must be updated.

2. To update the monitoring settings in the Directory database on the SQL server that hosts the Directory database, run the following SQL script:

USE EnterpriseVaultDirectory
UPDATE MonitoringSettings
SET SQLServer = 'SQL_server_name'

where SQL_server_name is the name of the new SQL server.

3. To update the FSA reporting settings in the Directory database

a. On the SQL server that hosts the Directory database, run the following SQL script to determine which SQL server hosted each FSA Reporting database:

USE EnterpriseVaultDirectory
SELECT SQLServer,DatabaseName FROM FSAReportingDatabase

b. Run the following SQL script:

USE EnterpriseVaultDirectory
UPDATE FSAReportingDatabase
SET SQLServer = 'SQL_server_name'
WHERE DatabaseName = 'FSA_reporting_database_name'

where:

SQL_server_name is the name of the new SQL server.
FSA_reporting_database_name is the name of the FSA Reporting database that was restored

Recovery procedure 3: Renaming servers

It is recommended to set up each server with the same name as the original server that it is being replaced. If unable to use the same name as the original server follow this step to set up a server with a different name than the old server.

Warning: If running Enterprise Vault in a clustered environment, do not perform this operation unless Veritas Support advises to do so.

Repeat the following steps for each server that is being recovered:

1. Run SQL Query Analyzer and connect to the server that is running the Enterprise Vault Directory service.

Run the following SQL command:

USE EnterpriseVaultDirectory
UPDATE ComputerEntry
SET ComputerNameAlternate = 'Name of new server'
WHERE ComputerNameAlternate = 'Name of old server'

2. Check that the DNS alias that was set up for the old server points to the name of the new server.  If unsure what the DNS alias is, run the following SQL query against the EnterpriseVaultDirectory database.

USE EnterpriseVaultDirectory
SELECT ComputerName FROM ComputerEntry

3. If recovering the system that provided the vault site alias (usually the first server that was added to the site), update the vault site alias to point to the new server. To do this, perform the following steps in the order listed:

a. Run SQL Query Analyzer and connect to the server running the Enterprise Vault Directory service.

b. Run the following SQL command:

USE EnterpriseVaultDirectory
SELECT SiteEntryId
FROM SiteEntry
 
The value returned contain the vault site alias at the end of a long string of numbers. For example, if the command returns the following then the vault site alias is sitealias:

10354B15D38FE5B41BAAC212490EBA5351d10000sitealias

c. In DNS, change the DNS alias entry so that it points at the new server.

Recovery procedure 4: Copy or move the Enterprise Vault data files

1. Restore the backups of the Enterprise Vault data files to their locations on the Enterprise Vault servers.

Depending on the original Enterprise Vault components that existed on the servers that are being recovered the following data files must be restored:

• If restoring a server that used to run a Storage service, or a server that is configured in a cluster, restore onto this server the saveset files for any vault stores managed by the original Storage service.
• If restoring a server that used to run an Indexing service, or a server that is configured in a cluster, restore onto this server the indexing data files managed by the original Indexing service.
• If restoring a server that used to run a Shopping service, or a server that is configured in a cluster, restore onto this server the shopping files managed by the original Shopping service.

2. The Enterprise Vault data should be restored to the locations where they existed on the original servers. For example, if recovering the server running the Indexing service and the indexing data was originally stored in the following location:

I:\Indexing

then this indexing data should be restored to the same location on the new server. To reorganize and move any SQL database devices on the disks, perform the procedures as listed in the following Microsoft Knowledge Base article: https://support.microsoft.com/?kbid=181602

3. This must be correct before starting any of the Enterprise Vault services, otherwise some cleanup operations may occur, resulting in information loss.

Recovery procedure 5: Clearing the directory database entries

The directory database entries for all of the Enterprise Vault servers in the environment can be cleared, or for selected servers. The SQL query that is provided in this section clears the entries in the database for all the Enterprise Vault servers. If there are multiple Enterprise Vault servers in the environment, you may want to recover only some of the servers. The following technical note provides alternative SQL scripts that let you specify the servers for which you want to clear directory entries:

https://www.veritas.com/docs/000006120

To clear the directory database entries for all of the Enterprise Vault servers:

1. Run SQL Query Analyzer and connect to the server running the Enterprise Vault Directory service.

2. Enter and run the following SQL command:

USE EnterpriseVaultDirectory
UPDATE StorageServiceEntry
SET StorageArchive = '', StorageRestore = '',
StorageReplayIndex = '', StorageSpool = ''
UPDATE RetrievalTask
SET RetrievalSpoolQueue = ''
UPDATE ArchivingRetrievalTask
SET MessageQueue = ''
UPDATE RetrievalTask
SET MessageQueue = ''
UPDATE JournalTask
SET MessageQueue = ''
UPDATE PublicFolderTask
SET MessageQueue = ''

Recovery procedure 6: Recreating services and tasks on the first

Directory service computer

If recovering all the Enterprise Vault servers that run a Directory service, use this procedure when recovering the first of these servers. When the subsequent servers including other servers that run a Directory service are recovered, use the procedure described in Recovery procedure 7. The Enterprise Vault Configuration wizard is able to detect missing services and tasks provided that the server name is identical to that in the original installation, and have correctly followed Recovery procedure 3.

To recreate services and tasks on the first Directory service computer:

1. Start the Enterprise Vault Configuration wizard.

2. Select Yes to create a new Directory service, and then click Next.

3. Enter the details of the Vault Service account, and then click Next.
 

• The Enterprise Vault Configuration wizard does the following:
• Converts the login for the Enterprise Vault Admin service so that it runs under the Vault Service account.
• Adds the Vault Service account to the Local Administrators group on the computer.
• Grants the user rights Log on as a service and Debug programs to the Vault Service account.
• Creates and starts the Enterprise Vault Directory service.

4. When prompted for the name of the SQL Server that will host the directory database, enter the name of SQL Server used to host the directory database for the original configuration of Enterprise Vault, and then click Next.

5. The Enterprise Vault Configuration wizard checks that the SQL Server exists and can connect to it. As long as the Directory service database has been recovered, the Enterprise Vault Configuration wizard now recreates the services and tasks installed on the Directory service computer.

6. To recreate the Enterprise Vault services on the Directory service computer, enter the password of the Vault Service account.

7. When the repair has finished, a success message is displayed.

8. If the Enterprise Vault Configuration wizard does not display a message, do not continue to run the wizard. Close the wizard and do the following:

a. Check that all previous steps have been successful, repeat any missed steps, and then run the Enterprise Vault Configuration wizard again.
b. Create a String registry value called UseLanManNameForSCM under the following registry key:

On a 64-bit installation of Windows:

HKEY_LOCAL_MACHINE
\SOFTWARE
\Wow6432Node
\KVS
\Enterprise Vault
\Admin

On a 32-bit installation of Windows:

HKEY_LOCAL_MACHINE
\SOFTWARE
\KVS
\Enterprise Vault
\Admin

c. Give UseLanManNameForSCM a value of 1.

d. Run the Enterprise Vault Configuration wizard again.

If the problem persists, contact Enterprise Vault Support for further assistance.

Recovery procedure 7: Recreating services and tasks on Enterprise Vault servers

If recovering all the Enterprise Vault servers that run a Directory service, for the first one use Recovery procedure 6.

When recovering the subsequent servers including other servers that run a Directory service, use the procedure described in this section.

The Enterprise Vault Configuration wizard is able to detect missing services and tasks provided that the server name is identical to that in the original installation, and have correctly followed Recovery procedure 3.
 
To recreate services on other Enterprise Vault servers:

1. Make sure the server running the Directory service is available on the network and the Directory service is started.

2. Make sure the Admin service is started on the local computer.

3. Start the Enterprise Vault Configuration wizard on the server.

4. When asked whether to create a directory or use an existing one, select No, use existing remote Vault Directory and enter the name of the server running the Directory service.

5. Enter the password of the Vault Service account. This is necessary to recreate the Enterprise Vault services on the computer.

6. The Enterprise Vault Configuration wizard recreates the Enterprise Vault services and tasks that used to run on the server and displays a message to indicate success.

7. If the Enterprise Vault Configuration wizard does not display a success message, do not continue to run the wizard. Close the wizard and then do the following:

a. Check that all previous steps have been successful, repeat any missed steps, and then run the Enterprise Vault Configuration wizard again.

b. Create a String registry value called UseLanManNameForSCM under the following registry key:

On a 64-bit installation of Windows:

HKEY_LOCAL_MACHINE
\SOFTWARE
Wow6432Node
\KVS
\Enterprise Vault
\Admin

On a 32-bit installation of Windows:

HKEY_LOCAL_MACHINE
\SOFTWARE
\KVS
\Enterprise Vault
\Admin

Give UseLanManNameForSCM a value of 1.

c. Run the Enterprise Vault Configuration wizard again

d. If all steps have been correctly and setting the registry key does not help, contact Enterprise Vault Support for further assistance.

8. Start all the Enterprise Vault services.

The message queues should automatically be recreated on the new server. If the Storage service is configured to start multiple processes, it may stop during message queue creation. This is because of a conflict between the processes creating the queues. To fix the problem, restart the Storage service.  If the Indexing service finds any inconsistency in the index metadata, it automatically synchronizes the metadata. The following events may be seen:

Event 41395 Index Volume metadata upgrade required

Event 41372 Index Volume metadata synchronization started

During the synchronization the Indexing service logs progress events every 10 minutes. At the end of the synchronization, one of the following events is logged:

Event 41373 Index Volume metadata synchronization completed

Event 41377 Index Volume metadata synchronization completed

The index synchronization may take some time. For example, an Enterprise Vault recommended specification server takes approximately 10 minutes to process 5,000 index volumes.  If any other index housekeeping is required there will be other progress messages every few minutes.

Recovery procedure 8: Checking the Web Access application settings

Ensure that the port and protocol settings for the Web Access application are correct. To check the Web Access application settings:

1. Open the Administration Console.

2. Expand the Enterprise Vault and Directory containers.

3. Right-click the Site entry, and then select Properties.

4. View the General page. Check that the port and protocol set for accessing the WebAccess application virtual directory, /EnterpriseVault, match the settings on the Default Web Site in IIS.

Recovery procedure 9: Checking registry entries

Check that the Enterprise Vault registry entries are all set correctly on the newly-recovered servers.

The main registry entries are under the following key:

HKEY_LOCAL_MACHINE
\SOFTWARE
Wow6432Node
\KVS
\Enterprise Vault

Additionally, the registry entries under HKEY_CURRENT_USER when logged in as the Vault Site account may need to be set. If so, restore these entries on each server too, under the following key:

HKEY_CURRENT_USER
\Software
Wow6432Node
\KVS
\Enterprise Vault

Recovery Procedure 10: Exporting the deleted archive.

Export the desired archive from the Vault Admin Console using the Export to PST option.

Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry. Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and workstation be made prior to making any registry changes.