Clearwell Enterprise Vault Collection fails on some items with error: Enterprise Vault is not running

Problem

Clearwell Enterprise Vault (EV) Collection fails when attempting to retrieve some items with error: Enterprise Vault is not running.

Error Message

Message: CONTENTMANAGEMENTAPI_E_UNAVAILABLE:Enterprise Vault is not running
Error code: 0x80040300

Cause

One of three conditions are at fault

A. DCOM ports are being blocked.

Communications with Enterprise Vault (EV) use Distributed COM services using dynamic port assignment. Firewalls can cause communications to breakdown between the EV and Clearwell servers.

B. EV Vault Store databases are offline.

Due to the new Enterprise Vault (EV) Optimized Single Instance Storage OSIS model, when an item being retrieved from Storage using the Fingerprint Database for Shared Message Objects (such as those with shared attachments or recipients spanning multiple Databases) all Vault Store Databases containing the Shared Message Objects need to be online in SQL during a StorageOnlineOpns call for batch-requested data retrievals, such as those performed by Clearwell Collections. Single item retrievals, such as Browser Search retrievals (performed by restoreo2k) target the Archive directly. Thus, Clearwell Collections (Content Retrieval) will require all Vault Store Databases within OSIS Sharing to be Online.
 

C. EV Resources are limited or services stopped preventing communications.

Once resources become limited on the server, Enterprise Vault (EV) services will stop running.

Solution

A. DCOM ports are being blocked.

1. Disable the Firewall on the Clearwell and all Enterprise Vault (EV) servers.  If the firewall must remain in place; for the Clearwell server, allow Clearwell to control the Firewall; on the EV servers, please follow the guidelines outlined in article: https://www.veritas.com/docs/000005093

2. Verify DCOM is enabled on the Clearwell server

a. Go to Start/Run.

b. Type dcomcnfg

c. Press <Enter>

d. Expand "Component Services" and "Computers" then right-click "My Computer" and choose "Properties" (Figure 1)

e. Select "Default Properties" and verify the option "Enable Distributed COM on this computer" is enabled. If not, enable the property and restart the server. (Figure 1)

Notes:
Additional DCOM logging can also be enabled, msdn.microsoft.com/en-us/library/ms686563(VS.85).aspx, and, msdn.microsoft.com/en-us/library/ms692700(VS.85).aspx 

B. EV Vault Store databases are offline.

To verify the issue using logs:

1. Review the Enterprise Vault (EV) Event Logs on the EV Storage Server for the following Event:

Event Type:     Warning
Event Source:   Enterprise Vault
Event Category: Storage Delete
Event ID:       13397
Description:
The connection 'Provider=SQLOLEDB;Server=SQLSERVER;Database=EVVAULTSTOREDATABASE;Trusted_Connection=yes' was lost and the system is waiting to reconnect (Thread Id: XXXX)

2. Run a DTrace 'StorageOnlineOpns' on the EV Storage Server as follows:

a. Log onto the EV Server as the Vault Service Account.

b. Open a 32-bit Command Prompt and navigate to the EV installation folder.

c. Type: dtrace [enter]

d. At the DT> prompt type: v [enter] (This will register and associate a number with each EV process running on the Server)

e. Record the ID number beside the process(es) to be traced. In this case: StorageOnlineOpns

f. At the DT> prompt type: set (service ID #) v [enter] (For example: set 59 v)

Enable DTrace logging by typing the following and replacing 'c:\dtrace.log' with a location and filename for the log. The log must have a .log extension.
g. At the DT> prompt type: log c:\dtrace.log [enter]

h. At the DT> prompt, leave the Command Line window open. The DTrace is now enabled. Run the Collection.

i. Once the issue has occurred, go to the Command Line window. At the DT> prompt type: quit [enter]

j. Respond accordingly to any additional prompts in order to save any logs and exit DTrace.

Review the DTrace for entries similar to:

(StorageOnlineOpns)    <XXXX> EV:M    Reconnection to the databased failed try 1234.  Sleeping for 20000 milliseconds
(StorageOnlineOpns)    <XXXX> EV:H    CADOContext::IsLoggingConnectivityIssueDue Result=No
(StorageOnlineOpns)    <XXXX> EV:M    The SQL database server for EVVAULTSTOREDATABASE is not available: |Check that the SQL Server is valid and is running|Internal reference(CC1)|Additional Microsoft supplied information:|Source: Microsoft OLE DB Provider for ODBC Drivers|Number:       0x80004005|SQL State:    37000|Native Error: 4060|HRESULT|0xc0043433

Solution:

Bring the Database listed (EVVAULTSTOREDATABASE in the above example) back Online on the SQL Server (SQLSERVER in the example above). Once the Database is back Online, the Collections should complete successfully.

 
C. EV Resources are limited or services stopped preventing communications.

1. Always use 3 retries on the Clearwell Task to allow for retries on the items (in production).

2. If this is an Active Journaling site (backlog of J MSMQ's), run the Collections during slow production periods.

3. Limited Hard Drive resources for the Vault Service Account (VSA) temp folder. By default, this is located on the C drive. Veritas recommends to have a minimum of 10GB free space.

4. Exclude Anti-Virus scanning on the VSA temp folder on the EV server.

5. Disable TCP Chimney and TOE on the EV server and Clearwell Server. See: https://www.veritas.com/docs/000094937

6. Increase CPU's and memory on the EV Storage servers

7. Place data on faster storage devices.