StorageOnlineOpns process fails to take lock of CAB files - Error: CCabFileMutex::wait : handle: ffffffff

StorageOnlineOpns process fails to take lock of CAB files - Error: CCabFileMutex::wait : handle: ffffffff

Article: 100004226
Last Published: 2015-04-01
Ratings: 0 0
Product(s): Enterprise Vault

Problem

StorageOnlineOpns process fails to take lock of CAB files with error "CCabFileMutex::wait : handle: ffffffff on the Cab files" during operations such as Move Archive or exporting through Compliance Accelerator or Discovery Accelerator.

Move archive(copying) fails as StorageOnlineOpns fails to take a lock of the CAB file "CCabFileMutex::wait : handle: ffffffff on the Cab files"

Enterprise Vault (EV) Compliance Accelerator or Discovery Accelerator (DA) exports / productions fail due to the same error.

Error Message

CCabFileMutex::wait[LOCK] :  Attempting to take a lock on file \\filer1
\EV_vault$\ABC\2010\01-01\A\Collection12345.cab.lock

CCabFileMutex::wait :  handle: ffffffff
 

Event ID: 6838: "Failed to recall a Saveset from its Collection. |Reason: The process
cannot access the file because it is being used by another process. 
[0x80070020] |Relative Saveset Filename: 2010\01-01\A\0E2
\ABC1234228C9A8E605798ECABF31.DVS |Relative Collection Filename: 2010\01-01
\A\Collection12345.cab |PartitionEntryID:
123454F28E5B394FAF7EBDB3135BB21C1q10000vault1 |PartitionGUID:
543211F28E5B394FAF7EBDB3135BB21C |Partition Root Path: \\filer1
\EV_vault$\ABC |Secondary Location: (null) |Migrated File Id:  |Reference:
NSDR/LMF |"

Cause

When Collections are enabled on an NTFS type Vault Store Partition, the individual DVS files that comprise a Saveset and its Single Instance Storage Parts (SISParts) are collected into CAB files for each day's Savesets and SISParts.  This makes each CAB file contain multiple files that could be needed by any EV storage related function, such as - but not limited to

  • CA or DA exporting
  • Move Archive processing
  • Backup postprocessing (looking to ensure all SISParts are present in a backup verification)

Example 1:

When CA or DA exports run, the SavesetID of each item in the export is passed to the EV Storage Service for the StorageOnlineOpns process to obtain the Saveset and any associated SISParts.  The StorageOnlineOpns process is a multi-threaded process, which allows multiple storage threads to work simultaneously to obtain Savesets and SISParts for the export. When an export needs multiple files from within a CAB file, the EV Storage service launches multiple threads to obtain those files. The first thread to access the CAB file locks the file to prevent any modifications while a copy of the needed file is extracted.

If the extraction of the Saveset or SISPart copy takes too long, the next thread that needs to access the CAB file has to wait.  If the thread has to wait too long, it notifies the Storage Service that there is a problem with the CAB file, as noted in the dtrace log as the mutex error of 'ffffffff'.  Each wait period delays the export from completing.  For large exports, these delays can cause significant export periods and can cause export failures.

The typical CA and DA export sends the SavesetIDs sequentially.  This sequential SavesetID usage contributes to multiple storage threads attempting to access the same CAB file.

 Example 2:

When using the Move Archive feature, all of the contents of the selected Archive are first copied from the source archive and pasted to the destination archive before the copied items are removed from the source archive.  As multiple items were typically archived each day into the source archive, multiple Savesets and SISParts would be needed from the same CAB file during the Move Archive processing.  The Move Archive feature is also a multi-threaded operation, so multiple storage threads would need access to the same CAB files at the same time to complete an archive move.

As with a CA or DA Export, if the extraction of the Saveset of SISPart copy takes too long, the next thread that need to access the CAB file has to wait.  If the thread has to wait too long, it notifies the Storage Service that there is a problem with the CAB file, using the same error as above.

Example 3:

The Backup postprocessing begins when the EV Storage Service is started after the completion of a backup of the Vault Stores.  Each item that was backed up is checked by the EV Storage Service to ensure it is present, including any associated SISParts.  When the Savesets or their associated SISParts are located in CAB files, the multiple threads used to confirm each Saveset and SISPart was backed up and present can cause the same CAB file locking error above to be thrown.

 


There are currently no approved workarounds to directly affect the CAB locking during any Move Archive or backup postprocessing activities.

 

Two approved workarounds exist for the CAB file locking during Discovery Accelerator (DA) exports, with one of those workarounds also approved for Compliance Accelerator (CA) exports.

Workaround for Compliance Accelerator (CA) or Discovery Accelerator (DA):

Reduce the need to access a CAB file to access multiple Savesets at the same time by lowering the number of production threads as follows:

1) Open the CA or DA Client using an account with sufficient application level permissions to adjust configuration settings (i.e., any user with the Discovery Administrator role assignment).

2) Click the Configuration tab.

3) Click the Settings sub-tab.

4) Display the hidden settings by holding the Ctrl key and clicking the banner text of Configuration

5) Click on the plus sign to the left of the Export/Production listing to expand those settings.

6) Locate the option Number of Production Threads.

7) Change the Value field entry to 1 (default value is 25).

8) Click the Save button.

9) Click the OK button to acknowledge the pop-up stating that Remoting must be restarted.

10) Close the CA or DA Client.

11) On the CA or DA Server, restart the Enterprise Vault Accelerator Manager Service (EVAMS)

12) Open the CA or DA Client.

13) Run the exports / productions as needed.

Note that this change will slow the export / production processing.  It may be possible to increase the number of production threads to optimize the system performance. To optimize the system, repeat Steps 1 through 13 above, with the exception of increasing the Number of Production Threads by increments of 5 until the errors occur again, then lower the threads by increments of 1 until the errors stop.

 Workaround for Discovery Accelerator (DA):

Randomize the SavesetIDs passed to the EV Storage Service by enabling Analytics on the DA Case in which the items to be exported have been captured.  The Analytics processing creates a hash value for each captured item.  The hash values are based on certain contents of each items are are unique to every item.  This hash value is not serialized as the SavesetIDs are.  This hash value is used by the DA export processing to provide the SavesetIDs to the EV Storage Service.  These hash values allow for some randomization of the SavesetIDs passed to the EV Storage Service, causing a lower likelihood of CAB file locking to occur.

To enable Analytics, complete the following steps:

1) Open the DA Client using an account with sufficient permissions to enable Analytics on the DA Case, such as the Case Owner.

2) Click on the Cases tab.

3) Select the Case in the filter (left) pane.

4) Click the Analytics tab above the Case's properties pane.

5) Click the Enable Analytics button.

6) Monitor the Analytics processing until all show as completed.

Once the Analytics processing has completed, configure and run the export(s) as normal.

Solution

This issue has been addressed in the following release:

Enterprise Vault 10.0.1 - Release Details
https://www.veritas.com/docs/000011056


Applies To

Enterprise Vault 8.0 or greater installation with the following:

  • NTFS type storage for the Vault Store Partitions
  • Vault Store Groups with multiple Vault Stores and Vault Store Partitions
  • Sharing enabled between the Vault Stores and Vault Store Partitions
  • Collections enabled on at least 1 Vault Store Partition.
  • Compliance Accelerator (CA) or Discovery Accelerator (DA) with at least 1 CA Department or DA Case with at least 1 search run and accepted into the review set.

 

References

Etrack : 2209713 Etrack : 2525126

Was this content helpful?