The Enterprise Vault (EV) StorageFileWatch and/or StorageArchive process returns Event ID: 28944 when attempting to verify an item on a 'Hitachi Content Platform (HCP) Streamer' device.

Problem

This issue can be seen in the StorageFileWatch or StorageArchive processes. When the issue occurs, the following behaviors can be seen:-

1. In the StorageArchive process, the issue occurs on one thread. Any work that is performed on that thread generates the error. The result is that the system has a broken storage archive thread that will constantly generate the error. Items will eventually be retried on a different thread so the issue does not stop the archiving of items. The result of the issue occurring will be the constant generation of the error in the event log and a very slight impact to archiving rate due to one thread being broken.

2. In the StorageFileWatch process this error can stop post processing from functioning correctly. This will eventually result in the informational message appearing in EV system status advising that x amount of items have not been backed up. If journal archiving is configured, this issue can cause a backlog of items in the journal mailbox.

Error Message

StorageFileWatch process Event Log errors

Log Name: Veritas Enterprise Vault
Source: Enterprise Vault
Event ID: 28944
Task Category: Storage File Watch
Level: Error
Computer: EVSERVER.domain.com
Description:
The 3rd party storage system application 'Hitachi Content Platform (HCP) Streamer' has logged the following message:
CStreamerObject::Info method failed
Reason = 0x8007023f
Description = Object not found. StoreIdentifier = 'evroot/3720648DF19B6E4CBED201E1D8AF0D7B/2016/Jan/15/6c726f48-22a9-49e3-9444-0dcc7a26f008/14_31_43/00188'.
ConnectionManager::Set_Curl_Operation method error (Curl error) Reason = 2 - easy handle already used in multi handle

v-437-28944

Log Name: Veritas Enterprise Vault
Source: Enterprise Vault
Event ID: 28944
Task Category: Storage File Watch
Level: Error
Computer: EVSERVER.domain.com
Description:
The 3rd party storage system application 'Hitachi Content Platform (HCP) Streamer' has logged the following message:
CStreamerObject::Info method failed
Reason = 0x80004005
Description = Object not found. StoreIdentifier = 'evroot/4C3043502693F94384572083D5D0B8FB/2012/Dec/19/55a408ad-1d6e-4f77-ab45-ea47c09d9999/01_07_26/00014'. 
 V-437-28944

A 6720 event may also occur indicating a connection issue to the HCP device:

Source:        Enterprise Vault
Event ID:      6720
Task Category: Storage File Watch
Level:         Error
Computer:      EVSERVER.domain.com
Description:
FileWatch scan thread for partition 'VaultStore02 Ptn1' cannot access the partition location 'hcp01.domain.com'. FileWatch will wait for the rescan interval and try again.

Error: Unspecified error  (0x80004005)
V-437-6720

StorageArchive process event log errors

Source:        Enterprise Vault
Event ID:      28944
Task Category: Storage Archive
Level:         Error
Computer:      EVSERVER.domain.com
Description:
The 3rd party storage system application 'Hitachi Content Platform (HCP) Streamer' has logged the following message:
CStreamerObject::Info method failed
Reason = 0x80070057
Description = Required input parameter is NULL. Parameter name = 'StoreIdentifier'.

Source:        Enterprise Vault
Event ID:      28944
Task Category: Storage Archive
Level:         Error
Computer:      EVSERVER.domain.com
Description:
The 3rd party storage system application 'Hitachi Content Platform (HCP) Streamer' has logged the following message:
CStreamerObject::Write method failed
Reason = 0x8007023f
ConnectionManager::Set_Curl_Operation method error (Curl error) Reason = 2 - easy handle already used in multi handle

Cause

The issue is caused by a catastrophic failure when writing the item to the HCP device. Setting up a DTRACE or Backtrace will show the following information:-

Error on the StorageArchive CStreamerObject::Write function
(StorageArchive) <16288> EV:M Hitachi Content Platform (HCP) Streamer: CStreamerObject::Write Entry
(StorageArchive) <16288> EV:M Hitachi Content Platform (HCP) Streamer: CStreamerObject::Write MetaData = '<EnterpriseVaultFile Version="1" FileIdentifier="3122B843C69D8BEB9278236EC1ECAA41" FileType="DVS" ArchivedDateUTC="42186.2388888889" PartitionEntryId="1874A68E87DD4D940B51E233795AC5FDB1q10000vault.ev.com"/>', RetentionType = 'Definite', RetentionDate (UTC) = '2025-07-01 05:44:00'
(StorageArchive) <16288> EV:M Hitachi Content Platform (HCP) Streamer: CStreamerObject::Write IEVContentStreamerCallback::GetProperty returned partitionGUID = '874A68E87DD4D940B51E233795AC5FDB'
(StorageArchive) <16288> EV:M Hitachi Content Platform (HCP) Streamer: CStreamerObject::Write root path for write is 'evroot/874A68E87DD4D940B51E233795AC5FDB'
(StorageArchive) <16288> EV:M Hitachi Content Platform (HCP) Streamer: CStreamerObject::Write Size of input content stream = '76329'
(StorageArchive) <16288> EV:H {CStreamerAPIDeviceWriter::StoreUserPart:#201} Exception [Catastrophic failure (0x8000ffff)]
(StorageArchive) <16288> EV:H {CStreamerAPIDeviceWriter::StoreUserPart} (Exit) Status: [Catastrophic failure (0x8000ffff)]
(StorageArchive) <16288> EV:H {CSISPartStripper::StoreSavesetAndSisParts:#56} _com_error exception: [Catastrophic failure (0x8000ffff)]
(StorageArchive) <16288> EV:H {CSISPartStripper::StoreSavesetAndSisParts} (Exit) Status: [Catastrophic failure (0x8000ffff)]
(StorageArchive) <16288> EV:H {CSavesetSISManager::StoreSISParts:#339} _com_error exception: [Catastrophic failure (0x8000ffff)]
(StorageArchive) <16288> EV:H {CSavesetSISManager::StoreSISParts} (Exit) Status: [Catastrophic failure (0x8000ffff)]
(StorageArchive) <16288> EV:H {CSavesetSISManager::StoreSaveset:#172} _com_error exception: [Catastrophic failure (0x8000ffff)]Once the error has occurred the archiving attempt is rolled back:-
(StorageArchive) <16288> EV:L {CSavesetSISManager::RollbackSIS} (Entry)(StorageArchive) <16288> EV:M {CSavesetSISManager::RollbackSIS:#556} Commencing SIS rollback...Once the original Catastrophic failure has taken place all subsequent attempts to archive items on that thread will fail with the following CURL error:- (StorageArchive) <16288> EV:M Hitachi Content Platform (HCP) Streamer: CStreamerObject::Write Entry
(StorageArchive) <16288> EV:M Hitachi Content Platform (HCP) Streamer: CStreamerObject::Write MetaData = '<EnterpriseVaultFile Version="1" FileIdentifier="3122B843C69D8BEB9278236EC1ECAA41" FileType="DVS" ArchivedDateUTC="42186.2388888889" PartitionEntryId="1874A68E87DD4D940B51E233795AC5FDB1q10000vault.ev.com"/>', RetentionType = 'Definite', RetentionDate (UTC) = '2025-07-01 05:44:00'
(StorageArchive) <16288> EV:M Hitachi Content Platform (HCP) Streamer: CStreamerObject::Write IEVContentStreamerCallback::GetProperty returned partitionGUID = '874A68E87DD4D940B51E233795AC5FDB'
(StorageArchive) <16288> EV:M Hitachi Content Platform (HCP) Streamer: CStreamerObject::Write root path for write is 'evroot/874A68E87DD4D940B51E233795AC5FDB'
(StorageArchive) <16288> EV:M Hitachi Content Platform (HCP) Streamer: CStreamerObject::Write Size of input content stream = '76329'
(StorageArchive) <16288> EV:M Hitachi Content Platform (HCP) Streamer: ConnectionManager::Set_Curl_Operation method error (Curl error) Reason = 2 - easy handle already used in multi handle
(StorageArchive) <16288> EV~E Event ID: 28944 The 3rd party storage system application 'Hitachi Content Platform (HCP) Streamer' has logged the following message: |CStreamerObject::Write method failed|Reason = 0x8007023f|ConnectionManager::Set_Curl_Operation method error (Curl error) Reason = 2 - easy handle already used in multi handle Error on the StorageFilewatch CStreamerObject::Info function
(StorageFileWatch) <12345> Hitachi Content Platform (HCP) Streamer: CStreamerObject::Info Entry
(StorageFileWatch) <12345> Hitachi Content Platform (HCP) Streamer: CStreamerObject::Info StoreIdentifier = 'evroot/57BF32C26B61234C973A4DC0400F6A26/2015/Nov/12/d7220b42-728d-40f0-bd21-6fd79af654a2/06_06_57/02394'
(StorageFileWatch) <12345> Hitachi Content Platform (HCP) Streamer: ConnectionManager::Set_Curl_Operation method https://EV.hcp.prod.main.EV/rest/evroot/57BF32C26B61234C973A4DC0400F6A26/2016/Apr/18/d7220b42-728d-40f0-bd21-6fd79af654a2/06_06_57/02394 Reason = 200 : OK - opr_HEAD
(StorageFileWatch) <12345> {CStreamerItemSecuredVerifier::CheckItem:#471 Exception [Catastrophic failure  (0x8000ffff)]
(StorageFileWatch) <12345> {CStreamerItemSecuredVerifier::CheckItem} (Exit) Status: [Catastrophic failure  (0x8000ffff)]Once that error is generated the following trace can be seen for subsequent post processing attempts:-
(StorageFileWatch) <12345> Hitachi Content Platform (HCP) Streamer: ConnectionManager::Set_Curl_Operation method error (Curl error) Reason = 2 - easy handle already used in multi handle
(StorageFileWatch) <12345> EV~E Event 28944    Event ID: 28944 The 3rd party storage system application 'Hitachi Content Platform (HCP) Streamer' has logged the following message: |CStreamerObject::Test method failed|Reason = 0x8007023f|ConnectionManager::Set_Curl_Operation method error (Curl error) Reason = 2 - easy handle already used in multi handle |
(StorageFileWatch) <12345> Hitachi Content Platform (HCP) Streamer: CStreamerObject::Test Exiting with HRESULT 0x8007023f
(StorageFileWatch) <12345> EV~E Event 6720  Event ID: 6720 FileWatch scan thread for partition 'VaultStore1 HCP Ptn1' cannot access the partition location 'EV.hcp.prod.main.EV'. FileWatch will wait for the rescan interval and try again. |Error:  The application was unable to start correctly (0xlx). Click OK to close the application.  (0x8007023f) |

Solution

Workaround
To resolve the CURL multi handle issue, restart the Enterprise Vault Storage Service. This will resolve the issue for an undetermined amount of time until the catastrophic error is generated again.

Solution
This issue is twofold.  The StorageFileWatch process generated error can be resolved by upgrading the HCP Streamer software on the EV server to the latest version. At this time of writing (January 2017) this is 1.3.  UPDATE October 2019. This issue still occurs in HCO streamer version 1.4.4.3.  Please contact Hitachi to obtain the latest version of the streamer.

The StorageArchive process generated error referenced above is still under investigation by Hitachi, as this is related to the HCP Adapter (Streamer software) that Hitachi produce for Enterprise Vault to connect and utilize an HCP Platform.  It should be noted however, as previously stated,  Items will eventually be retried on a different thread so the issue does not stop the archiving of items. The result of the issue occurring will be the constant generation of the error in the event log and a very slight impact to archiving rate due to one thread being broken.

Veritas has acknowledged that the above-mentioned issue is present in the version(s) of the product(s) referenced in this article.
 
This issue is currently under investigation by Veritas. Pending the outcome of the investigation, this issue may be resolved by way of a cumulative hotfix or service pack in the current or future versions of the software. However, this particular issue is not currently scheduled for any release.  If you feel this issue has a direct business impact for you and your continued use of the product, please contact your Veritas Sales representative or the Veritas Sales group to discuss these concerns.  For information on how to contact Veritas Sales, please see https://www.veritas.com .
 
Please be sure to refer back to this document periodically as any changes to the status of the issue will be reflected here.