Unable to search and index when receiving "Error in deserializing body of reply message for operation 'SearchCollectionStatus' ".

Problem

When an index reports "Error in deserializing body of reply message for operation 'SearchCollectionStatus' " the index will become unsearchable. In addition, any attempts to synchronize the index will fail and no additions to the index will be apparent.

Error Message

The report from a failed index synchronization will show the following:

13/05/2014 11:48:59 The synchronize subtask has failed.
Error Code: 57
Description:Indexing engine unreachable. Error in deserializing body of reply message for operation 'SearchCollectionStatus'. There is an error in XML document (1, 4760). Value was either too large or too small for an Int32.

A dtrace of 'EVIndexVolumesProcessor' when performing an index synchronization will show the following:

11:25:56.480  [7808] (EVIndexVolumesProcessor) <Agent Thread for 19A59BD9DA7C0C444B90A85E98BB40C511110000evjournal1_635:12504> EV-H {IndexingWCFProxy`1} Exception: Error in deserializing body of reply message for operation 'SearchCollectionStatus'.
Info: Diag: Type:System.ServiceModel.CommunicationException ST:|Server stack trace: |   at System.ServiceModel.Dispatcher.XmlSerializerOperationFormatter.DeserializeBody(XmlDictionaryReader reader, MessageVersion version, XmlSerializer serializer, MessagePartDescription returnPart, MessagePartDescriptionCollection bodyParts, Object[] parameters, Boolean isRequest)|   at System.ServiceModel.Dispatcher.XmlSerializerOperationFormatter.DeserializeBody(XmlDictionaryReader reader, MessageVersion version, String action, MessageDescription messageDescription, Object[] parameters, Boolean isRequest)|   at System.ServiceModel.Dispatcher.OperationFormatter.DeserializeBodyContents(Message message, Object[] parameters, Boolean isRequest)|   at System.ServiceModel.Dispatcher.OperationFormatter.DeserializeReply(Message message, Object[] parameters)|  
at System.ServiceModel.Dispatcher.ProxyOperationRuntime.AfterReply(ProxyRpc& rpc)|  
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)|  
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)|  
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)|  
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)|Exception rethrown at [0]: |  
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)|  
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)|  
at EnterpriseVault.Indexing.IndexingEngineService.VelocitySrvRef.VelocityPort.SearchCollectionStatus(SearchCollectionStatusRequest request)|  
at lambda_method(ExecutionScope , VelocityPort )|   at KVS.EnterpriseVault.Runtime.WCF.WCFProxy`1.Invoke[TResult](Expression`1 method, String server)|  
at KVS.EnterpriseVault.Runtime.WCF.WCFProxy`1.Invoke[TResult](Expression`1 method)|  
at EnterpriseVault.Indexing.Common.IndexingWCFProxy`1.Invoke[TResult](Expression`1 method)
Inner:System.InvalidOperationException: There is an error in XML document (1, 2541). ---> System.OverflowException: Value was either too large or too small for an Int32.|  
at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info)|  
at System.Xml.XmlConvert.ToInt32(String s)|  
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVelocityPort.Read52_vseindexfile(Boolean isNullable, Boolean checkType)|  
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVelocityPort.Read83_vseindexstatus(Boolean isNullable, Boolean checkType)|  
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVelocityPort.Read87_vsestatus(Boolean isNullable, Boolean checkType)|  
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVelocityPort.Read88_SearchCollectionStatusResponse(Boolean isNullable, Boolean checkType)|  
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVelocityPort.Read590_Item()|  
at Microsoft.Xml.Serialization.GeneratedAssembly.ArrayOfObjectSerializer115.Deserialize(XmlSerializationReader reader)|  
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)|   --- End of inner exception stack trace ---|  
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)|  
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle)|  
at System.ServiceModel.Dispatcher.XmlSerializerOperationFormatter.DeserializeBody(XmlDictionaryReader reader, MessageVersion version, XmlSerializer seri

The WCF logs will show the following for the problematic thread:

<t:SearchCollectionStatusResponse xmlns:t="urn:/velocity/types" xmlns="urn:/velocity/objects">
<vse-status version="6.0" which="live" token="U018y3gQDjio" config-md5="6005674986c8adff74e19332233a8796" crawler-config-md5="3e2c0813f1935990795398aec9d12cff" indexer-config-md5="a6e6c9110b8dc769fa3432e332388c3e">
<crawler-status config-md5="3e2c0813f1935990795398aec9d12cff" time="1399400695" elapsed="1059391" this-elapsed="1401" host="all" n-input="0" n-output="0" n-errors="0" n-http-errors="0" n-http-location="0" n-filtered="0" n-robots="0" conversion-time="0" n-sub="0" n-bytes="0.000000" n-dl-bytes="0.000000" converted-size="0.000000" n-pending="0" n-redirect="0" n-duplicates="0" n-deleted="0" n-cache-complete="0" n-awaiting-index-input="0" n-offline-queue="0" n-awaiting-index-reply="0" n-pending-internal="0" n-awaiting-gate="0" n-awaiting-input="0" complete="complete" idle="idle" expected-stop-time="1399400750" version="255fd9527e3ee4eb5416ae78ecfba531
" id="U2kjfmzFnyMc" service-status="stopped" error="Unknown reason">
 

<crawl-hop hop="0" n="9223372036854768876"></crawl-hop>

NOTE: The value above may vary, but will generally begin with '9223' indicating an overflow of a property or an unexpected value.

Cause

The cause of this issue is a value that is being received for a particular property causing an overflow.

Workaround:

Restore the index folder from a point in time before the issue occurred and perform a synchronization or rebuild the index.

NOTE: Different variations of this issue have been addressed in Enterprise Vault 10 SP2 and SP4. This particular instance has not been addressed. Therefore it is recommended to upgrade the environment to Enterprise Vault 10 SP4 CHF3 and validate that your issue has not already been addressed.

Solution

Enterprise Vault 11.0.1 - Release Details
http://www.veritas.com/docs/000004504

Applies To

Enterprise Vault 10 SP4 CHF1, CH2, and CHF3.

Enterprise Vault 11.0.0