Discovery Accelerator Searches cause index volumes to fail interrupting time critical searches.

Problem

After an upgrade to Enterprise Vault 12.4.0, Discovery Accelerator (DA) searches will cause index volumes to fail intermittently and interrupting searches. 

Error Message

The event log on the Enterprise Vault server that hosts the problematic index volume will show the following:

Type :        Warning
Event :        41315
Source :        Enterprise Vault 
Category :    Index Query Server
Description:
A search failed with error "The indexing engine could not load the index".

Archive: JournalArchive
Archive ID: 13A2D20306B17E6459DB440FDAC32EC291110000EV8alias-JournalArchive
Index volume ID: 303935
Maximum results: 10000
Timeout (seconds): 14400
Sort order: snum

For more information, see Help and Support Center at http://telemetry.community.veritas.com/entt?product=ev&language=english&version=12.3.0.0&build=12.3.0.1365&error=V-437-41315

A back trace of EVIndexQueryServer will show the following:

EV-M        {VelocitySearch} TraceTimer [212435ms] Velocity.CollectionBrokerSearch returned for collection 13A2D20306B17E6459DB440FDAC32EC29_303935.

EV-M        {SearchPipeline} TraceTimer [212436ms] [13/19] Completed EngineSearchCommand, search 1374, ID , user EV\vault, source JournalArchive (279144)

EV-L        {SourcesInUse} Remove: Source 13A2D20306B17E6459DB440FDAC32EC291110000EV8alias-JournalArchive/303935 is no longer in use. 4 total sources in use.

EV-H        {SearchTask} Exception: Missing result count in response.  Indexer-service process may have terminated prematurely.  Source status = queried. Parse status = processed, parsed, fetched Info:Search 1374 failed, Attempt=1 Diag: Type:Symantec.EnterpriseVault.Indexing.Search.SearchEngineIntermittentFailureException ST:   at Symantec.EnterpriseVault.Indexing.Search.VelocitySearch.CheckResultsForErrors(queryresults queryresults)|   at Symantec.EnterpriseVault.Indexing.Search.VelocitySearch.GetVelocityResultsFromCollectionBrokerResponse(Search search, String velocityResultsFileName, VelocityResults results, collectionbrokersearchresponse response)|   at Symantec.EnterpriseVault.Indexing.Search.VelocitySearch.GetVelocityResults(Search search, Source source)|   at Symantec.EnterpriseVault.Indexing.Search.VelocitySearch.DoSearch(Search search, Source source)|   at Symantec.EnterpriseVault.Indexing.Search.SearchBase.Search(Search search, Source source)|   at Symantec.EnterpriseVault.Indexing.Search.EngineSearchCommand.Execute(SearchPipelineContext context)|   at Symantec.EnterpriseVault.Indexing.Search.SearchPipeline.Execute(SearchPipelineContext context)|   at Symantec.EnterpriseVault.Indexing.Search.SearchTask.Execute() Inner:None

EV-L        {IndexVolumes} GetIndexVolumeInfo from directory: JournalArchive/13A2D20306B17E6459DB440FDAC32EC291110000EV8alias-JournalArchive/303935

EV-L        {SourceDirectory} TraceTimer [380ms] Mark source as failed

EV-H        {IndexQueryServerSearch} Exception: The indexing engine could not load the index Info:Exception occurred during the execution of the pipeline for source 13A2D20306B17E6459DB440FDAC32EC291110000EV8alias-JournalArchive/303935 Diag: Type:Symantec.EnterpriseVault.Indexing.Search.SearchEngineIndexInvalidException ST:   at Symantec.EnterpriseVault.Indexing.Search.VelocitySearch.CheckLogMessages(addedsource source)|   at Symantec.EnterpriseVault.Indexing.Search.VelocitySearch.CheckResultsForErrors(queryresults queryresults)|   at Symantec.EnterpriseVault.Indexing.Search.VelocitySearch.GetVelocityResultsFromCollectionBrokerResponse(Search search, String velocityResultsFileName, VelocityResults results, collectionbrokersearchresponse response)|   at Symantec.EnterpriseVault.Indexing.Search.VelocitySearch.GetVelocityResults(Search search, Source source)|   at Symantec.EnterpriseVault.Indexing.Search.VelocitySearch.DoSearch(Search search, Source source)|   at Symantec.EnterpriseVault.Indexing.Search.SearchBase.Search(Search search, Source source)|   at Symantec.EnterpriseVault.Indexing.Search.EngineSearchCommand.Execute(SearchPipelineContext context)|   at Symantec.EnterpriseVault.Indexing.Search.SearchPipeline.Execute(SearchPipelineContext context)|   at Symantec.EnterpriseVault.Indexing.Search.SearchTask.Execute() Inner:Symantec.EnterpriseVault.Indexing.Search.SearchExecutionFailedException: SEARCH_ENGINE_STARTING_INDEXER_SERVICE: Failed to start the indexer service for collection [13A2D20306B17E6459DB440FDAC32EC29_303935]: [<xmlnode name="error"><log><log><error time="0" date="1580999297" id="COLLECTION_SERVICE_TERMINATING">The collection-service couldn't process the request because it is currently in the process of terminating.  </error></log></log></xmlnode>]

EV-H        {SearchService} Exception: The indexing engine could not load the index Info:SearchService.Search failed. Diag: Type:Symantec.EnterpriseVault.Indexing.Search.SearchEngineIndexInvalidException ST:   at Symantec.EnterpriseVault.Indexing.Search.IndexQueryServerSearch.HandleErrors(SearchTask task)|   at Symantec.EnterpriseVault.Indexing.Search.IndexQueryServerSearch.Search(Search search, String callerLogin)|   at Symantec.EnterpriseVault.Indexing.Search.SearchService.Search(Search search) Inner:Symantec.EnterpriseVault.Indexing.Search.SearchExecutionFailedException: SEARCH_ENGINE_STARTING_INDEXER_SERVICE: Failed to start the indexer service for collection [13A2D20306B17E6459DB440FDAC32EC29_303935]: [<xmlnode name="error"><log><log><error time="0" date="1580999297" id="COLLECTION_SERVICE_TERMINATING">The collection-service couldn't process the request because it is currently in the process of terminating.  </error></log></log></xmlnode>]

EV~W        |Event ID: 41315 A search failed with error "The indexing engine could not load the index".|Archive: JournalArchiveArchive ID: 13A2D20306B17E6459DB440FDAC32EC291110000EV8alias-JournalArchive|Index volume ID: 303935|User: EV\vault​​​​​​​|Maximum results: 10000|Timeout (seconds): 14400|Sort order: snum

System-Reporting log shows the following which appears like a crash, but there are not associated events in the application or system logs:

The [live] [indexer] for collection [13A2D20306B17E6459DB440FDAC32EC29_303935] terminated unexpectedly: [collection [13A2D20306B17E6459DB440FDAC32EC29_303935]: exception C0000005 at address 00007FF9320E33D8 [Thu Feb 06 14:28:12 2020]]. 

The behavior will repeat itself when performing DA search on various archives and varying index volumes. The issue is not archive or volumes specific but instead related to the indexing engine. 

Cause

EV 12.4 introduced an updated version of the Indexing engine where this issue occurs. It cause any index volume to fail unexpectedly and the search completes however with errors. 

NOTE: This issue is not to be confused with the issue discussed in https://www.veritas.com/content/support/en_US/article.100045903. While the issue in 100045903 is also a result of the new indexing engine it is a separate issue with a separate resolution. 

WorkaAround:

As a quick workaround, it is possible to synchronize the affected volume and to bring it online and then resubmit the search.

Solution

 This issue has been fixed in the product in EV 12.5.2. 

Upgrade to 12.5.2 to permanently resolve this issue.