Running Searches after upgrading to Enterprise Vault 12.4 causes multiple Index Volume failures

Problem

After upgrading to Enterprise Vault (EV) 12.4, running Searches in EV Search (EVS) or Discovery Accelerator (DA) can cause random Index Volumes to fail. Error Event ID 41315 is logged in the Event Logs with the description below. In addition, a review of the Indexing system-reporting log for that day, usually found under <Install_Path>\Enterprise Vault\EVIndexing\data\indexmetadata\reporting\system-reporting, will display an error referencing a missing column, as listed below.

Note - In order to review this log, it must be viewed in a SQLite browser. Veritas does not recommend any specific SQLite browser; however, there are numerous browsers that may be found online that can be used.

Error Message

- In the Enterprise Vault Event Log on the EV Indexing Server:

Event ID 41315
Category : Index Query Server
Description: A search failed with error "The indexing engine could not load the index".
Archive: JournalArchive01
Archive ID: 1A23BC456DEF123456789ABCDEFGHI12A1110000EVSERVER1
Index volume ID: X
User: evlab\vsa
Maximum results: 10000
Timeout (seconds): 3600
Sort order: snum

- In the error table of the system-reporting logs:

SQL error code [1]: no such column: frozen

v-437-41315

Cause

EV 12.4 introduced an updated version of the Indexing engine. An EV upgrade processes each Index to also update its internal schema to match the Indexing engine's schema. This schema update is performed during the first startup of the Indexing service after the EV 12.4 upgrade. An informational event 41467 is logged advising the successful schema update of all Indexes, however, the missing column is still not present.

Performing an Index Volume Verify | Basic is sufficient to return the Index Volume to a healthy status and another search can be attempted.  If issue persists a rebuild may be required. Rebuilding the affected Index Volumes creates the proper schema.

Since the random Index Volume failures have only been observed during multiple simultaneous searches using a high number of keywords in EVS or DA, searches may be resubmitted one at a time until they complete successfully.

Note: This issue is not to be confused with the one described in https://www.veritas.com/content/support/en_US/article.100048271.html. They symptoms and behavior is very similar, but it has a completely different resolution. 

Solution

Affected versions of EV: 12.4.0, 12.4.1, 12.4.2, 12.5.0. 

This issue has been resolved with the release of Enterprise Vault 12.5.1 available from Downloads. 

If issue persists after upgrading to 12.5.1 please contact Support for assistance.