Java.exe terminates after certain amount of time and leaves EV Elasticsearch Indexing Engine unreachable.

Problem

In an Enterprise Vault (EV) 14.2 or later environment, the Java.exe process which is responsible for hosting the EV Elasticsearch Indexing Engine terminates, which results in the EV Elasticsearch Indexing Engine being unreachable.

Error Message

The DTrace at the time when Java.exe terminates show below errors:

Exception: The remote server returned an error: (429) Too Many Requests. Call: Status code 429 

ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [<http_request>] would be [8034955088/7.4gb], which is larger than the limit of [7888427417/7.3gb], real usage: [8034955088/7.4gb], new bytes reserved: [0/0b], usages [request=72531968/69.1mb, fielddata=0/0b, in_flight_requests=1365594296/1.2gb, model_inference=0/0b, eql_sequence=0/0b, accounting=35528162/33.8mb]

Event ID 41309 will be logged continuously, as EV Index Admin Service will fail to communicate with Elasticsearch indexing engine.

Event ID:      41309
Task Category: Index Admin Service 
Level:         Error
Description:
The following error message occurred in the core indexing engine: Failed to communicate with Elasticsearch indexing engine. Try restarting the Indexing Service to fix the issue. 

Cause

This issue occurs when EV Elasticsearch JVM Heap usage exceeds the predefined limit. By default, it is set to 8 GB. 

Solution

Increasing the JVM Heap size can help. If it is less than 8 GB, it would be recommended to set it to at least 8 GB. The maximum it can be set to is 50% of memory, with a maximum limit at 30720 MB.

NOTE: Post making these changes, the Enterprise Vault Indexing Service needs to be restarted.

Please refer to the document below for more information:

Enterprise Vault™ Indexing Best Practices: 14.2