"java.lang.OutOfMemoryError: Java heap space" exception is thrown by Data Insight

Problem

DataInsight services throw a java exception when heap memory exceeded or when Garbage Collection (GC) overhead limit is reached.

java.lang.OutOfMemoryError: GC overhead limit exceeded

java.lang.OutOfMemoryError: Java heap space

This article provides the steps for increasing the heap memory for Data Insight services using two exceptions that were recorded in the DataInsightWatchdog service.


 

Error Message

In this example, the exception is thrown by the DataInsightWatchdog service during the  "UpdateConfigJob".

2015-07-05 02:21:20 SEVERE:  #{14.EN_US} [ErrorLogger.schedulerError] Job (matrix.UpdateConfigJob threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: GC overhead limit exceeded]
                at org.quartz.core.JobRunShell.run(JobRunShell.java:224)
                at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

The next error may be seen in conjunction with a GC overhead limit reached, which means that heap space is severely limited, or it may be thrown as part of regular system execution.

2015-07-05 02:22:24 SEVERE:  #{13.EN_US} [JobRunShell.run] Job matrix.UpdateConfigJob threw an unhandled Exception: 
java.lang.OutOfMemoryError: Java heap space
2015-07-05 02:22:24 SEVERE:  #{13.EN_US} [ErrorLogger.schedulerError] Job (matrix.UpdateConfigJob threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
                at org.quartz.core.JobRunShell.run(JobRunShell.java:224)
                at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: java.lang.OutOfMemoryError: Java heap space

Cause

Java has a Garbage Collection process to free memory to enable it to be reused. The log extract for java process showing "GC overhead limit exceeded" indicates that the Java "Garbage Collection" process has run but only a small amount of memory has been freed.  This means that the small amount of heap memory freed will be quickly used, and the GC process will need to run again, and potentially the CPU would then become fully utilized in GC and not allow any work to be done.  This exception is indication the application has used all available memory and has been unable to free it.

Solution

Initial troubleshooting is to increase the amount of heap memory available to the application.  It may also be appropriate to increase the memory of the server should the increase in java heap memory will utilize significant proportion of the available server memory.

Follow these steps to increase the heap memory for Data Insight services.  These steps follow on for the example output above, where UpdateConfigJob throws the exception, the service to modify would be DataInsightWatchdog.exe:

Using the above error extract as example on where to adjust java heap space, the errors are reported in the watchdog0.0.log.  This log is related to the DataInsightWatchdog service.

1. From Windows Services (services.msc) open the properties for the service that is throwing the "OutOfMemory" exception. Notice the "Path to executable." This contains the path to the Data Insight binaries.
2. Go to the folder that contains the Data Insight binaries. This is the path found in the previous step. By default, this path is: C:\Program Files\veritas\DataInsight\bin.
    
3. Find the *.vmoptions file that is named after the service that is throwing the exception. Here it is the "DataInsightWatchdog" service is throwing the exception, and so the file name would be "DataInsightWatchdog.vmoptions", not the "Watchdog.vmoptions" file.
    
4. Open the file using a text editor.
    
5. Notice the "–Xmx" option. This can be modified to increase the amount of memory that is allocated to this service. See Figure 1 for details on this options.
    
6. Restart the service.
    

Figure 1 - Details for the Xmxsize option

-Xmxsize Specifies the maximum size (in bytes) of the memory allocation pool in bytes. This value must be a multiple of 1024 and greater than 2 MB. Append the letter k or K to indicate kilobytes, m or M to indicate megabytes, g or G to indicate gigabytes. The default value is chosen at runtime based on system configuration. For server deployments, -Xms and -Xmx are often set to the same value. See the section "Ergonomics" in Java SE HotSpot Virtual Machine Garbage Collection Tuning Guide at https://docs.oracle.com/javase/8/docs/articles/guides/vm/gctuning/index.html. The following examples show how to set the maximum allowed size of allocated memory to 80 MB using various units: -Xmx83886080 -Xmx81920k -Xmx80m The -Xmx option is equivalent to -XX:MaxHeapSize.