VCS ERROR V-16-1-10119 GabHandle::push returned = 12, gh_src = 0, gh_gen = 0, gh_size = XXX

Problem

The HAD (High Availability Daemon) stops when the the GAB (Global Atomic Broadcast) and LLT (Low Latency Transport) protocols cannot find enough contiguous memory.

Error Message

From the engine_A.log file:

VCS ERROR V-16-1-10119 GabHandle::push returned = 12, gh_src = 0, gh_gen = 0, gh_size = 65527

This example shows that GAB was trying to get 65KB of contiguous memory, but failed.

Cause

The system may not have enough contiguous memory. This can be confirmed with the following procedure.

1. Use the following command to enable memory info in sysrq.

# echo 'm' > /proc/sysrq-trigger

2. Use dmesg to check the memory status.

# dmesg
...
kernel: [11653135.296463] Node 0 Normal: 201112*4kB 11267*8kB 165*16kB 2*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 897288kB

Note: The excerpt from dmesg, above, shows there is no contiguous memory chunk bigger than 64K.

 Around 20 chunks of contiguous 128KB chunks is required to join the node (i.e. just around 2MB, but contiguous in every 128K). The actual number will depend on the main.cf.

After the node joins, neither VCS nor GAB/LLT will hold this memory and it will be released back to the OS.

Solution

Contact the operating system vendor for assistance with defragmenting memory, or reboot the system as a temporary solution.