Cluster Server (VCS) Agents fail to start with ASSERTION FAILED message

Problem

During VCS startup all agents fail and dump core with below assertion failed message.

Error Message

In System messages
AgentFramework[25641]: [ID 702911 daemon.alert] ASSERTION FAILED: file VCSAgXRM.C, line 58, expression (_handle = (VCSHINSTANCE) VCSDlopen(libname))

In VCS engine_A.log
VCS NOTICE V-16-1-10024 Agent HostMonitor has not sent startup message since Thu Jan 23 20:44:01 2014
.. Restarting the agent
VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/HostMonitor for resource type HostMonitor successfully started at Thu Jan 23 20:45:07 2014

Core stack
> ::stack
libvcsagfw.so.2`__1cJVCSAssert6Fpc0Ipkc_v_+0xc0(fe597342, fe59736e, 3a, 0, 0, 0)
libvcsagfw.so.2`__1cIVCSAgXRM2t5B6M_v_+0x107(80a5728, 0)
libvcsagfw.so.2`__1cIVCSAgXRMIinstance6F_p0_+0x52(feffe22c, fe618fcc, fe5e3b9c,feffba54, fe4a2312, 8093870)
libvcsagfw.so.2`__1cKvcsag_init6Fpkcp0nMulog_dbg_lev__v_+0x1a7(feffe2c3, 0, 2d)
libvcsagfw.so.2`VCSAgMain+0x977(5, feffe108)
libvcsagfw.so.2`main+0x2e(5, feffe108, feffe120, feffe0fc)
_start+0x67(5, feffe29c, feffe2bd, feffe2c3, feffe2cf, feffe2d6)

Cause

Problems accessing the /opt/VRTSvcs/lib/libvcsxrm.so results in agents failing.

The problem could be due to missing/corrupted /opt/VRTSvcs/lib/libvcsxrm.so or could be related to any of the parent directory access.

Solaris truss command on hastart can be used to confirm if the issue is due to missing/corrupted /opt/VRTSvcs/lib/libvcsxrm.so or is related to any of the parent directory access.

# truss -failed -o $OUTFILE  hastart 

Solution

-  Restore the the missing/corruption /opt/VRTSvcs/lib/libvcsxrm.so file.

Or reinstall the VCS package (recommended)

- If the issue is due to parent directory access then set the correct directory permission.

In this specific case, customer had symbolic link created to /opt/VRTSvcs/bin for some legacy customer agents, but link for /opt/VRTSvcs/lib was missing.

Applies To

Solaris systems with VCS 5.x / 6.x