VCS agent in failed state

Problem

Unable to probe/online/offline VCS resource because the agent for the type of the resource is in failed state.

Error Message

"hastatus -sum" shows

-- AGENTS FAILED
-- Type            System
I  Application     node01 

Cause

"ps -ef | egrep ApplicationAgent" shows the agent process is running, but its age is older than VCS engine process 'had', this means the communication link between this agent process and the 'had' process is broken.

Solution

1. Stop VCS locally (this only stops the monitoring, not the services actually running):
# hastop -local -force

2. Kill all processes containing VRTSvcs or VRTSagents:
# kill -9 $(ps -ef | egrep 'VRTSvcs|VRTSagents' | awk '{print $2}')

3. Start VCS monitoring back up on the node:
# hastart

At this point the agent should be running and there should be no failed agents. If there are still agents failed, make sure the agents were actually killed. If the issue persists please contact Veritas support.