[ ISSUE ]
CVM went faulted then unable to join CVM
[ ERROR MESSAGES ]
Jun 9 20:47:24 Symc_linux1 Had: VCS ERROR V-16-1-1005 (Symc_linux1) CVMCluster:cvm_clus:monitor:node - state: out of cluster reason: user initiated stop
Jun 9 20:47:25 Symc_linux1 Had: VCS ERROR V-16-1-10303 Resource cvm_clus (Owner: unknown, Group: cvm) is FAULTED (timed out) on sys Symc_linux1
Jun 9 20:50:30 Symc_linux1 Had: VCS ERROR V-16-1-11009 Exiting: Another copy of VCS may be running
On a given system, there can be only one instance of VCS daemon HAD can be running.
If another instance is started, the later invoked process finds out that an instance is already running, hence it exits by giving above message.
Normally this is not an issue as one might have accidently run the command to start VCS.
If your HA operations are running fine, there is no concern here.
However if you find this message when you are deliberately starting VCS, and you find no response to ha commands, then one should check if there is any instance of HAD process running.
If HAD process is not present, then check if /var/VRTSvcs/lock/.hadtrylock2 is present (as leftover of any previous issue);
if it exists, delete it and start VCS again.
[ VERSION OF OS/PACKAGE ]
Linux Symc-llnux 2.6.18-194.17.4.el5 #1 SMP Wed Oct 20 13:03:08 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux