Unable to register nodes to CPS server "CPS ERROR V-97-1400-399 node 3 not found" with CPS on Solaris 10 5.1SP1 and 5.1SP1RP2
Problem
Generally we face issues while configuring IO fencing using CPS as there is not much information on troubleshooting part.
Here is one of the reasons for it to fail and how to troubleshoot.
There are various phases (scripts getting executed ) in configuring CPS IOfencing.
e.g.
- local_info.sh
- generate_snapshot.sh
- join_local_node.sh
- unjoin_local_node.sh
Error Message
Output from file /var/VRTSvcs/log/vxfen/vxfend_A.log:
2012/04/26 17:41:05 VXFEN vxfend INFO V-11-2-5109 ----Begin: output from join_local_node.sh
Begin: join_local_node.sh
my node id: 3
my cluster: 3
Coord point 1 membership:
Attempting to register 3 with CP server 192.168.29.10
CPS ERROR V-97-1400-462 Register operation failed.
CPS ERROR V-97-1400-399 Node 3 not found. Please do 'add_node' before trying to register it.
Unable to register with [192.168.29.10]:14250, error no: 1
checking membership: and my_cluster: 3
GAB membership differs from [192.168.29.10]:14250 membership
Unable to register with coordination point [192.168.29.10]:14250
Unable to register with a majority of the coordination points
End: join_local_node.sh returning FAILURE (100)
Cause
This issue is generally caused due to inconsistent NODE IDs reported by "gabconfig -a" and "CPS server".
e.g.
bash-3.00# cat /etc/llthosts
3 punv240-d5e
4 punv240-d5f
GAB Port Memberships
===============================================================
Port a gen 3243c02 membership ; 34
Port h gen 3243c1c membership ; 34
While the node ids that are taken by CPS starts from 0 and hence different set of node IDs reported by both.
root@punv210-a3c:/opt/VRTScps/bin # cpsadm -s 192.168.29.10 -a list_nodes
ClusterName UUID Hostname(Node ID) Registered
=========== =================================== ================ ===========
cluster1 {bbaa2e02-1dd1-11b2-8cb4-00144f5a02de} punv240-d5e(0) 0
cluster1 {bbaa2e02-1dd1-11b2-8cb4-00144f5a02de} punv240-d5f(1) 0
CPS server by default start adding nodes starting from node ID 0, if it matches the same with /etc/llthosts it registers the node else configuration fails.
Solution
There are two ways to do it:
Process1:
Run the relative port shutdown commands on both application nodes ( punv240-d5e and punv240-d5f ) simultaneously.
Now unload gab and llt modules and after that make changes to /etc/llthosts on both nodes.
Then startup the relative ports on both nodes and check if port membership is reflecting changes in "gabconfig -a" command.
Once the changes are reflected, try to add nodes to CPS using the command : /opt/VRTS/install/installvcs -fencing
For port shutdown/startup commands refer to article TECH49819.
https://www.veritas.com/business/support/index?page=content&id=TECH49819
Process 2:
Setting up server-based I/O fencing manually :
https://sfdoccentral.Veritas.com/sf/5.1/linux/html/vcs_install/ch16s02.htm
Applies To
Solaris 10 SFHA5.1RP2