Unable to register nodes to CPS server "CPS ERROR V-97-1400-399 node 3 not found" with CPS on Solaris 10 5.1SP1 and 5.1SP1RP2

Problem

Generally we face issues while configuring IO fencing using CPS as there is not much information on troubleshooting part.

Here is one of the reasons for it to fail and how to troubleshoot.

There are various phases (scripts getting executed ) in configuring CPS IOfencing.

e.g.

• local_info.sh
• generate_snapshot.sh
• join_local_node.sh
• unjoin_local_node.sh

 

Error Message

Output from file /var/VRTSvcs/log/vxfen/vxfend_A.log:

2012/04/26 17:41:05 VXFEN vxfend INFO V-11-2-5109  ----Begin: output from join_local_node.sh
Begin: join_local_node.sh
         my node id: 3
         my cluster: 3
Coord point 1 membership:
Attempting to register 3 with CP server 192.168.29.10
CPS ERROR V-97-1400-462 Register operation failed.
CPS ERROR V-97-1400-399 Node 3 not found. Please do 'add_node' before trying to register it.
Unable to register with [192.168.29.10]:14250, error no: 1
checking membership:  and my_cluster: 3
GAB membership differs from [192.168.29.10]:14250 membership
Unable to register with coordination point [192.168.29.10]:14250
Unable to register with a majority of the coordination points
End: join_local_node.sh returning FAILURE (100)
 

Cause

This issue is generally caused due to inconsistent NODE IDs reported by "gabconfig -a" and "CPS server".

e.g.

bash-3.00# cat /etc/llthosts
3 punv240-d5e
4 punv240-d5f

GAB Port Memberships
===============================================================
Port a gen  3243c02 membership ;  34
Port h gen  3243c1c membership ;  34

 While the node ids that are taken by CPS starts from 0 and hence different set of node IDs reported by both.

root@punv210-a3c:/opt/VRTScps/bin # cpsadm -s 192.168.29.10  -a list_nodes

ClusterName      UUID                               Hostname(Node ID) Registered
===========   ===================================   ================  ===========
cluster1      {bbaa2e02-1dd1-11b2-8cb4-00144f5a02de}  punv240-d5e(0)    0
cluster1      {bbaa2e02-1dd1-11b2-8cb4-00144f5a02de}  punv240-d5f(1)    0

CPS server by default start adding nodes starting from node ID 0, if it matches the same with /etc/llthosts it registers the node else configuration fails.

 

Solution

There are two ways to do it:

Process1:

Run the relative port shutdown commands on both application nodes ( punv240-d5e and punv240-d5f ) simultaneously.

Now unload gab and llt modules and after that make changes to /etc/llthosts on both nodes.

Then startup the relative ports on both nodes and check if port membership is reflecting changes in "gabconfig -a" command.

 

Once the changes are reflected, try to add nodes to CPS using the command : /opt/VRTS/install/installvcs -fencing

For port shutdown/startup commands refer to article TECH49819.

https://www.veritas.com/business/support/index?page=content&id=TECH49819

 Process 2:

Setting up server-based I/O fencing manually :

https://sfdoccentral.Veritas.com/sf/5.1/linux/html/vcs_install/ch16s02.htm

Applies To

Solaris 10 SFHA5.1RP2