CVM nodes fails to join the cluster, with protocol version mismatch messages.

Problem

After patching all nodes of a CVM cluster to exactly the same versions for all Veritas software, CVM would only start on one node at a time.

Error Message

These messages are displayed when booting the server;

VxVM V-0-0-0 Received message has a different protocol version
VxVM vxio V-0-0-0 (B) We should now stop trying to join the cluster, if we are part of the cluster just leave it
VxVM vxio V-5-0-1379 Node failed to join because of protocol version mismatch with the running cluster
 

In the VCS engine_A.log file following messages are displayed when trying to join the CVM cluster;

VCS ERROR V-16-20007-1001 (cvmclus1) CVMVolDg:share_dg:online:CVM cluster is not activated on this nodeVCS WARNING V-16-20007-1041 (cvmclus1) CVMVolDg:share_dg:online:cvmvoldg_online_setup failed

Cause

These outputs indicate the source of the problem;

On the server cvmclus1;#vxdctl supportSupport information:  vxconfigd_vrsn:   32  dg_minimum:       20  dg_maximum:       160  kernel:           32  protocol_minimum: 90  protocol_maximum: 100  protocol_current: 0  <<<<<<<<< This is protocol version does not match the other server in the cluster e.g

On the server cvmclus2# vxdctl supportSupport information:  vxconfigd_vrsn:   32  dg_minimum:       20  dg_maximum:       160  kernel:           32  protocol_minimum: 90  protocol_maximum: 100  protocol_current: 100Looking /etc/vx/volboot file from the server cvmclus1, we can see the difference in version.

volboot 3.1 0.2 90   <<<<<<<<<<< The protocol version must be the same on all Cluster nodes.hostid cvmclus1hostguid {e8c319ae-1dd1-11b2-90c6-0021280fc8a3}request_threads 0endThis the /etc/vx/volboot file from the server cvmclus2;

volboot 3.1 0.2 100  <<<<<<<<<<< The protocol version must be the same on all Cluster nodes.hostid cvmclus2hostguid {538270a0-1dd2-11b2-b31d-0021280f6fa6}request_threads 0end

Solution

On all nodes in the cluster run the following command to rebuild the /etc/vx/volboot files, during this procedure protocol version supported by VxVM is also updated.

# vxdctl init

Additional Information about above command:

     init      Reinitializes the volboot file with a new host ID,
               highest   supported   cluster   protocol  version,
               system-wide default disk group name, and boot disk
               group  name  (if  the root disk is under VxVM con-
               trol).

               If a hostid operand is specified, this  string  is
               used;  otherwise,  a default host ID is used.  The
               default host ID is normally taken as  the  network
               node  name  for the host (see uname(1M)).  On sys-
               tems  with  a  hardware-defined  system  ID,   the
               default   host  ID  might  be  derived  from  this
               hardware ID.