Veritas Cluster Server (VCS) I/O Fencing parameters for racing (Solaris)

When communication between cluster nodes fails, causing the cluster to be divided into sub-clusters, these sub-clusters start a race to grab coordinator disks for data protection (VCS I/O Fencing). vxfen has a mechanism that enables cluster administrators to give larger sub-clusters better odds to win this race. This document describes the differences in implementation between VCS versions and their tunable parameters.

Note: While this mechanism can be used to give larger sub-clusters much better odds to win the race condition, it cannot be used to guarantee that the larger sub-cluster will always win.

1. How to give the odds

Prior to 4.1 MP2

If the number of nodes in a sub-cluster is less than the number of nodes leaving the original cluster, the sub-cluster repeats reading the coordinator disks to delay the start of the race. By default, the number of reads is calculated as cube of (the number of leaving nodes). For example, if a 5-node cluster is divided into a 3-node and a2-node cluster,  the 2-node sub-cluster repeats reading coordinator disks27 (= 3 cubed) times. A tunable parameter, max_read_coord_disk , can be used to change this value, as described later.

 

4.1 MP2 and 5.0 or later

If the number of nodes in a sub-cluster is less than the number of nodes leaving from the original cluster,the sub-cluster waits for a number of seconds before joining the race. This wait time is calculated as (the number of leaving nodes) x 5. For example, if a5-node cluster is divided into a 3-node and a 2-node cluster,  the 2-nodesub-cluster will wait for 15 (3 times 5) seconds. Tunable parameters min_delay and max_delay (4.1MP2), or vxfen_min_delay and vxfen_max_delay (5.0 or later), can be employed to change this wait time, as described later.

2. Tunable parameters

The following parameters can be specified in the file /kernel/drv/vxfen.conf. Use/tune these parameters only in situations where you often see a larger sub-cluster losing the vxfen race. Note that careful and ample testing is required to determine the most optimal values for a specific environment.

 

Prior to 4.1 MP2

max_read_coord_disk:The maximum number of times vxfen will loop reading coordinator disks. If the calculated repeat count exceeds this limit, this value will be used instead.

Default = 25

Min = 1

Max = 1000

 

4.1MP2

min_delay: The lower limit of the wait time in seconds. If the calculated wait time is below this limit, this value will be used instead.

Default = 1

Min = 1

Max = N/A

 

max_delay: The upper limit of the wait time in seconds. If the calculated wait time exceeds this limit, this value will be used instead.

Default: 600

Min = N/A

Max = 600

 

Limitations: The implementation of 4.1 MP2 is a subset 5.0 (or later) implementation and has some limitations. vxfen in 4.1 MP2 will "round down" the min_delay and max_delay values to a number that is a multiple of 5. For example, if the calculated delay time is 20and max_delay specified is 18, the wait time value chosen will be 18. However, vxfen will only the wait for 15seconds, ignoring the remaining 3 seconds. Therefore, to avoid confusion, it is recommended that only numbers that are a multiple of 5 be specified.

 

Note: With 4.1 MP2, the default and minimum values implemented for min_delay are a bit confusing, as they are both 1 and not a multiple of 5. Magic in the code prevents this value from being rounded down to 0, so specifying 1 here - or using the default value of 1 - is safe and will not be a problem.

 

VCS 5.0 or later allows more granularity,  so the delay can be specified in any number of seconds within the minimum and maximum boundaries.

 

5.0 or later

vxfen_min_delay:The lower limit of the wait time in seconds. If the calculated wait time is below this, this value will be used instead.

Default = 1

Min = 1

Max = 600

 

vxfen_max_delay:The upper limit of the wait time in seconds. If the calculated wait time exceeds this, this value will be used instead.

Default: 60

Min = the value of vxfen_min_delay

Max = 600