VCS CRITICAL V-16-1-10037 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing

Problem

After a node reboot, throwing following error, unable to start the VCS further.

2012/03/03 20:51:47 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:02 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:17 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:32 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:47 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:53:02 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:53:17 VCS CRITICAL V-16-1-10031 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing <<<< Problem, showing related to fencing.

Error Message

2012/03/03 20:51:47 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:02 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:17 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:32 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:47 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:53:02 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:53:17 VCS CRITICAL V-16-1-10031 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing <<<< Problem, showing related to fencing.

Cause

########## Node 1 /var/adm/messages file ###########

2012/03/03 20:46:24 VCS INFO V-16-1-10196 Cluster logger started
2012/03/03 20:46:24 VCS NOTICE V-16-1-11022 VCS engine (had) started
2012/03/03 20:46:24 VCS NOTICE V-16-1-11050 VCS engine version=5.0
2012/03/03 20:46:24 VCS NOTICE V-16-1-11051 VCS engine join version=5.0.30.2
2012/03/03 20:46:24 VCS NOTICE V-16-1-11052 VCS engine pstamp=Veritas-5.0MP3RP2-01/22/09-15:17:00
2012/03/03 20:46:24 VCS NOTICE V-16-1-10114 Opening GAB library
2012/03/03 20:46:24 VCS NOTICE V-16-1-10619 'HAD' starting on: Node0
2012/03/03 20:46:24 VCS INFO V-16-1-10125 GAB timeout set to 15000 ms
2012/03/03 20:46:38 VCS CRITICAL V-16-1-11306 Did not receive cluster membership, manual intervention may be needed for seeding
2012/03/03 20:51:47 VCS INFO V-16-1-10077 Received new cluster membership
2012/03/03 20:51:47 VCS NOTICE V-16-1-10112 System (Node0) - Membership: 0x3, DDNA: 0x0
2012/03/03 20:51:47 VCS NOTICE V-16-1-10322 System  (Node '1') changed state from UNKNOWN to INITING
2012/03/03 20:51:47 VCS NOTICE V-16-1-10086 System Node0 (Node '0') is in Regular Membership - Membership: 0x3
2012/03/03 20:51:47 VCS NOTICE V-16-1-10086 System  (Node '1') is in Regular Membership - Membership: 0x3
2012/03/03 20:51:47 VCS NOTICE V-16-1-10453 Node: 1 changed name from: '' to: 'Node1'
2012/03/03 20:51:47 VCS NOTICE V-16-1-10322 System Node1 (Node '1') changed state from INITING to CURRENT_DISCOVER_WAIT
2012/03/03 20:51:47 VCS NOTICE V-16-1-10322 System Node0 (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD
2012/03/03 20:51:47 VCS NOTICE V-16-1-10322 System Node1 (Node '1') changed state from CURRENT_DISCOVER_WAIT to CURRENT_PEER_WAIT  <<<< Node 1 waiting to join Node 0
2012/03/03 20:51:47 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:02 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:17 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:32 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:52:47 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:53:02 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...
2012/03/03 20:53:17 VCS CRITICAL V-16-1-10031 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing <<<< Problem, showing related to fencing.
2012/03/04 15:40:32 VCS NOTICE V-16-1-11022 VCS engine (had) started   <<<< Here the VCS was started again.
2012/03/04 15:40:32 VCS INFO V-16-1-10196 Cluster logger started
2012/03/04 15:40:32 VCS NOTICE V-16-1-11050 VCS engine version=5.0


The above log's show problem related to Fencing. Checked for the fencing specific log's,


Sat Mar  3 20:29:13 HKT 2012 vxfenconfig -U returned 1
Sat Mar  3 20:29:13 HKT 2012 vxfenconfig -U output is VXFEN vxfenconfig ERROR V-11-2-1071 Ioctl VXFEN_IOC_CLUSTSTAT failed!
Sat Mar  3 20:29:13 HKT 2012 output of vxfenadm is
Device Name: /dev/sdq
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[1]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[2]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
key[3]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
Device Name: /dev/sds
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[1]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[2]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
key[3]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
Device Name: /dev/sdr
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[1]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[2]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
key[3]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
Device Name: /dev/sdg
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[1]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[2]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
key[3]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
Device Name: /dev/sdi
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[1]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[2]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
key[3]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
Device Name: /dev/sdh
Total Number Of Keys: 4
key[0]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[1]:
        Key Value [Numeric Format]:  66,45,45,45,45,45,45,45
        Key Value [Character Format]: B-------
key[2]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
key[3]:
        Key Value [Numeric Format]:  65,45,45,45,45,45,45,45
        Key Value [Character Format]: A-------
Sat Mar  3 20:29:13 HKT 2012 vxfenconfig -U returned with 1. exiting
Sat Mar  3 20:41:19 HKT 2012 \nstarting in vxfen-startup
Sat Mar  3 20:41:19 HKT 2012 calling regular vxfenconfig
Sun Mar  4 16:22:45 HKT 2012 \nstarting in vxfen-startup
Sun Mar  4 16:22:45 HKT 2012 calling regular vxfenconfig

After going through the log's, there was a reservation key's present when the server was booting up, and hence Fencing was not coming up.

Solution

Clear these keys to start I/O fencing, using these commands:

# /opt/VRTSvcs/vxfen/bin/vxfenclearpre

# reboot

Note: Reboot all nodes in the cluster.


Applies To

SFHA 5.0MP3, Linux RHEL 5.

Terms of use for this information are found in Legal Notices.

Search

Survey

Did this article answer your question or resolve your issue?

No
Yes

Did this article save you the trouble of contacting technical support?

No
Yes

How can we make this article more helpful?

Email Address (Optional)