VCS 4 node Cluster experiencing chronic scsi "reservation conflict" warnings; vxdisk-list indicates VxVM vxdisk ERROR V-5-1-539 Device tagmastore-usp0_2113: get_contents failed

Problem

vxdisk list command line shows the error. VxVM vxdisk ERROR V-5-1-539 Device tagmastore-usp0_2113: get_contents failed:;  pegsdb25 scsi: [ID 107833 kern.warning.... (ssd205):     reservation conflict    i/o to invalid geometry

Error Message

Multiple nodes experiencing similar scsi errors:

Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): reservation conflict

Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): i/o to invalid geometry

Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): reservation conflict

Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): reservation conflict

Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): i/o to invalid geometry

 

Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211): reservation conflict

Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211): reservation conflict

Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211):

Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211):

Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211): reservation conflict

Cause

---When VCS managed VxVM disk group is vxdg-imported on one node,   VCS i/o fencing applies SCSI-3 PGR registration and write-exclusive reservation keys on the disk group data disks;  for example:

# vxfenadm -g /dev/rdsk/c1t50060E80047E0427d71s2

Reading SCSI Registration Keys...

Device Name: /dev/rdsk/c1t50060E80047E0427d71s2

Total Number Of Keys: 2

key[0]:

Key Value [Numeric Format]: 67,86,67,83,0,0,0,0

Key Value [Character Format]: CVCS

key[1]:

Key Value [Numeric Format]: 67,86,67,83,0,0,0,0

Key Value [Character Format]: CVCS

 

vxfenadm -r /dev/rdsk/c1t50060E80047E0427d71s2

Reading SCSI Reservation Information...

Device Name: /dev/rdsk/c1t50060E80047E0427d71s2

Total Number Of Keys: 1

Key[0]:

Reservation Type: SCSI3_RESV_WRITEEXCLUSIVEREGISTRANTSONLY

Key Value [Numeric Format]: 67,86,67,83,0,0,0,0

Key Value [Character Format]: CVCS

---- but other nodes on the cluster experience "reservation conflict" warnings such as................

Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): reservation conflict

Feb 29 15:13:10 nodedb25 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/SUNW,emlxs@0,1/fp@0,0/ssd@w50060e80047e0432,59 (ssd205): i/o to invalid geometry

Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211): reservation conflict

Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.warning] WARNING: /pci@0,600000/pci@0/pci@9/SUNW,emlxs@0/fp@0,0/ssd@w50060e80047e0401,24 (ssd211): reservation conflict

Mar 15 05:45:53 nodedb01 scsi: [ID 107833 kern.w

Solution

---- While array was placing reservation key SCSI3_RESV_WRITEEXCLUSIVEREGISTRANTSONLY the reservation key was behaving READEXCLUSIVEREGISTRANTSONLY instead. This prevented other hosts from even reading the LUNs imported to the other host.  Commands like prtvtoc and dd read would fail. 

----Discovered the Hitachi Tagmastore UPS600 host mode option settings were not complete.  HMO2 was not set.  Required settings to support SFHA for both DMP and VCS I/O Fencing are:

Host mode 09 =  Solaris hosts

Host mode option 02  =  VCS, CVM-CFS, SFRAC I/O fencing configured

Host mode option 07  =  Auto-recognize array LUN resize;

                                      *  requires host mode option 09  and  latest Solaris 10  HBA drivers

Host mode option 22   =  VCS cluster with common disk groups for failover or parallel LUN I/O access; (if AIX requires host mode 0F)


Applies To

----Cluster node SFHA  is in upgrade transition from 5.0MP3 to 5.1SP1RP1P1

----SF is being upgraded on nodes in phases while production service group migrated between upgraded nodes

----VCS will remain 5.0MP3 until SF is upgraded

----Solaris MPxIO is also being replaced by VxDMP

----also under VxVM 5.1SP1RP1P1 "dmp_native_support" is enabled to support ZFS LUNs

$ egrep 'SunOS' */uname_a ; cat */etc/release
SunOS nodedb01 5.10 Generic_144488-12 sun4u sparc SUNW,SPARC-Enterprise
                   Oracle Solaris 10 9/10 s10s_u9wos_14a SPARC
     Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
                            Assembled 11 August 2010
$ egrep 'vx|llt|gab|emcp' */modinfo
 29  1380800  50c10 317   1  vxdmp (VxVM 5.1SP1RP1P1 DMP Driver)
 31 7be00000 21a918 318   1  vxio (VxVM 5.1SP1RP1P1 I/O driver)
 33  13caa80   1190 319   1  vxspec (VxVM 5.1SP1RP1P1 control/status)
226 7bbf7308    d40 320   1  vxportal (VxFS 5.1_SP1RP1 portal driver)
227 7a600000 1fe098  21   1  vxfs (VxFS 5.1_SP1RP1 SunOS 5.10)
228 7ab18000  26470 323   1  llt (LLT 5.0MP3)
229 7ab38000  4a028 324   1  gab (GAB device 5.0MP3)
230 7ab6a000  51c10 325   1  vxfen (VRTS Fence 5.0MP3)
$

 

bash-3.00# uname -a
SunOS nodedb03 5.10 Generic_138888-06 sun4u sparc SUNW,SPARC-Enterprise

bash-3.00# modinfo | egrep 'vx|llt|gab|emcp'
 36  13428a8  3e4e0 317   1  vxdmp (VxVM 5.0MP3: DMP Driver)
 38 7be00000 209248 318   1  vxio (VxVM 5.0MP3 I/O driver)
 40 7bfe90f0    c78 319   1  vxspec (VxVM 5.0MP3 control/status driv)
220 7af953f0    cb0 320   1  vxportal (VxFS 5.0_REV-5.0MP3A25_sol port)
221 7aa00000 1d89e0  21   1  vxfs (VxFS 5.0_REV-5.0MP3A25_sol SunO)
222 7abc8000  26470 322   1  llt (LLT 5.0MP3)
223 7a600000  4a028 323   1  gab (GAB device 5.0MP3)
224 7a632000  51c10 324   1  vxfen (VRTS Fence 5.0MP3)


# uname -a
SunOS nodedb25 5.10 Generic_144488-12 sun4u sparc SUNW,SPARC-Enterprise

# modinfo | egrep 'vx|llt|gab|emcp'
 29  1380800  50c10 324   1  vxdmp (VxVM 5.1SP1RP1P1 DMP Driver)
 31 7be00000 21a918 325   1  vxio (VxVM 5.1SP1RP1P1 I/O driver)
 33  13caa80   1190 326   1  vxspec (VxVM 5.1SP1RP1P1 control/status)
227 7bbf5308    d40 327   1  vxportal (VxFS 5.1_SP1RP1 portal driver)
228 7a600000 1fe098  21   1  vxfs (VxFS 5.1_SP1RP1 SunOS 5.10)
229 7ab1e000  26470 329   1  llt (LLT 5.0MP3)
230 7ab3e000  4a028 330   1  gab (GAB device 5.0MP3)
231 7ab70000  51c10 331   1  vxfen (VRTS Fence 5.0MP3)
232 7aba2000  21ec0 332   1  vxglm (VxGLM 5.0MP3 (SunOS 5.10))
233 7ae1c000   5418 333   1  vxgms (VxGMS 5.0MP3 (SunOS))

bash-3.00# uname -a
SunOS nodedb24 5.10 Generic_138888-06 sun4u sparc SUNW,SPARC-Enterprise

bash-3.00# modinfo | egrep 'vx|llt|gab|emcp'
 36  13428a8  3fdc0 322   1  vxdmp (VxVM 5.0MP3RP4: DMP Driver)
 38 7be00000 211590 323   1  vxio (VxVM 5.0MP3RP4 I/O driver)
 40 7bff0428    c78 324   1  vxspec (VxVM 5.0MP3RP4 control/status d)
222 7ae9d180    cb0 325   1  vxportal (VxFS 5.0_REV-5.0MP3RP4g_sol por)
223 7aa00000 1dd7d0  21   1  vxfs (VxFS 5.0_REV-5.0MP3RP4g_sol Sun)
224 7abcc000  26470 327   1  llt (LLT 5.0MP3)
225 7a600000  4a028 328   1  gab (GAB device 5.0MP3)
226 7a632000  51c10 329   1  vxfen (VRTS Fence 5.0MP3)
 

Terms of use for this information are found in Legal Notices.

Search

Survey

Did this article answer your question or resolve your issue?

No
Yes

Did this article save you the trouble of contacting technical support?

No
Yes

How can we make this article more helpful?

Email Address (Optional)