VRTScavf (CVM) 7.4.2.2201 agent enhanced on AIX to handle EMC SRDF VxVM vxdg ERROR V-5-1-19179 Disk group AIXSRDF: import failed: SCSI-3 PR operation failed failures
Problem
The following issue is specific to AIX environments only, when the "dmp_cache_open" tunable is enabled on AIX in relation to Veritas SCSI-3 disk based fencing.
# vxdmpadm gettune dmp_cache_open Tunable Current Value Default Value
------------------------------ ------------- -------------
dmp_cache_open on on
EMC hardware replicated SRDF devices may fail to import the VxVM managed disk group following a SRDF role switch.
The issue does not impact other hardware replicated solution such as Hitachi TrueCopy and other array vendor solutions. The issue is specific soley to EMC SRDF devices only,
The issue impacts both EMC PowerPath and Veritas DMP.
Error Message
# /usr/sbin/vxdg -s -o groupreserve=VCS -o clearreserve -cC -t import AIXSRDFVxVM vxdg ERROR V-5-1-19179 Disk group AIXSRDF: import failed:
SCSI-3 PR operation failed
Please refer to system log for details.
Cause
The issue normally occurs following a system restart of the servers hosting the EMC srdf-r2 devices.
If a SRDF role switch is performed to promote the existing srdf-r2 devices to srdf-r1 devices, the import will fail to import as the revised device state has not been updated correctly on AIX.
SRDF ROLE SWITCH
in this instance, the EMC device group is named "AIXSRDF":
# symrdf -g AIXSRDF failover -establish -nopr
An RDF 'Failover' operation execution is
in progress for device group 'AIXSRDF'. Please wait...
Write Disable device(s) on SA at source (R1)..............Done.
Suspend RDF link(s).......................................Done.
Swap RDF Personality......................................Started.
Swap RDF Personality......................................Done.
Suspend RDF link(s).......................................Done.
Resume RDF link(s)........................................Started.
Resume RDF link(s)........................................Done.
Read/Write Enable device(s) on SA at target (R2)..........Done.
The RDF 'Failover' operation successfully executed for
device group 'AIXSRDF'.
To overcome this AIX specific issue, the newly promoted EMC srdf-r2 devices must be closed and rescanned by VxVM.
# vxdisk rm <dmpnode-names>
# vxdisk scandisks
To address this via Veritas Cluster Server, Veritas has introduced a series of new VCS CVMVolDg related attributes (ScanDisks = 1 & DGOptions) and a new vxdg import option -o usereplicatedev=only in connection with Veritas DMP.
Sample /etc/VRTSvcs/conf/config/main.cf
group srdf_SG (
SystemList = { aixserver1-v3 = 0, aixserver2-v4 = 1 }
AutoFailOver = 0
Parallel = 1
ClusterList = { PR_clust = 0, DR_clust = 1 }
AutoStartList = { aixserver1-v3, aixserver2-v4 }
)
CFSMount childdir_cfs (
Critical = 0
MountPoint = "/parentdir/childdir"
BlockDevice = "/dev/vx/dsk/AIXSRDF/vol2"
MountOpt = rw
)
CFSMount parentdir_cfs (
Critical = 0
MountPoint = "/parentdir"
BlockDevice = "/dev/vx/dsk/AIXSRDF/vol1"
MountOpt = rw
)
CVMVolDg srdf_dg (
CVMDiskGroup = AIXSRDF
CVMVolume = { vol1, vol2 }
CVMActivation = sw
CVMDeportOnOffline = 1
ClearClone = 1
DGOptions = { "-t -o usereplicatedev=only" }
ScanDisks = 1
)
requires group cvm online local firm
childdir_cfs requires parentdir_cfs
childdir_cfs requires srdf_dg
parentdir_cfs requires srdf_dg
Solution
Veritas DMP requires the following Private hot-fixes to be installed for both packages, VRTSvxvm and VRTScavf.
VRTSvxvm (Veritas Volume Manager) 7.4.2 Hot Fix 2601
VRTScavf hot-fix 7.4.2.2102
The new DMP vxdg option "-o usereplicatedev=only" is now required for Hardware Replication solutions like EMC SRDF, Hitachi TrueCopy etc.
This new option ensures only Hardware Replicated devices are selected during the LUN selection process for the import operation.
CVMTypes.cf updated following the VRTScavf hot-fix install
Sample: /etc/VRTSvcs/conf/config/CVMTypes.cf
type CVMVolDg (
static keylist RegList = { CVMActivation, CVMVolume, NodeList }
static int ActionTimeout = 300
static keylist SupportedActions = { vxdctlenable }
static int OnlineRetryLimit = 2
static int OnlineTimeout = 400
static int OnlineWaitLimit = 10
static str ArgList[] = { CVMVolume, CVMActivation, CVMVolumeIoTest, CVMDGAction, CVMDeportOnOffline, State, CVMDeactivateOnOffline, ClearClone, NodeList, Sc
anDisks, DGOptions }
str CVMDGAction
keylist CVMVolume
str CVMActivation
keylist CVMVolumeIoTest
int CVMDeportOnOffline
int CVMDeactivateOnOffline
int ClearClone
temp int voldg_stat
keylist DGOptions
int ScanDisks
keylist NodeList
)
Improved logging done by CVMVolDg, tells a story of events now and errors seen during the import process
# tail -36 /var/VRTSvcs/log/CVMVolDg_A.log
2022/11/18 10:09:27 VCS INFO V-16-10011-1091 CVMVolDg:srdf_dg:online:Importing the diskgroup - AIXSRDF
2022/11/18 10:09:27 VCS INFO V-16-20007-1113 CVMVolDg:srdf_dg:online:ScanDisks is set to [1], disks associated with shared dg AIXSRDF will be scanned before importing
2022/11/18 10:09:28 VCS INFO V-16-20007-1440 CVMVolDg:srdf_dg:online:Disk Access (DA) Names associated with shared dg AIXSRDF are [emc1_0862 emc1_0863 emc1_0864 emc1_0865 emc1_0866 emc1_0867 emc1_0868 ]
2022/11/18 10:09:28 VCS INFO V-16-20007-1114 CVMVolDg:srdf_dg:online:Selective device scanning enabled, scanning OS device paths [hdisk219,hdisk157,hdisk42,hdisk101,hdisk220,hdisk158,hdisk43,hdisk102,hdisk221,hdisk159,hdisk103,hdisk44,hdisk222,hdisk160,hdisk45,hdisk104,hdisk161,hdisk223,hdisk46,hdisk105,hdisk162,hdisk224,hdisk47,hdisk106,hdisk225,hdisk163,hdisk48,hdisk107] prior to importing shared AIXSRDF
2022/11/18 10:09:29 VCS INFO V-16-20007-1113 CVMVolDg:srdf_dg:online:ClearClone set for shared dg AIXSRDF to be imported with vxdg [ -c] options to clear udid_mismatch and clone flags
2022/11/18 10:09:29 VCS NOTICE V-16-20007-1113 CVMVolDg:srdf_dg:online:Special DGOptions: [-t -o usereplicatedev=only] set for shared DiskGroup [AIXSRDF]
2022/11/18 10:09:29 VCS WARNING V-16-20007-1013 CVMVolDg:srdf_dg:online:Diskgroups will be imported with reservations
2022/11/18 10:09:29 VCS INFO V-16-20007-1113 CVMVolDg:srdf_dg:online:Attempting shared import for DiskGroup [AIXSRDF] with vxdg [ -c -t -o usereplicatedev=only -o groupreserve="VCS" -s] options
2022/11/18 10:09:30 VCS WARNING V-16-20007-1001 CVMVolDg:srdf_dg:online:vxdg import failed for dg AIXSRDF - error - VxVM vxdg ERROR V-5-1-19179 Disk group AIXSRDF: import failed:
Disk in use by another cluster
Please refer to system log for details.
2022/11/18 10:09:30 VCS INFO V-16-20007-1001 CVMVolDg:srdf_dg:online:Trying again with options [ -c -t -o usereplicatedev=only -o groupreserve="VCS" -s -C] to clear hostid.
2022/11/18 10:09:32 VCS INFO V-16-20007-1001 CVMVolDg:srdf_dg:online:VxVM vxdg ERROR V-5-1-19179 Disk group AIXSRDF: import failed:
SCSI-3 PR operation failed
Please refer to system log for details.
2022/11/18 10:09:32 VCS INFO V-16-20007-1001 CVMVolDg:srdf_dg:online:Disk group AIXSRDF failed with: VxVM vxdg ERROR V-5-1-19179 Disk group AIXSRDF: import failed:
SCSI-3 PR operation failed
Please refer to system log for details.
2022/11/18 10:09:32 VCS INFO V-16-20007-1113 CVMVolDg:srdf_dg:online:ScanDisks is set to [1], running clearimport for disks associated with shared dg AIXSRDF prior to running vxdctl enable
2022/11/18 10:09:32 VCS WARNING V-16-20007-1111 CVMVolDg:srdf_dg:online:Running clearimport against disk(s): [emc1_0862 emc1_0863 emc1_0864 emc1_0865 emc1_0866 emc1_0867 emc1_0868 ]
2022/11/18 10:09:33 VCS INFO V-16-20007-1113 CVMVolDg:srdf_dg:online:ScanDisks is set to [1], removing disks associated with shared dg AIXSRDF prior to running vxdctl enable
2022/11/18 10:09:33 VCS WARNING V-16-20007-1111 CVMVolDg:srdf_dg:online:Removing disk(s): [emc1_0862 emc1_0863 emc1_0864 emc1_0865 emc1_0866 emc1_0867 emc1_0868 ]
2022/11/18 10:09:34 VCS WARNING V-16-20007-1002 CVMVolDg:srdf_dg:online:Shared Disk group AIXSRDF import using [ -c -t -o usereplicatedev=only -o groupreserve="VCS" -s -C] failed. Will try running vxdctl enable.
2022/11/18 10:09:35 VCS INFO V-16-20007-1113 CVMVolDg:srdf_dg:online:/usr/sbin/vxdctl enable performed
2022/11/18 10:09:35 VCS INFO V-16-20007-1113 CVMVolDg:srdf_dg:online:Attempting shared Disk group AIXSRDF import again following vxdctl enable using [ -c -t -o usereplicatedev=only -o groupreserve="VCS" -s -C -o noreonline].
2022/11/18 10:09:38 VCS NOTICE V-16-20007-1005 CVMVolDg:srdf_dg:online:Shared vxdg import (after vxdctl enable) succeeded for Disk Group AIXSRDF using [ -c -t -o usereplicatedev=only -o groupreserve="VCS" -s -C -o noreonline]
2022/11/18 10:09:38 VCS INFO V-16-20007-1009 CVMVolDg:srdf_dg:online:AIXSRDF enabled,shared,cds 1662552090.2110.claix04-v1
2022/11/18 10:09:41 VCS INFO V-16-10011-1046 CVMVolDg:srdf_dg:online:resource srdf_dg is online
VCS will attempt to run "vxdisk rm" against the related DMPNODE names associated with the VxVM disk group being imported.
# vxdisk -eo alldgs list | grep AIXSRDFemc1_0c8e auto:cdsdisk emc1_0862 AIXSRDF online shared hdisk56 srdf-r1
emc1_0c8f auto:cdsdisk emc1_0863 AIXSRDF online shared hdisk92 srdf-r1
emc1_0c90 auto:cdsdisk emc1_0864 AIXSRDF online shared hdisk93 srdf-r1
emc1_0c91 auto:cdsdisk emc1_0865 AIXSRDF online shared hdisk59 srdf-r1
emc1_0c92 auto:cdsdisk emc1_0866 AIXSRDF online shared hdisk95 srdf-r1
emc1_0c93 auto:cdsdisk emc1_0867 AIXSRDF online shared hdisk61 srdf-r1
emc1_0c94 auto:cdsdisk emc1_0868 AIXSRDF online shared hdisk97 srdf-r1
Note: The Veritas private hot-fixes have only been tested with Veritas DMP. We do not have a supported solution for EMC PowerPath (PP) impacting EMC storage arrays with Veritas fencing on AIX only.
Linux and Solaris platforms are not impacted by the AIX specific issue with EMC storage.
The new vxdg "-o usereplicatedev=only" option will only work with Veritas DMP as the multi-pathing driver, due to the extra value-add DMP extended attributes exposed.
Related Knowledge Base Articles
Was this content helpful?
Translated Content
Please note that this document is a translation from English, and may have been machine-translated. It is possible that updates have been made to the original version after this document was translated and published. Veritas does not guarantee the accuracy regarding the completeness of the translation. You may also refer to the English Version of this knowledge base article for up-to-date information.