Sign In
Problem
When using Storage Foundation Cluster Volume Manager (SFRAC/SFCFS) with shared disk groups of version 120 or higher, disk groups contain an attribute called dgfailpolicy. This attribute determines how the node should react if it loses access to disk in the corresponding disk group. If shared disk groups are set to the default dgfailpolicy of dgdisable a cluster wide panic could ensue and/or the database can halt clusterwide, should the Cluster Volume Manager (CVM) master lose connectivity to storage. To avoid this behaviour dgfailpolicy should be set to leave for shared disk groups.
Error Message
An example vxdg list <diskgroup> on a shared disk group with DG version 150 is shown below:
Group: ocrvotedg
dgid: 1285787736.67.host1.test.temp.com
import-id: 33792.75
flags: shared
version: 150
alignment: 8192 (bytes)
local-activation: shared-write
cluster-actv-modes: host2=sw host1=sw host3=sw host43=sw host5=sw
ssb: on
autotagging: on
detach-policy: global
dg-fail-policy: dgdisable <<< =================== Currently set to default i.e. dgdisable
copies: nconfig=default nlog=default
config: seqno=0.1422 permlen=0 free=0 templen=0 loglen=0
io-fencing: enabled
Cause
In a CVM RAC environment where a shared disk group is using a dgfailpolicy of dgdisable, should the master lose connectivity to all disks in the disk group, the master will disable the disk group (dgdisable). As this is a CVM environment the disk group is also disabled across all slave nodes (as all nodes must have a consistent view of the configuration as seen by the master).
Once a disk group is dgdisabled any new opens against volumes in that disk group will fail. Some examples of when opens are attempted are:
- When a volume containing a file system is mounted
- When an I/O is attempted against a raw volume device
This scenario can have potentially severe implications. For example if using Oracle RAC with vote devices on raw volumes, as soon as the corresponding disk group is dgdisabled cluster wide, all nodes will be unable to perform I/O to vote disks meaning that they can no longer heartbeat. As a result of this all nodes will be panic'd by Oracle Cluster Ready Services (CRS) causing a cluster wide loss of service.
Solution
To avoid this issue all shared disk groups of version 120 and higher should be set to use a dgfailpolicy of leave. Once set, should the master lose connectivity to disks in the disk group, it will panic and leave the cluster rather than disabling the disk group cluster wide. This then allows one of the surviving slave nodes to take over the master role and assuming that the new master has not issues with connectivity to storage allows the surviving members of the cluster to continue to function as normal.
vxdg -g <diskgroup> set dgfailpolicy=leave
This policy is consistent through reboots.
=================
In SFCFS 6.0 later, the dg fail policy is obsolete. From SFCFS 6.0 release notes:
Availability of shared disk group configuration copies
If the Cluster Volume Manager (CVM) master node loses access to a configuration
copy, CVM redirects the read or write requests over the network to another node
that has connectivity to the configuration copy. This behavior ensures that the
disk group stays available.
In previous releases, CVM handled disconnectivity according to the disk group
failure policy (dgfail_policy). This behavior still applies if the disk group version
is less than 170. The dgfail_policy is not applicable to disk groups with a version
of 170 or later.
Applies To
Storage Foundation for Oracle RAC (SFRAC) or Storage Foundation for Cluster File System (SFCFS)
All supported Unix versions
Terms of use for this information are found in Legal Notices.
