Veritas recommends pro-actively disabling related DMP paths and controllers when migrating to a new SAN switch using vxdmpadm

Description

When performing any intentional SAN related activity, Veritas always recommends pro-actively disabling all related Veritas Dynamic Multi-Pathing (DMP) paths and controllers being migrated to a new SAN switch using the vxdmpadm command.

This dramatically reduces any unforeseen issues being caused by events beneath DMP.
 

Pro-active Intervention
 

By pro-actively disabling the impacted paths/controller, DMP ensures any in-flight I/O is only targeted to the surviving ENABLED paths, preventing active I/O going down a path/controller which is to be migrated to a new SAN switch.
 

The paths can be disabled using the vxdmpadm CLI command syntax:

# vxdmpadm disable path=os-device-name

# vxdmpadm disable ctlr=controller-ID
 

Figure 1.0

Unless DMP receives the correct signal(s) from the lower layers upstream back to DMP, DMP then may fail to cater for all use cases & scenarios.

vxesd

The Veritas eventsource Daemon (aka vxesd) is a vital Veritas Volume Manager (VxVM) component process that receives notifications of SAN related events & provides additional intelligence surrounding events happening beneath DMP.

Where the DMP tunable dmp_monitor_fabric is disabled (off), DMP may not receive additional SAN related event messaging intelligence upstream to DMP and can restrict how DMP can react to more specific use cases.


AIX specific:
 

The DMP tunable dmp_monitor_fabric is turned off by default on AIX platforms. The dmp_monitor_fabric tunable is turned on by default across other platforms.

Veritas is working with IBM to find a solution to the performance issues related to the SNIA (Storage Network Industry Association) HBA APIs specially on AIX.


Example:

Traditionally, DMP has taken a reactive approach to failing paths by only disabling paths when active I/O have failed on the storage sub system.

By using the Storage Networking Industry Association (SNIA) HBA API library, Veritas vxesd is able to receive SAN fabric events via the HBA. This information allows DMP to take a pro-active role by checking suspect devices from the SAN events, even if there is no active I/O.

New I/O can be directed to healthy (ENABLED) paths whilst the suspect devices are verified.  
 

In the scenario that some LUNs or paths are disconnected from a SAN, the HBA will notify vxesd of the SAN event, specifying the PWWNs that are affected.

The vxesd daemon uses this event information and correlates it with the previous topology information to determine which set of device paths have been affected. The vxesd daemon will then send the affected set to vxconfigd (DDL) so that the device paths can be marked as suspect.
 

When the path is marked as suspect, DMP will not send new I/O to the path unless it is the last path to the device.

In the background, the DMP restore daemon will check the accessibility of the paths on its next periodic cycle (300 seconds by default) using a SCSI inquiry probe.

If the SCSI inquiry fails, DMP will disable the path the to affected LUNs, which will also be logged in the /etc/vx/dmpevents.log.

By default, the DMP restore daemon will use the check_disabled mode, which means that on its test cycle it will check disabled, suspect & idle paths.

If the LUNs are re-connected at a later time, the HBA will inform vxesd of the SAN event. When the DMP restore daemon runs its next test cycle, the disabled paths will be checked with the SCSI probe and re-enabled if successful.
 

NOTE: If a HBA LINK UP event is received by vxesd, the DMP restore daemon will be restarted and the SCSI probes will run immediately, without waiting for the next periodic cycle. Once restarted, the DMP restore daemon will start a new periodic cycle based upon when the daemon restarted. If the disabled paths are not accessible by the time of the first SCSI probe, they will be re-tested on the next cycle (300s by default).
 

DMP restore policy
 

DMP maintains a kernel task that re-examines the condition of paths at a specified interval. The type of analysis that is performed on the paths depends on the checking policy that is configured.

The default dmp_restore_policy is set to check_disabled and will check every 300 seconds.

Veritas DMP be tuned to reduce the time window to safely disable paths associated with a disabled switch port event whilst ensuring the in-flight I/O’s are processed correctly.

1. The path restoration thread analyzes all paths in the system and revives the paths that are back online, as well as disabling the paths that are inaccessible. The command to configure this policy is:
 

# vxdmpadm settune dmp_restore_policy=check_all


2. The dmp_restore_interval tunable parameter specifies how often the path restoration thread examines the paths
 

# vxdmpadm settune dmp_restore_interval=150

Note: Do not set the dmp_restore_interval value too low, as this can have a negative impact and add a performance overhead on the server. The more paths and LUNs presented, the more workload on DMP and other components.

DMP iopolicy of round-robin should be used over the default minimumq when the number of paths for a DMPNODE is 4 or more.