How to use the Veritas eventsource daemon (vxesd)

The Veritas eventsource Daemon (ESD aka vxesd) is a Veritas Volume Manager ( VxVM) component process, first introduced in 4.0 on Solaris. In 5.0 and later, vxesd was included across all Unix/Linux platforms. The information in this article is specific to 5.0MP3 but may also apply to previous releases.

In layman terms, vxesd is a 'helper' process that receives notifications of device related events which are used to take appropriate actions. The benefits of vxesd include:

 

 

Fabric monitoring & Pro-active Error Detection
Traditionally DMP has taken a reactive approach to failing paths by only disabling paths when active i/o have failed on the storage sub system. By using the Storage Networking Industry Association (SNIA) HBA API library, vxesd is able to receive SAN fabric events via the HBA. This information allows DMP to take a pro-active role by checking 'suspect' devices from the SAN events, even if there is no active i/o. New i/o can be directed to healthy paths whilst the suspect devices are verified.  

Technical background:
During startup, vxesd queries the HBA (via the SNIA library) to obtain the SAN topology, which allows it to determine the set of Port World Wide Names (PWWN) that correspond to each of the device paths that are visible to the o/s. After obtaining the topology, vxesd registers with the HBA for SAN event notification.

 

In the scenario that some LUNs are disconnected from a SAN, the HBA will notify vxesd of the SAN event, specifying the PWWNs that are affected. The vxesd daemon uses this event information and correlates it with the previous topology information to determine which set of device paths have been affected. The vxesd daemon will then send the affected set to vxconfigd (DDL) so that the device paths can be marked as suspect.
 

When the path is marked as suspect, DMP will not send new i/o to the path unless it is the last path to the device. In the background, the DMP restore daemon will check the accessibility of the paths on its next periodic cycle (300s by default) using a SCSI inquiry probe. If the SCSI inquiry fails, DMP will disable the path the to affected luns, which will also be logged in the event log. By default the DMP restore daemon will use check_disabled mode which means that on its test cycle it will check disabled, suspect & idle paths.
 

If the luns are re-connected at a later time, the HBA will inform vxesd of the SAN event. When the DMP restore daemon runs its next test cycle, the disabled paths will be checked with the SCSI probe and re-enabled if successful.
 

NOTE: If a HBA LINK UP event is received by vxesd, the DMP restore daemon will be restarted and the SCSI probes will run immediately, without waiting for the next periodic cycle. Once restarted, the DMP restore daemon will start a new periodic cycle based upon when the daemon restarted. If the disabled paths are not accessible by the time of the first SCSI probe, they will be re-tested on the next cycle (300s by default).
 

The fabric monitor functionality is enabled by default and can be disabled/enabled using the command below:

Enabled:

Changes to dmp_monitor_fabric are persistent across reboots.

 

DMP Event logging.
DMP notifies vxesd of major events, which allows vxesd to log the event in a log file (/etc/vx/dmpevents.log). These events include:

 

 

The logfile is located in /var/adm/vx/dmpevents.log but is symbolically linked to /etc/vx/dmpevents.log. When the file reaches 10,000 lines, the log is rotated: dmpevents.log is renamed dmpevents.log. X and a new dmpevents.log file is created.
 

The level of detail in the event log file can be changed via the tunable dmp_log_level (1-4):

# vxdmpadm settune dmp_log_level= X

 

For details on the various log levels, please consult vxdmpadm(1M).
 

 

Automated Device Discovery (Solaris only)

Traditionally, VxVM device discovery required manual invocation of commands i.e vxdisk scandisks, vxdctl enable. Vxesd automates the discovery process by interfacing with the Reconfiguration Coordination Manager (RCM) framework on Solaris.

Technical background:

The vxesd daemon registers the script es_devfs.pl with the Solaris syseventd daemon for device arrival events. In the event that cfgadm is invoked to attach a new device to the system, the syseventd daemon will execute the scripts that are registered for device arrival events, including es_devfs.pl. The es_devfs.pl script establishes a socket with vxesd and transfers the event parameter (physical path of device) to the daemon. The vxesd daemon in turn connects to the vxconfigd daemon to initiate DDL device discovery for the device that had arrived. The whole operation takes place asyncronsouly such that the cfgadm command returns after the event has been added to the syseventd queue. This feature was first introduced in VxVM 4.0.

In the event that a device is removed with cfgadm, a similar process exists which uses the es_rcm.pl script to disable the relevant DMP paths. The removal operation is synchronous such that the cfgadm command will wait until all the registered detach scripts have completed execution.

Note: On systems with EMC PowerPath, it is feasible that a slow PowerPath discovery process may lead to a device being automatically claimed and controlled by DMP control. In such scenarios, the vxesd daemon should be stopped prior to the addition of the disk and restarted after PowerPath has claimed control of the device.

iSCSI Topology Discovery
vxesd builds up a topology of iSCSI devices that are visible to the host. On Solaris the iSCSI management API (IMA) is used to build the topology. On other operating systems, the platform specific CLI is used to gather similar information.

Displaying SAN/iSCSI topology
The command below displays the hierarchical listing of fibre channel and iSCSI devices:
 

For more information, consult vxddladm(1M).

vxesd Startup/Shutdown
For generic control, the vxesd daemon can be started & stopped via the vxddladm utility:

 

By default vxesd is enabled to start at boot time by the VxVM start script/service ( vxvm-startup2). To disable vxesd from starting at boot, the command vxddladm start eventsource should be commented out of the start script.

 

To permanently disable 5.0 vxesd in the linux OS, comment out the startup sequence in the /etc/init.d/vxvm-recover startup script:
 

Change:

# Start the event source daemon
if [ -x /usr/sbin/vxddladm ]
then
/usr/sbin/vxddladm start eventsource
fi

to:

# Start the event source daemon
#if [ -x /usr/sbin/vxddladm ]
#then
# /usr/sbin/vxddladm start eventsource
#fi

 

Support & Debug information

In addition to generic control, the vxesd daemon can be started directly:

A running vxesd process can be recycled (stopped and started) via the undocumented -k  option:

A debug option (-d) also exists that will display debug information to standard output:

To save the debug information to a file, redirect the stderr & stout:

The operation of the HBA API can be checked with the following utility:
 

The operation of the IMA API can be checked with the following utility