A Start job is running for dev-vx_dsk_bootdg_rootvol.device

Problem

After the root disk is encapsulated, the message "A Start job is running for dev-vx_dsk_bootdg_rootvol.device" appears during the reboot process after a system panic. The system may then get stuck indefinitely trying to save the kernel core dump.   Problem happens on Red Hat Enterprise Linux platform.


 

Error Message

After the system panicked, the reboot process is stuck, the message below is logged on console:

Cause

This issue occurs if the following conditions are true:

•           Encapsulation is performed AFTER kdump is configured
•           And then kdump initrd is recreated afresh AFTER the above encapsulation

 
Encapsulation behavior
Once the encapsulation is completed – when the entry in grub.cfg is created  for the menu to be shown on the grub-menulist, the root device is mentioned as vx_dsk_bootdg_rootvol. Whenever you boot from this entry of the grub – there are various files which have contents depending on where it has booted from. The file /proc/cmdline can be checked to determine the boot options and device of the kernel .

kdump service behavior     
Kdump service is a systemd service, which internally calls the /usr/bin/kdumpctl script. This script controls the kdump. In this script, there is a function which checked whether the kdump-initramfs should be recreated or not. This is decided on the basis of the modification time of a few files which kdump-initramfs is dependent on.

Two of the dependent files are:

• /etc/kdump.conf
• /boot/initrd-$(uname -r)kdump.img

Notes:

• When any of these files are changed, the kdump initrd is recreated.
• When kdump-initrd is recreated, the prepare_cmdline() function reads the /proc/cmdline file.
• After the encapsulation is performed, /proc/cmdline has vx_dsk_bootdg_rootvol set. 

Hence, kdump initrd is prepared with the boot-device (vx_dsk_bootdg_rootvol) rather than the root-device. When the kdump initrd is loaded to save the kernel core of the panic, the device it searches is vx_dsk_bootdg_rootvol which doesn't exist.
 

Solution

After Root Disk Encapsulation is enabled successfully through vxdiskadm (option 2) or the vxencap command, follow the below steps to have the kernel core saved in the /var/crash/ directory successfully. 

After the Root Disk Encapsulation Process and the subsequent system reboot, perform the following.

1. Check the kdump service status :
# systemctl status kdump

2.a. If not "ACTIVE," go to Step 5

2.b. If "ACTIVE," then, open the /etc/sysconfig/kdump file.

3.a. Search the KDUMP_COMMANDLINE= parameter in the kdump file. Determine if this option is set to one of the following:

• "NOTHING" 
• root=/dev/vx_dsk_bootdg_rootvol

3.b. If the option is set to one of the values shown in Step 3.a, set the parameter to:
​KDUMP_COMMANDLINE="root=UUID=<UUID_of_BOOT_DEVICE>  ro"

Where

UUID_of_BOOT_DEVICE = blkid <boot-device>

Example: 

# blkid /dev/sda2
/dev/sda2: UUID="79e2b9c7-c011-4ab6-85a9-f204efc0675e" TYPE="ext3"

KDUMP_COMMANDLINE="root=UUID=79e2b9c7-c011-4ab6-85a9-f204efc0675e ro"

4. Restart the kdump service:
# systemctl restart kdump

5. Done.

When system panic occurs, the kernel core will then be saved into the /var/crash/ directory successfully.

Etrack : 3912158