NetBackup Appliance Veritas VxVM and SystemD integration

Problem

In NetBackup appliance 3.x a new startup solution was introduced.
Previously SysV scripts started sequentially, where SystemD starts services in parallel.

This highlighted a timing issue for services that are dependent on a mount point that belongs to a VxVM device.

Error Message

Mount Error:

nbuapp_sf: - [Error] Failed to mount the 'MSDP' partition '0'.

vxvm-fstab-automount: VxVM: Auto-mount of /dev/vx/dsk/nbuapp/pdvol on /msdp/data/dp1/pdvol failed!

Umount Error:

hostname umount: UX:vxfs umount.vxfs: ERROR: V-3-26299: cannot umount /msdp/data/dp1/pdvol: Device or resource busy

Cause

1. Udev integration is needed  to register the /dev/vx/dsk/<dg>/<volume>  back to systemd
2. VxVM requires the  udev daemon to re-read the udev rules after VxVM has fully started up
3. There is a requirement to create service dependencies by adding a order-after-vxvm.conf entry into /etc/systemd/system/service.service.d
4. Modify the  remote-fs-pre.target so that it has a dependency on vxvm-boot.service
5. SystemD default timeout of 90 seconds is insufficient time to allow for veritas services to startup and the devices to be made available via UDEV
6. VEKI startup script does not include a LSB header, newer versions of systemd set the dependency order for the veki startup script to start after vxvm
7. VxVM startup script has a dependency on the following services "iscsi iodrive raw" which means vxvm will start later and after the operating system has probed all devices
8. The "nofail" attribute for Veritas VxVM devices in /etc/fstab  causes  a shutdown issue for others systemD services that are dependent on the VxFS filesystem
9. Systemd-udevd: inotify_add_watch(7, /dev/VxDMP[X], failed: No such file or directory messages in the journal/messages log
10. systemd-fsck reports the following messages "systemd-fsck[xxxx]: UX:vxfs fsck.vxfs: ERROR: V-3-20113: Cannot open: No such device or address"
11. VxVM has an additional 2 minute sleep to wait for infiniband devices to become available. this can create an addtional delay if you are not using these devices in your environment
12. With the native systemd services for VeKI.service and VxFS.service we have simplified the dependency order
13. SystemD VxFS .mount units may reach the systemD default timeout and processes called by mount can be subsequently killed
14. Volumes that are under an RVG will report IO error as the RVG has not yet be enabled/active
15. Systemd-fsck@[xxx] start failed with result 'dependency'
16. The netbackup service may fail if VxVM volumes are not yet available

Solution

NetBackup service initialization may fail if VxVM services have not yet started. 

This HotFix will add a systemD generator script which adds each mount point from the 'nbuapp' disk-group in the correct dependency order.

This HotFix is available here

These volumes are defined via the “nbu_vols” attribute in the /etc/vx/systemd/system.conf file. 

This will mitigate issues identified above which pertain to the startup and shutdown sequence.