NVME: InfoScale 7.4 is reporting udid_mismatch for NVMe devices & vxdisk -c updateudid fails to persistently update on-disk UDID content

Article ID:100044352
Last Published:2018-11-01
Product(s):InfoScale & Storage Foundation

Problem

Platform: Solaris 11 Sparc

NVMe (Non-Volatile Memory Express) is a host controller interface & storage protocol created to accelerate the transfer of data using solid-state drives (SSDs)

Uses a servers high-speed Peripheral Component Interconnect Express (PCIe) interface

InfoScale (IS) 7.3.1 introduces support for NVMe (Non-Volatile Memory Express) devices with Solaris Sparc servers

IS 7.3 initially introduced support for Linux

- Generic ASL support was developed for Infoscale (IS) 7.3 for NVMe & SATA disks
- The existing IS 7.3 based Intel NVMe ASL was also enhanced to claim other NVMe disks as well
- The Generic NVMe ASL will attempt to claim NVMe type devices

When upgrading to InfoScale (IS) 7.4 the NVMe devices may display the udid_mismatch state when running vxdisk list or related command.

The 'vxdisk -c updateudid <veritas-access-name>' command can be used to update the on-disk UDID (Unique Disk Identifier), however this command fails to persistently update the UDID content and clear the udid_mismatch flag.

NOTE: Solaris 11.4 is not currently supported with InfoScale 7.4 or below

Cause

The product is encountering a string buffer overflow issue when updating the UDID (Unique Disk Identifer) from the DDL (Device Discovery Layer) down to the private region area on-disk, hence the udid_mismatch flag is set.

# vxdisk -px LIST_UDID list
<snippet>
oracle7_nvme0_0 NVMe%5F8086108E%5FCVMD532400AX1P6NGN%5FINTEL%20SSDPEDME016T4S%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%208DV1RA13 NVMe%5F8086108E%5FCVMD532400AX1P6NGN%5FINTEL%20SSDPEDME016T4S%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%208DV1RA1

# echo NVMe%5F8086108E%5FCVMD532400AX1P6NGN%5FINTEL%20SSDPEDME016T4S%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%208DV1RA13 | wc -c
130

We can see UDID kernel length of the NVMe device is 130 characters.

NOTE: The maximum UDID length is 376

echo NVMe%5F8086108E%5FCVMD532400AX1P6NGN%5FINTEL%20SSDPEDME016T4S%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%208DV1RA1 | wc -c
129

NULL characters are lost whilst copying the UDID from the kernel to the private region.

To overcome this issue, the Generic ASL 'libvxnvme_sol.so' used to claim NVMe devices has been modified.

The NVMe ASL has been modified to remove blank spaces from the LUN Serial Number (LSN) which reduces the overal size of the UDID string for the NVMe device.

With the updated ASLAPM (Array Support Library) package, it is now possible to persistently update the on-disk UDID content.

Test Environment Details:

# cat /etc/*release
                            Oracle Solaris 11.3 SPARC
Copyright (c) 1983, 2018, Oracle and/or its affiliates. All rights reserved.
                              Assembled 09 May 2018

# pkg info entire
             Name: entire
          Summary: entire incorporation including Support Repository Update (Oracle Solaris 11.3.33.5.0).
      Description: This package constrains system package versions to the same
                   build. WARNING: Proper system update and correct package
                   selection depend on the presence of this incorporation.
                   Removing this package will result in an unsupported system.
                  For more information see:
                   https://support.oracle.com/rs?type=doc&id=2045311.1
         Category: Meta Packages/Incorporations
            State: Installed
        Publisher: solaris
          Version: 0.5.11 (Oracle Solaris 11.3.33.5.0)
    Build Release: 5.11
           Branch: 0.175.3.33.0.5.0
   Packaging Date: Wed May 30 18:36:17 2018
Last Install Time: Wed Oct 31 01:18:07 2018
             Size: 5.46 kB
             FMRI: pkg://solaris/entire@0.5.11,5.11-0.175.3.33.0.5.0:20180530T183617Z

# pkg list entire
NAME (PUBLISHER) VERSION IFO
entire (solaris) 0.5.11-0.175.3.33.0.5.0 i--

# pkginfo -l VRTSaslapm

   PKGINST: VRTSaslapm
      NAME: Array Support Libraries and Array Policy Modules for Veritas Volume Manager
CATEGORY: system
      ARCH: sparc
   VERSION: 7.4.0.000,REV=05.15.2018.18.24
   BASEDIR: /
    VENDOR: Veritas Technologies LLC
      DESC: Array Support Libraries and Array Policy Modules for Veritas Volume Manager
INSTDATE: Nov 01 2018 01:18
   HOTLINE: https://www.veritas.com/content/support/en_US/contact-us.html
    STATUS: completely installed

Solution

The following ASLAP (Array Support Library) has been enhanced to handle a more efficient method of retrieving the UDID content from the DDL (Device Discovery Layer) and enabling the user to update the UDID content.

Please connect to sort.veritas.com and download the updated 7.4.0.1001 ASLAPM package using the following link:

https://sort.veritas.com/asl/details/839

Direct download link:

https://sort.veritas.com/aslcentral/Solaris/839/VRTSaslapm_Solaris_7.4.0.1001.tar.gz

UPDATING ASLAPM

# pkg install --accept -g ./VRTSaslapm.p5p VRTSaslapm
Packages to update: 1
Create boot environment: No
Create backup boot environment: Yes

DOWNLOAD PKGS FILES XFER (MB) SPEED
Completed 1/1 149/149 0.3/0.3 3.1M/s

PHASE                                          ITEMS
Updating modified actions                    152/152
Updating package state database                 Done
Updating package cache                           1/1
Updating image state                            Done
Creating fast lookup database                   Done
Updating package cache                           1/1

# pkginfo -l VRTSaslapm
   PKGINST: VRTSaslapm
      NAME: Array Support Libraries and Array Policy Modules for Veritas Volume Manager
CATEGORY: system
      ARCH: sparc
   VERSION: 7.4.0.1001,REV=10.19.2018.18.33
   BASEDIR: /
    VENDOR: Veritas Technologies LLC
      DESC: Array Support Libraries and Array Policy Modules for Veritas Volume Manager
INSTDATE: Nov 01 2018 01:31
   HOTLINE: https://www.veritas.com/content/support/en_US/contact-us.html
    STATUS: completely installed

# vxdisk scandisks

# vxddladm list devices
DEVICE               TARGET-ID    STATE   DDL-STATUS (ASL)
===============================================================
c1t0d0s2             -            Online CLAIMED (ALUA)
c6t5000CCA01D1EF4B9d0 -            Online CLAIMED (Disk)
c3t1d0               -            Online CLAIMED (libvxnvme_sol.so)
c5t5000CCA02D100E79d0 -            Online CLAIMED (Disk)

Once the updated 7.4.0.1001 ASLAPM package has been applied, the system should be rebooted for the required disk changes to be discovered and populated by the Veritas Volume Manager vxconfigd daemon.

Post-reboot sample output:

The NVMe device is reporting a udid_mismatch flag following the system reboot and following the installation of the ASLAPM package:

# vxdisk list
DEVICE          TYPE            DISK         GROUP        STATUS
oracle7_aluadisk_0 auto:none       -            -            online invalid
oracle7_disk_0 auto:ZFS        -            -            ZFS
oracle7_disk_1 auto:ZFS        -            -            ZFS
oracle7_nvme0_1 auto:cdsdisk    -            -            online udid_mismatch

The 'vxdisk -px LIST_UDID list' attribute view listing will also reporting a mismatch between the UDID (in kernel) and PRIV_UDID (on-disk):

# vxdisk -px LIST_UDID list
DEVICE              UDID    PRIV_UDID
oracle7_aluadisk_0 MICRON%5FeUSB%20DISK%5FALUAdisk%5F2BF0022700183545                 -
oracle7_disk_0     HGST%5FH101212SESUN1.2T%5FDISKS%5F5000CCA01D1EF4B8                 -
oracle7_disk_1     HGST%5FH101812SFSUN1.2T%5FDISKS%5F5000CCA02D100E78                 -
oracle7_nvme0_1    NVMe%5F8086108E%5FCVMD532400AX1P6NGN%5FINTELSSDPEDME016T4S8DV1RA13 NVMe%5F8086108E%5FCVMD532400AX1P6NGN%5FINTEL%20SSDPEDME016T4S%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%208DV1RA1VXVM

The -c option should be used to update the on-disk UDID content for the NVMe disk(s):

# vxdisk -c updateudid oracle7_nvme0_1

# vxdisk -px LIST_UDID list
DEVICE              UDID                                                               PRIV_UDID
oracle7_aluadisk_0 MICRON%5FeUSB%20DISK%5FALUAdisk%5F2BF0022700183545                 -
oracle7_disk_0     HGST%5FH101212SESUN1.2T%5FDISKS%5F5000CCA01D1EF4B8                 -
oracle7_disk_1     HGST%5FH101812SFSUN1.2T%5FDISKS%5F5000CCA02D100E78 -
oracle7_nvme0_1    NVMe%5F8086108E%5FCVMD532400AX1P6NGN%5FINTELSSDPEDME016T4S8DV1RA13 NVMe%5F8086108E%5FCVMD532400AX1P6NGN%5FINTELSSDPEDME016T4S8DV1RA13

# vxdisk -e list
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
oracle7_aluadisk_0 auto:none      -            -           online invalid       c1t0d0s2         -
oracle7_disk_0 auto:ZFS       -            -           ZFS                  c6t5000CCA01D1EF4B9d0 -
oracle7_disk_1 auto:ZFS       -            -           ZFS                  c5t5000CCA02D100E79d0 -
oracle7_nvme0_1 auto:cdsdisk   -            -           online               c3t1d0           ssd, ssd

NOTE: The udid_mismatch flag is now longer reported, even after the execution of vxdisk scandisks.

# vxdisk scandisks

# vxdisk list
DEVICE          TYPE            DISK         GROUP        STATUS
oracle7_aluadisk_0 auto:none       -            -            online invalid
oracle7_disk_0 auto:ZFS        -            -            ZFS
oracle7_disk_1 auto:ZFS        -            -            ZFS
orcle7_nvme0_1 auto:cdsdisk    -            -            online

Purely for testing purposes, the Veritas Volume Manager vxconfigd daemon is also recycled to ensure the udid_mismatch flag does not return.

# vxconfigd -k

# vxdisk -e list
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
oracle7_aluadisk_0 auto:none      -            -           online invalid       c1t0d0s2         -
oracle7_disk_0 auto:ZFS       -            -           ZFS                  c6t5000CCA01D1EF4B9d0 -
oracle7_disk_1 auto:ZFS       -            -          ZFS                  c5t5000CCA02D100E79d0 -
oracle7_nvme0_1 auto:cdsdisk   -            -           online               c3t1d0           ssd, ssd

Additional testing is also performed to ensure a removed NVMe related Veritas disk access name returns without the udid_mismatch flag:

# vxdisk rm p7461-04_nvme0_1
# vxdisk scandisks

# vxdisk -e list
DEVICE       TYPE           DISK        GROUP        STATUS               OS_NATIVE_NAME   ATTR
oracle7_aluadisk_0 auto:none      -            -           online invalid       c1t0d0s2         -
oracle7_disk_0 auto:ZFS       -            -           ZFS                  c6t5000CCA01D1EF4B9d0 -
oracle7_disk_1 auto:ZFS       -            -           ZFS                  c5t5000CCA02D100E79d0 -
oracle7_nvme0_1 auto:cdsdisk   -            -           online               c3t1d0           ssd, ssd

Summary:

The udid values have been shortened and aligned with each other following the installation of the updated ASLAPM, system rebooted and vxdisk -c updateudid <veritas-access-name> update:

Before and After UDID values

Before rebooting system and updating UDID values:

# vxdisk -px LIST_UDID list
<snippet>
oracle7_nvme0_0 NVMe%5F8086108E%5FCVMD532400AX1P6NGN%5FINTEL%20SSDPEDME016T4S%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%208DV1RA13 NVMe%5F8086108E%5FCVMD532400AX1P6NGN%5FINTEL%20SSDPEDME016T4S%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%208DV1RA1

<snippet>

After reboot and after updating UDID content:

# vxdisk -px LIST_UDID list
<snippet>
oracle7_nvme0_1 NVMe%5F8086108E%5FCVMD532400AX1P6NGN%5FINTELSSDPEDME016T4S8DV1RA13 NVMe%5F8086108E%5FCVMD532400AX1P6NGN%5FINTELSSDPEDME016T4S8DV1RA13
<snippet>

NOTE: UDID values are now aligned and much shorter

Was this content helpful?

Rating submitted. Please provide additional feedback (optional):

NVME: InfoScale 7.4 is reporting udid_mismatch for NVMe devices & vxdisk -c updateudid fails to persistently update on-disk UDID content

Problem

Cause

Solution

Was this content helpful?

Get Support

Create Case

Chat

Phone

NVME: InfoScale 7.4 is reporting udid_mismatch for NVMe devices & vxdisk -c updateudid fails to persistently update on-disk UDID content

Problem

Cause

Solution

Was this content helpful?

Article Languages

Translated Content

Get Support

Create Case

Chat

Phone

Hello! We noticed that while you have a Veritas Account, you aren’t yet registered to manage cases and use chat. We have created a case for our Customer Care team to get you registered. If you require immediate assistance, please call us and we would be happy to assist.