InfoScale 7.4.1 is not discovering multiple NVME devices as separate enclosures

InfoScale 7.4.1 is not discovering multiple NVME devices as separate enclosures

Article: 100045733
Last Published: 2019-06-17
Ratings: 4 0
Product(s): InfoScale & Storage Foundation

Problem


InfoScale (IS) 7.3 introduces support for NVMe (Non-Volatile Memory Express) devices for Linux. IS 7.3.1 included NVMe support for Solaris Sparc servers.
 

InfoScale 7.4.1 is not discovering multiple NVME devices as separate enclosures.

In this example, only a single NVMe enclosure is being reported by DMP.


# vxdmpadm listctlr all output
CTLR_NAME   ENCLR_TYPE    STATE     ENCLR_NAME     PATH_COUNT
=========================================================================
c0          EMC           ENABLED   emc0           145
c2          NVMe          ENABLED   nns8a2_nvme0   1
c1          OTHER_DISKS   ENABLED   other_disks    2  


 

The format command is showing the NVMe devices as c2t1d0 and c3t1d0:
 

       0. c2t1d0 <Intel-7335943:ICDPC5ED-QDV1-2.91TB>

          /pci@307/pci@1/pci@0/pci@1/nvme@0/disk@1

          /dev/chassis/SYS/MB/PCIE2/NVME0/disk

       1. c3t1d0 <Intel-7335943:ICDPC5ED-QDV1-2.91TB>

          /pci@307/pci@1/pci@0/pci@2/nvme@0/disk@1

          /dev/chassis/SYS/MB/PCIE2/NVME1/disk

 


In the below output, NVMe device (c3t1d0) is excluded by the DDL (Device Discovery Layer), so will not be shown in any of the VxVM outputs.
 

# vxddladm list devices 
DEVICE               TARGET-ID    STATE   DDL-STATUS (ASL)
===============================================================
c3t1d0               -            Online  EXCLUDED
.
.
c1d1                 -            Online  CLAIMED (OTHER_DISKS)
c1d1s2               -            Offline -
c1d0                 -            Online  CLAIMED (OTHER_DISKS)
c2t1d0               -            Online  CLAIMED (libvxnvme_sol.so)

 

Error Message


Sample vxdmp messages

May 29 09:55:55 host vxdmp: [ID 144536 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 [Info] disabled controller /pci@307/pci@1/pci@0/pci@1/nvme@0 connected to disk array PHLE810101046P4BGN-
May 29 09:55:55 host vxdmp: [ID 241556 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 [Info] removed disk array PHLE810101046P4BGN-, datype = NVMe
May 29 09:55:55 host vxdmp: [ID 144536 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 [Info] disabled controller /pci@307/pci@1/pci@0/pci@2/nvme@0 connected to disk array PHLE810101046P4BGN-

May 29 09:55:55 host vxvm:vxconfigd: [ID 702911 daemon.warning] V-5-1-11551 array_da_to_disk: Cannot find active path for dmpnode host_nvme0_0
May 29 09:55:55 host vxvm:vxconfigd: [ID 702911 daemon.warning] 
May 29 09:55:55 host vxvm:vxconfigd: [ID 702911 daemon.warning] V-5-1-11551 array_da_to_disk: Cannot find active path for dmpnode host_nvme0_0
May 29 09:55:55 host vxvm:vxconfigd: [ID 702911 daemon.warning] 

May 29 09:55:55 host vxdmp: [ID 581508 kern.notice] NOTICE: VxVM vxdmp V-5-0-34 [Info] added disk array PHLE810101046P4BGN-, datype = NVMe
May 29 09:55:55 host vxdmp: [ID 144536 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 [Info] enabled controller /pci@307/pci@1/pci@0/pci@2/nvme@0 connected to disk array PHLE810101046P4BGN-

May 29 09:57:50 host vxdmp: [ID 144536 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 [Info] enabled controller /pci@307/pci@1/pci@0/pci@1/nvme@0 connected to disk array PHLE810101046P4BGN-
May 29 09:58:16 host vxdmp: [ID 241556 kern.notice] NOTICE: VxVM vxdmp V-5-0-0 [Info] removed disk array PHLE810101046P4BGN-, datype = NVMe
 

Cause

NVMe (Non-Volatile Memory Express) is a host controller interface & storage protocol created to accelerate the transfer of data using solid-state drives (SSDs)

Uses a servers high-speed Peripheral Component Interconnect Express (PCIe) interface.

Escalation: STESC-3229

InfoScale 7.4.1 running Solaris 11.4 is unable to detect multiple NVMe devices correctly due to Cabinet (Enclosure)Serial Number (CSN) length. This can potential result in only a single NVMe device being discovered and others excluded from the DDL (Device Discovery Layer) claiming process.

The Solaris nvmeadm command can be used to display NVMe related device information:
 

# nvmeadm  list -v

SUNW-NVME-1
      PCI Vendor ID:                  0x8086
      Serial Number:                  PHLE810101046P4BGN-1
      Model Number:                   7335943:ICDPC5ED2ORA6.4T
      Firmware Revision:              QDV1RD22
      Number of Namespaces:           1

SUNW-NVME-2
       PCI Vendor ID:                  0x8086
 
     Serial Number:                  PHLE810101046P4BGN-2  
   
   Model Number:                   7335943:ICDPC5ED2ORA6.4T
 
     Firmware Revision:              QDV1RD22
 
     Number of Namespaces:           1

 

The DMP_MAX_CAB_SNO_NVME was originally defined with a length of 20 bytes. Therefore, DMP cuts the string too short, hence why in instance only “PHLE810101046P4BGN-” is detected by DMP. DMP fails to recognize other NVMe devices (enclosures).

VxVM uses only “PHLE810101046P4BGN-” as the Enclosure (Cabinet) serial number for a NVMe devices.

May 30 14:24:22 host vxdmp: [ID 581508 kern.notice] NOTICE: VxVM vxdmp V-5-0-34 [Info] added disk array PHLE810101046P4BGN-, datype = NVMe"

By increasing the size of the DMP_MAX_CAB_SNO_NVME attribute size, the product will now be able to discover multiple NVME devices.

Solution

Once the updated VRTSaslapm package has been applied, unique enclosures are detected for multiple NVMe devices (enclosures).

NOTICE: VxVM vxdmp V-5-0-34 [Info] added disk array OTHER_DISKS, datype = OTHER_DISKS
NOTICE: VxVM vxdmp V-5-0-34 [Info] added disk array PHLE810101046P4BGN-17335943:ICDP, datype = NVMe
NOTICE: VxVM vxdmp V-5-0-34 [Info] added disk array 000296700818, datype = EMC
NOTICE: VxVM vxdmp V-5-0-34 [Info] added disk array PHLE810101046P4BGN-27335943:ICDP, datype = NVMe

By increasing the DMP_MAX_CAB_SNO_NVME attribute size, the product is now able to discover multiple NVME devices as individual enclosures.

The output from "vxdisk -p list" now clearly shows the unique CAB_SERIAL_NO for each NVMe device:

DISK                : host_nvme1_0
VID                 : NVMe
UDID                : NVMe%5F8086108E%5FPHLE810101046P4BGN-17335943%3AICDP%5F14F1F700005CD2E40000000100000001
SCSI_VERSION        : 0
PID                 : 8086108E
PHYS_CTLR_NAME      : /pci@307/pci@1/pci@0/pci@1/nvme@0
PGR_CAPABLE         : N
MEDIA_TYPE          : ssd
LUN_TYPE            : std
LUN_SNO_ORDER       : 0
LUN_SERIAL_NO       : 14F1F700005CD2E40000000100000001
LIBNAME             : libvxnvme_sol.so
DMP_DEVICE          : host_nvme1_0
DDL_DEVICE_ATTR     : ssd
CONN_TYPE           : local
CAB_SERIAL_NO       : PHLE810101046P4BGN-17335943:ICDP
ATYPE               : A/A
ANAME               : NVMe
TRANSPORT           : SCSI
ENCLOSURE_NAME      : host_nvme1
DMP_SINGLE_PATH     : /dev/rdsk/c2t1d0
LUN_SIZE            : 6251233968
NUM_PATHS           : 1
STATE               : online
DISK_TYPE           : auto
FORMAT              : cdsdisk
DA_INFO             : format=cdsdisk,privoffset=208,pubslice=2,privslice=2
PRIV_OFF            : 208
PRIV_LEN            : 65536
PUB_OFF             : 65744
PUB_LEN             : 6251168128
PRIV_UDID           : NVMe%5F8086108E%5FPHLE810101046P4BGN-17335943%3AICDP%5F14F1F700005CD2E40000000100000001
DG_NAME             : sfcache_defaultdg
DGID                : 1559309762.168.host
DG_STATE            : deported
DISKID              : 1559058982.16.host
DISK_TIMESTAMP      : Tue May 28 05:56:22 PM 2019

DISK                : host_nvme2_0
VID                 : NVMe
UDID                : NVMe%5F8086108E%5FPHLE810101046P4BGN-27335943%3AICDP%5F14F1F700005CD2E40000000100000001
SCSI_VERSION        : 0
PID                 : 8086108E
PHYS_CTLR_NAME      : /pci@307/pci@1/pci@0/pci@2/nvme@0
PGR_CAPABLE         : N
MEDIA_TYPE          : ssd
LUN_TYPE            : std
LUN_SNO_ORDER       : 0
LUN_SERIAL_NO       : 14F1F700005CD2E40000000100000001
LIBNAME             : libvxnvme_sol.so
DMP_DEVICE          : host_nvme2_0
DDL_DEVICE_ATTR     : ssd
CONN_TYPE           : local
CAB_SERIAL_NO       : PHLE810101046P4BGN-27335943:ICDP
ATYPE               : A/A
ANAME               : NVMe
TRANSPORT           : SCSI
ENCLOSURE_NAME      : host_nvme2
DMP_SINGLE_PATH     : /dev/rdsk/c3t1d0
LUN_SIZE            : 6251233968
NUM_PATHS           : 1
STATE               : online
DISK_TYPE           : auto
FORMAT              : none
DA_INFO             : format=none

 

Patch is available for download from SORT site:

https://sort.veritas.com/asl/details/858

Was this content helpful?