Storage Foundation (SF) dynamic multipathing driver (vxdmp) may fail to load on IBM AIX machines with large amounts of physical memory

Problem

The DMP driver load fails after upgrading to 5.1SP1 on AIX 5.3, AIX 6.1 or 5.1SP1PR1 (Platform Release 1) on AIX 7.1. This issue is noticed on some systems with memory greater than 35GB. This is due to failure in creating memory pool which is utilized to serve I/Os in DMP.

The problem can also cause system panic if VxVM command tries to open DMP device files (e.g. /dev/vx/dmpconfig) after the DMP driver failed to load into the memory.

Error Message

The following messages are seen in the errpt when the vxdmp module fails to load:

WARNING VxVM vxdmp V-5-0-1442 : Could not create dmppool error code 12 retry 0
WARNING VxVM vxdmp V-5-0-1481:dmp ldata pool creation failed

Following this the Volume Manager configuration daemon (vxconfigd) fails to start due to the vxdmp module not being loaded on the system:

The following is the system panic stack when a VxVM command (vxesd in this case) tried to open the DMP device file (/dev/vx/dmpconfig) after the DMP driver failed to load into the memory.

(0)> th * | grep pvthread+02A200
pvthread+02A200  674*vxesd    RUN   2A2023 03C    0         0               <<< vxesd command

CRASH INFORMATION:
CPU 0 CSA F00000002FF47600 at time of crash, error code for LEDs: 70000000
pvthread+02A200 STACK:
[04CF9760]04CF9760 ()
[003F7380]devcopen+00033C (??, ??, ??, ??, ??)
[003F6CE0]rdevopen+000094 (??, ??, ??, ??, ??)
[0052E970]cdev_open+000054 (F10001005FE8B5F0, 0000000300000003, 0000000000000000)
[004CE018]spec_open@AF69_29+00016C (??, ??, ??, ??, ??, ??)
[0040AD3C]vnop_open+0000E8 (??, ??, ??, ??, ??)
[0044CF98]openpnp+0002FC (??, ??, ??, ??, ??)
[0044D47C]openpath+0000BC (??, ??, ??, ??, ??, ??, ??)
[0044D738]copen+0001F4 (??, ??, ??, ??, ??)
[0044CC38]open+00001C (??, ??, ??)
[04DA5A88]my_open+0003A4 (000000002001D2A0, 0000000200000002, 2002831820028318)
[00003810].svc_instr+000110 ()
[kdb_get_memory] no real storage @ 2FF22980

In some cases, we have seen the following panic stack pointing to OS syscall ldata_create().

Cause

The issue is caused by the way in which the vxdmp module attempts to perform initial memory allocation when loaded on the system. In the 5.1SP1 product the module attempts to reserve a given percentage of physical memory for use by vxdmp memory pools. On systems with large amounts of physical memory this can lead to a request for an extremely large allocation of memory which the system cannot honour, and as such the module fails to load as expected.

Solution

To prevent the issue, changes have been made to vxdmp code to ensure that an upper threshold is placed on the amount of memory requested during module load. This ensures that even on systems with large amounts of physical memory DMP will only request a reasonable amount of memory be allocated for vxdmp memory pools.

This issue is resolved with following patches for Veritas Volume Manager on AIX:

- Veritas Storage Foundation 5.1SP1 RP1 for AIX 5.3 and AIX 6.1  (Rolling Patch 1)
- VxVM 5.1SP1 P2 for AIX 5.3 and AIX 6.1 (P-Patch 2)
- VxVM 5.1SP1 PR1P1 for AIX 7.1 (Platform Release 1 + P-Patch 1)
 

The above patches can be downloaded from SORT site https://sort.Veritas.com/patch/matrix
 

Applies To

AIX 5.3/6.1/7.1

Storage Foundation 5.1SP1 for AIX 5.3 and 6.1 or 5.1SP1PR1 (Platform Release 1) for AIX 7.1.

Large amount of physical memory present on the system (i.e. > 35Gb)