The DMP driver load fails after upgrading to 5.1SP1 on AIX 5.3, AIX 6.1 or 5.1SP1PR1 (Platform Release 1) on AIX 7.1. This issue is noticed on some systems with memory greater than 35GB. This is due to failure in creating memory pool which is utilized to serve I/Os in DMP.
The problem can also cause system panic if VxVM command tries to open DMP device files (e.g. /dev/vx/dmpconfig) after the DMP driver failed to load into the memory.
The following messages are seen in the errpt when the vxdmp module fails to load:
WARNING VxVM vxdmp V-5-0-1442 : Could not create dmppool error code 12 retry 0
WARNING VxVM vxdmp V-5-0-1481:dmp ldata pool creation failed
Following this the Volume Manager configuration daemon (vxconfigd) fails to start due to the vxdmp module not being loaded on the system:
# vxconfigd -k
VxVM vxconfigd FATAL ERROR V-5-1-8379 vxdmp driver not loaded on the system. This version of VxVM mandatorily requires vxdmp to be loaded. Exiting.
# lsdev -C | grep vx
vxdmp Defined Veritas VxDMP Device Driver <<<= Defined but not Available
vxfen Available N/A
vxfend Available N/A
vxglm0 Available N/A
vxgms0 Available N/A
vxio Defined Veritas VxIO Device Driver
vxportal0 Available VERITAS VxPORTAL Device Driver
vxqio0 Available VERITAS VxQIO Device Driver
vxspec Defined Veritas VxSPEC Device Driver
The following is the system panic stack when a VxVM command (vxesd in this case) tried to open the DMP device file (/dev/vx/dmpconfig) after the DMP driver failed to load into the memory.
(0)> th * | grep pvthread+02A200
pvthread+02A200 674*vxesd RUN 2A2023 03C 0 0 <<< vxesd command
CPU 0 CSA F00000002FF47600 at time of crash, error code for LEDs: 70000000
[003F7380]devcopen+00033C (??, ??, ??, ??, ??)
[003F6CE0]rdevopen+000094 (??, ??, ??, ??, ??)
[0052E970]cdev_open+000054 (F10001005FE8B5F0, 0000000300000003, 0000000000000000)
[004CE018]spec_open@AF69_29+00016C (??, ??, ??, ??, ??, ??)
[0040AD3C]vnop_open+0000E8 (??, ??, ??, ??, ??)
[0044CF98]openpnp+0002FC (??, ??, ??, ??, ??)
[0044D47C]openpath+0000BC (??, ??, ??, ??, ??, ??, ??)
[0044D738]copen+0001F4 (??, ??, ??, ??, ??)
[0044CC38]open+00001C (??, ??, ??)
[04DA5A88]my_open+0003A4 (000000002001D2A0, 0000000200000002, 2002831820028318)
[kdb_get_memory] no real storage @ 2FF22980
In some cases, we have seen the following panic stack pointing to OS syscall ldata_create().
[0052E89C]vm_place_common+00027C (0000000000000000, 0000000000000000,
00000000000002A0, 00000000000017A0, 000000000000000E, 0000000000000000,
vm_place+000034 (??, ??, ??, ??, ??, ??)
[001F0CE4]ldata_create+000444 (??, ??, ??, ??, ??) <== NOTE
[05CCA3FC]dmp_ldata_create+000044 (0000000005D40D40, 0000000005D40D60)
[05CCA530]ldata_pool_init_create+0000E8 (0000000005D40D40, 0000000000000188,
[05CC1510]vxdmp_config+000224 (800000280000FFFF, 0000000100000001,
[005F2430]config_dd+000270 (??, ??, ??)
[005F2F00]sysconfig+000240 (??, ??, ??)
[kdb_get_virtual_memory] no real storage @ 2FF21A20
[kdb_read_mem] no real storage @ FFFFFFFFFFF9900
The issue is caused by the way in which the vxdmp module attempts to perform initial memory allocation when loaded on the system. In the 5.1SP1 product the module attempts to reserve a given percentage of physical memory for use by vxdmp memory pools. On systems with large amounts of physical memory this can lead to a request for an extremely large allocation of memory which the system cannot honour, and as such the module fails to load as expected.
To prevent the issue, changes have been made to vxdmp code to ensure that an upper threshold is placed on the amount of memory requested during module load. This ensures that even on systems with large amounts of physical memory DMP will only request a reasonable amount of memory be allocated for vxdmp memory pools.
This issue is resolved with following patches for Veritas Volume Manager on AIX:
- Veritas Storage Foundation 5.1SP1 RP1 for AIX 5.3 and AIX 6.1 (Rolling Patch 1)
- VxVM 5.1SP1 P2 for AIX 5.3 and AIX 6.1 (P-Patch 2)
- VxVM 5.1SP1 PR1P1 for AIX 7.1 (Platform Release 1 + P-Patch 1)
The above patches can be downloaded from SORT site https://sort.symantec.com/patch/matrix
Storage Foundation 5.1SP1 for AIX 5.3 and 6.1 or 5.1SP1PR1 (Platform Release 1) for AIX 7.1.
Large amount of physical memory present on the system (i.e. > 35Gb)