Server panicked (Panic String : BAD TRAP: type=31 rp=2a101f4d480 addr=8 mmu_fsr=0 occurred in module "vxdmp" due to a NULL pointer dereference)

Problem

 Server was panicked with the following panic string :

BAD TRAP: type=31 rp=2a101f4d480 addr=8 mmu_fsr=0 occurred in module "vxdmp" due to a NULL pointer dereference

Cause

The panic occurs as the inconsistence of dmpdb/devlist between vxesd and vold/vxdmp. When panic occurs, because of SAN outage/restore, devices were lost then detected in vxesd but they were not in vold/vxdmp. Thus, if we issue device map updates from vxesd, the dmpnode may do not exists in vxdmp, panic occurs.

Solution

 Analysis of Crash Dump : SolarisCAT(415-098-321_vmcore.2/10U)> panic

 panic on CPU 9

panic string:   BAD TRAP: type=31 rp=2a101f4d480 addr=8 mmu_fsr=0 occurred in module "vxdmp" due to a NULL pointer dereference

==== panic user (LWP_SYS) thread: 0x3000baca740  PID: 561  on CPU: 9  affinity CPU: 9 ====

cmd: /sbin/vxesd

t_procp: 0x6007da08050

  p_as: 0x60069fb4910  size: 945283072  RSS: 135831552

  hat: 0x30010402700

    cnum: CPU0:2463/371 CPU2:2111/140 CPU4:2066/131 CPU6:2099/133 CPU8:2713/251 CPU10:2260/109 CPU12:2209/270 CPU14:2231/138

    cpusran: 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15

  zone: global

t_stk: 0x2a101f4dae0  sp: 0x18c0791  t_stkbase: 0x2a101f48000

t_pri: 59(TS)  t_tid: 1  pctcpu: 0.010421

t_lwp: 0x600637880a0  machpcb: 0x2a101f4dae0

  mstate: LMS_SYSTEM  ms_prev: LMS_KFAULT

  ms_state_start: 0.0000741 seconds earlier

  ms_start: 190 days 1 hours 57 minutes 5.3898810 seconds earlier

psrset: 0  last CPU: 9

idle: 0 ticks (0 seconds)

start: Sat Jan 29 00:08:16 2011

age: 16422865 seconds (190 days 1 hours 54 minutes 25 seconds)

syscall: #54 ioctl(, 0xffbffb20) (sysent: genunix:ioctl+0x0)

tstate: TS_ONPROC - thread is being run on a processor

tflg:   T_PANIC - thread initiated a system panic

        T_DFLTSTK - stack is default size

tpflg:  TP_TWAIT - wait to be freed by lwp_wait

        TP_MSACCT - collect micro-state accounting information

tsched: TS_LOAD - thread is in memory

        TS_DONT_SWAP - thread/LWP should not be swapped

pflag:  SMSACCT - process is keeping micro-state accounting

        SMSFORK - child inherits micro-state accounting

pc:      unix:panicsys+0x48:   call     unix:setjmp

unix:panicsys+0x48(0x10a4bb8, 0x2a101f4d228, 0x18c1160, 0x1, , , 0x9900001607, , , , , , , , 0x10a4bb8, 0x2a101f4d228)

unix:vpanic_common+0x78(0x10a4bb8, 0x2a101f4d228, 0x36921, 0x2a101f4cd21, 0x2a101f4d520, 0x4480001602)

unix:panic+0x1c(0x10a4bb8, 0x31, 0x2a101f4d480, 0x8, 0x0, 0x30009fb6240, 0x183a368)

unix:die+0x78(0x31, 0x2a101f4d480, 0x8, 0x0)

unix:trap+0x9e0(0x2a101f4d480, 0x8)

unix:ktl0+0x48()

-- trap data  type: 0x31 (data access MMU miss)  rp: 0x2a101f4d480  --

  addr: 0x8

pc:  0x7be5c688 vxdmp:dmp_decode_modmap_dmpnode+0x1c4:   ldx    [%l4 + 0x8], %o2

npc: 0x7be5c68c vxdmp:dmp_decode_modmap_dmpnode+0x1c8:   subcc  %o2, 0x0, %g0   ( cmp   %o2, 0x0             )

  global:                       %g1         0x7be5c678

        %g2              0x200  %g3           0x41798f

        %g4          0x19bd000  %g5             0x19bd

        %g6                  0  %g7      0x3000baca740

  out:  %o0             0x19bf  %o1             0x1800

        %o2            0x10080  %o3              0x200

        %o4          0x10a1000  %o5             0x10a1

        %sp      0x2a101f4cd21  %o7         0x7be5c680

  loc:  %l0      0x11800000008  %l1                  0

        %l2           0x100003  %l3                0x9

        %l4                  0  %l5         0x375f6070

        %l6                  0  %l7                0x1

  in:   %i0      0x2a101f4d698  %i1           0x100003

        %i2         0x375f6064  %i3          0x19b8cb8

        %i4                  0  %i5                  0

        %fp      0x2a101f4cde1  %i7         0x7be576d0

<trap>vxdmp:dmp_decode_modmap_dmpnode+0x1c4(0x2a101f4d698, 0x100003, 0x0, 0x19b8cb8)

vxdmp:dmp_decipher_instructions+0x364(0x2a101f4d758, 0x4f, 0x100003, 0x0, 0x19b8cb8)

vxdmp:dmp_process_instruction_buffer+0x150(0x1180003ffff, 0x78ae8, 0x100003, 0x0, 0x19b8cb8)

vxdmp:dmp_reconfigure_db+0x48(, 0x78ae8, 0x100003)

vxdmp:gendmpioctl(0x1180003ffff, , 0x78ae8, 0x100003, 0x6007d831ee0, 0x2a101f4dadc) - frame recycled

vxdmp:dmpioctl+0x20(, 0x444d5040, 0x78ae8, 0x100003, 0x6007d831ee0)

specfs:spec_ioctl(0x60069b53500, 0x444d5040, 0x78ae8, 0x100003, 0x6007d831ee0) - frame recycled

genunix:fop_ioctl+0x20(0x60069b53500, 0x444d5040, 0x78ae8, 0x100003, , 0x2a101f4dadc)

genunix:ioctl+0x184()

unix:syscall_trap32+0xcc()

-- switch to user thread's user stack --

======================================

Found etrack 2141176: Panic: BAD TRAP: type=31 in dmp_decode_modmap_dmpnode()

HF4 for 5.0MP3RP4 contains the fix for this issue. Please contact Veritas Technical support for this Hot Fix.

Workaround for this issue :

/sbin/vxesd -k -x dmp_monitor_fabric=off 

Applies To

 OS Version : Solaris 10