NetBackup nbsu command on AIX platform core dumps with segmentation fault in extend_brk.

Problem

NetBackup nbsu command on AIX platform core dumps with segmentation fault in extend_brk.

Error Message

As seen in the Core stack below, the nbsu command failed after the nbemmcmd -listmedia -mergetable:

Segmentation fault in extend_brk at 0xd0110c78 ($t1)
0xd0110c78 (extend_brk+0x238) 90040004         stw   r0,0x4(r4)
(dbx) thread
thread  state-k     wchan    state-u    k-tid   mode held scope function
>$t1     run                  running  6750541     k   no   sys
(dbx) thread current
thread  state-k     wchan    state-u    k-tid   mode held scope function
>$t1     run                  running  6750541     k   no   sys
(dbx) where
libdebug assertion "(framep->getGpr(STKP, &addr) == DB_SUCCESS && *nextStkpp == addr)" failed at line 1299 in file ../../../../../../../../../../../src/bos/usr/ccs/lib/libdbx/libdebug/modules/stackdebug/POWER/stackdb_FrameProgress.C
extend_brk(internal error: assertion failed at line 3552 in file frame.c
??, internal error: assertion failed at line 3552 in file frame.c
??, internal error: assertion failed at line 3552 in file frame.c
??) at 0xd0110c78

Cause

The extend_brk is called to extend the address or heap space. This is related to AIX address space. For 32bit user land, binary default address space is 256MB, if memory usage for process grows beyond 256MB then new allocations can cause the process to core dump.

The core file size in this case is 256.9 MB (> 256MB), similar to above issue. bash-4.1# ls -l core-rw------- 1 root system 269409340 Jan 25 14:38 core
The nbsu binary has maxdata set to default 1 segment.
In this case it appears due to large number/size of data it may be requiring more memory. bash-4.1# dump -ov /usr/openv/netbackup/bin/support/nbsu /usr/openv/netbackup/bin/support/nbsu: ***Object Module Header***# Sections Symbol Ptr # Symbols Opt Hdr Len Flags 5 0x00000000 0 72 0x100fFlags=( RELFLG EXEC LNNO LSYMS DYNLOAD DEP_SYSTEM )Timestamp = "Apr 11 21:18:10 2006"Magic = 0x1df (32-bit XCOFF) ***Optional Header***Tsize Dsize Bsize Tstart Dstart0x0010f97d 0x00014793 0x00003fa8 0x10000150 0x20000acd SNloader SNentry SNtext SNtoc SNdata0x0004 0x0002 0x0001 0x0002 0x0002 TXTalign DATAalign TOC vstamp entry0x0005 0x0003 0x200123bc 0x0001 0x2000d380 maxSTACK maxDATA SNbss magic modtype0x00000000 0x00000000 0x0003 0x010b 1L <<<<< max data default 1 segment

Solution

Set the maxdata size for nbsu binary to 8 segments by following the below steps:

1.  Ensure no nbsu command is running

2.  Save a copy of existing nbsu binary
# cp -p /usr/openv/netbackup/bin/support/nbsu /usr/openv/netbackup/bin/support/nbsu.bk

3. Set the maxdata size to 8 segments for nbsu binary
# /usr/ccs/bin/ldedit -bmaxdata:0x80000000 /usr/openv/netbackup/bin/support/nbsu
/usr/ccs/bin/ldedit: File /usr/openv/netbackup/bin/support/nbsu updated.

4. Verify that the new maxdata value is set
# dump -ov /usr/openv/netbackup/bin/support/nbsu /usr/openv/netbackup/bin/support/nbsu: ***Object Module Header***# Sections Symbol Ptr # Symbols Opt Hdr Len Flags 5 0x00000000 0 72 0x100fFlags=( RELFLG EXEC LNNO LSYMS DYNLOAD DEP_SYSTEM )Timestamp = "Apr 11 21:18:10 2006"Magic = 0x1df (32-bit XCOFF) ***Optional Header***Tsize Dsize Bsize Tstart Dstart0x0010f97d 0x00014793 0x00003fa8 0x10000150 0x20000acd SNloader SNentry SNtext SNtoc SNdata0x0004 0x0002 0x0001 0x0002 0x0002 TXTalign DATAalign TOC vstamp entry0x0005 0x0003 0x200123bc 0x0001 0x2000d380 maxSTACK maxDATA SNbss magic modtype0x00000000 0x80000000 0x0003 0x010b 1L <<<<< New value
5. Perform the nbsu command again.