System panic in vxfs code vx_qiostats_update()

Problem

System panic with a redzone violation while releasing vxfs inodes File Input/Output (FIO) statistics structure.

Error Message

Panic stack shows:

CRASH INFORMATION:
CPU 12 CSA F1000815B0337D00 at time of crash, error code for LEDs: 70000000
pvthread+06E800 STACK:
[0001BF00]abend_trap+000000 ()
[005098A4]slock+000524 (??, ??)
[00009558].simple_lock+000058 ()
[04AD7470].vx_aio_iodone+000200 ()
[00176F54]internal_iodone_offl+000174 (??, ??)
[00176DA0]iodone_offl+000080 ()
[0017A2B4]i_softmod+000294 ()
[001A5DE4]flih_util+000250 ()
[...]
[049E3280].vx_qiostats_update+000074 ()
[049E4310].vx_fiostats_update@AF48_34+000020 ()
[049E39E4].vx_fiostats_update+00001C ()
[04C93A70].fdd_rw+000790 ()
[04C91C54].fddstrategy1+0005D4 ()
[04C91E1C].fddstrategy+000100 ()
[04C9064C].fddstrategy_skey+000024 ()
[002D3BA4]std_devstrat+000364 (??)
[002D41D0]devstrat+000050 (??)
[00014F50].kernel_add_gate_cstack+000030 ()
[04D3C9C4]devstrat_lio@AF64_18+0000A4 (??)
[04D37574]devstrat_lio+000014 (??)
[04D3B124]kaio_rdwr+000184 (??, ??, ??, ??)
[00003850]ovlya_addr_sc_flih_main+000130 ()
 

56)> dr r3
r3 : EEEE0000DA278030            <== ESYSERROR_SL_RECURLOCK
(56)>
 
(0)> dd vx_fiostats_tunable
vx_fiostats_tunable+000000: 0000000A          <== Note the value of "0000000A" when enabled

Cause

This issue is tracked via etrack # 2726056.

Different types of statistics are maintained when a file is accessed in Quick Input/Output (QIO) and non-QIO mode. Some common statistics are copied when the file access mode is changed from QIO to non-QIO or vice versa. While switching from QIO mode to non-QIO, the QIO statistics structure is freed and FIO statistics structure is allocated to maintain FIO file-level statistics. There is a race between the thread freeing the QIO statistics which also allocates the FIO statistics and the thread updating the QIO statistics when the file is opened in QIO mode. Thus, the FIO statistics gets corrupted as another thread writes to it assuming that the QIO statistics is allocated.
 

Solution

Veritas has made code changes to protect the allocation/releasing of FIO/QIO statistics using the read-write lock/spin lock for file statistics structure. This change is available from 5.1SP1RP3 patch which is released in October 2012.

Please follow the patch README file for installation details.

Unit the patch is installed, please follow the following workaround to disable FIO stat gathering.

Workaround for AIX
===============

On AIX the vx_fiostats_tunable has to be tuned above 100 to turn off fiostat gathering. Recommendation is this tunable to be tuned to 1000.
So to disable, update /etc/vx/vxfssystem  with the below line. If this file doesn't exist then please create one.

vx_fiostats_tunable 1000  

Please note a reboot is required for this change to take effect. 
 

Patch links for 5.1SP1RP3

AIX

https://sort.Veritas.com/patch/detail/6806

AIX 7.1

https://sort.Veritas.com/patch/detail/6807

Solaris SPARC

https://sort.Veritas.com/patch/detail/6816

https://sort.Veritas.com/patch/detail/6817

solaris x64

https://sort.Veritas.com/patch/detail/6818

https://sort.Veritas.com/patch/detail/6819

RHEL5 x86_64

https://sort.Veritas.com/patch/detail/6808

https://sort.Veritas.com/patch/detail/6809

RHEL6 x86_64

https://sort.Veritas.com/patch/detail/6814

https://sort.Veritas.com/patch/detail/6815

SLES10 x86_64

https://sort.Veritas.com/patch/detail/6811

SLES11 x86_64

https://sort.Veritas.com/patch/detail/6813
 

Applies To

This issue is applicable to systems running:

-AIX, HP-UX, Linux and Solaris
-VxFS 5.1SP1 and above