Problem
System panic with a redzone violation while releasing vxfs inodes File Input/Output (FIO) statistics structure.
Error Message
Panic stack shows:
CRASH INFORMATION:
CPU 12 CSA F1000815B0337D00 at time of crash, error code for LEDs: 70000000
pvthread+06E800 STACK:
[0001BF00]abend_trap+000000 ()
[005098A4]slock+000524 (??, ??)
[00009558].simple_lock+000058 ()
[04AD7470].vx_aio_iodone+000200 ()
[00176F54]internal_iodone_offl+000174 (??, ??)
[00176DA0]iodone_offl+000080 ()
[0017A2B4]i_softmod+000294 ()
[001A5DE4]flih_util+000250 ()
[...]
[049E3280].vx_qiostats_update+000074 ()
[049E4310].vx_fiostats_update@AF48_34+000020 ()
[049E39E4].vx_fiostats_update+00001C ()
[04C93A70].fdd_rw+000790 ()
[04C91C54].fddstrategy1+0005D4 ()
[04C91E1C].fddstrategy+000100 ()
[04C9064C].fddstrategy_skey+000024 ()
[002D3BA4]std_devstrat+000364 (??)
[002D41D0]devstrat+000050 (??)
[00014F50].kernel_add_gate_cstack+000030 ()
[04D3C9C4]devstrat_lio@AF64_18+0000A4 (??)
[04D37574]devstrat_lio+000014 (??)
[04D3B124]kaio_rdwr+000184 (??, ??, ??, ??)
[00003850]ovlya_addr_sc_flih_main+000130 ()
56)> dr r3
r3 : EEEE0000DA278030 <== ESYSERROR_SL_RECURLOCK
(56)>
(0)> dd vx_fiostats_tunable
vx_fiostats_tunable+000000: 0000000A <== Note the value of "0000000A" when enabled
Cause
This issue is tracked via etrack # 2726056.
Different types of statistics are maintained when a file is accessed in Quick Input/Output (QIO) and non-QIO mode. Some common statistics are copied when the file access mode is changed from QIO to non-QIO or vice versa. While switching from QIO mode to non-QIO, the QIO statistics structure is freed and FIO statistics structure is allocated to maintain FIO file-level statistics. There is a race between the thread freeing the QIO statistics which also allocates the FIO statistics and the thread updating the QIO statistics when the file is opened in QIO mode. Thus, the FIO statistics gets corrupted as another thread writes to it assuming that the QIO statistics is allocated.
Solution
Veritas has made code changes to protect the allocation/releasing of FIO/QIO statistics using the read-write lock/spin lock for file statistics structure. This change is available from 5.1SP1RP3 patch which is released in October 2012.
Please follow the patch README file for installation details.
Unit the patch is installed, please follow the following workaround to disable FIO stat gathering.
Workaround for AIX
===============
On AIX the vx_fiostats_tunable has to be tuned above 100 to turn off fiostat gathering. Recommendation is this tunable to be tuned to 1000.
So to disable, update /etc/vx/vxfssystem with the below line. If this file doesn't exist then please create one.
vx_fiostats_tunable 1000
Please note a reboot is required for this change to take effect.
Patch links for 5.1SP1RP3
AIX
https://sort.Veritas.com/patch/detail/6806
AIX 7.1
https://sort.Veritas.com/patch/detail/6807
Solaris SPARC
https://sort.Veritas.com/patch/detail/6816
https://sort.Veritas.com/patch/detail/6817
solaris x64
https://sort.Veritas.com/patch/detail/6818
https://sort.Veritas.com/patch/detail/6819
RHEL5 x86_64
https://sort.Veritas.com/patch/detail/6808
https://sort.Veritas.com/patch/detail/6809
RHEL6 x86_64
https://sort.Veritas.com/patch/detail/6814
https://sort.Veritas.com/patch/detail/6815
SLES10 x86_64
https://sort.Veritas.com/patch/detail/6811
SLES11 x86_64
https://sort.Veritas.com/patch/detail/6813
Applies To
This issue is applicable to systems running:
-AIX, HP-UX, Linux and Solaris
-VxFS 5.1SP1 and above