System panic in vxfs code vx_qiostats_update()

  • Article ID:100027629
  • Modified Date:
  • Product(s):


System panic with a redzone violation while releasing vxfs inodes File Input/Output (FIO) statistics structure.

Error Message

Panic stack shows:

CPU 12 CSA F1000815B0337D00 at time of crash, error code for LEDs: 70000000
pvthread+06E800 STACK:
[0001BF00]abend_trap+000000 ()
[005098A4]slock+000524 (??, ??)
[00009558].simple_lock+000058 ()
[04AD7470].vx_aio_iodone+000200 ()
[00176F54]internal_iodone_offl+000174 (??, ??)
[00176DA0]iodone_offl+000080 ()
[0017A2B4]i_softmod+000294 ()
[001A5DE4]flih_util+000250 ()
[049E3280].vx_qiostats_update+000074 ()
[049E4310].vx_fiostats_update@AF48_34+000020 ()
[049E39E4].vx_fiostats_update+00001C ()
[04C93A70].fdd_rw+000790 ()
[04C91C54].fddstrategy1+0005D4 ()
[04C91E1C].fddstrategy+000100 ()
[04C9064C].fddstrategy_skey+000024 ()
[002D3BA4]std_devstrat+000364 (??)
[002D41D0]devstrat+000050 (??)
[00014F50].kernel_add_gate_cstack+000030 ()
[04D3C9C4]devstrat_lio@AF64_18+0000A4 (??)
[04D37574]devstrat_lio+000014 (??)
[04D3B124]kaio_rdwr+000184 (??, ??, ??, ??)
[00003850]ovlya_addr_sc_flih_main+000130 ()

56)> dr r3
r3 : EEEE0000DA278030            <== ESYSERROR_SL_RECURLOCK
(0)> dd vx_fiostats_tunable
vx_fiostats_tunable+000000: 0000000A          <== Note the value of "0000000A" when enabled


This issue is tracked via etrack # 2726056.

Different types of statistics are maintained when a file is accessed in Quick Input/Output (QIO) and non-QIO mode. Some common statistics are copied when the file access mode is changed from QIO to non-QIO or vice versa. While switching from QIO mode to non-QIO, the QIO statistics structure is freed and FIO statistics structure is allocated to maintain FIO file-level statistics. There is a race between the thread freeing the QIO statistics which also allocates the FIO statistics and the thread updating the QIO statistics when the file is opened in QIO mode. Thus, the FIO statistics gets corrupted as another thread writes to it assuming that the QIO statistics is allocated.


Veritas has made code changes to protect the allocation/releasing of FIO/QIO statistics using the read-write lock/spin lock for file statistics structure. This change is available from 5.1SP1RP3 patch which is released in October 2012.

Please follow the patch README file for installation details.

Unit the patch is installed, please follow the following workaround to disable FIO stat gathering.

Workaround for AIX

On AIX the vx_fiostats_tunable has to be tuned above 100 to turn off fiostat gathering. Recommendation is this tunable to be tuned to 1000.
So to disable, update /etc/vx/vxfssystem  with the below line. If this file doesn't exist then please create one.

vx_fiostats_tunable 1000  

Please note a reboot is required for this change to take effect. 

Patch links for 5.1SP1RP3


AIX 7.1

Solaris SPARC

solaris x64

RHEL5 x86_64

RHEL6 x86_64

SLES10 x86_64

SLES11 x86_64

Applies To

This issue is applicable to systems running:

-AIX, HP-UX, Linux and Solaris
-VxFS 5.1SP1 and above

Was this content helpful?

Get Support