System panic with a redzone violation while releasing vxfs inodes File Input/Output (FIO) statistics structure.
Panic stack shows:
CPU 12 CSA F1000815B0337D00 at time of crash, error code for LEDs: 70000000
[005098A4]slock+000524 (??, ??)
[00176F54]internal_iodone_offl+000174 (??, ??)
[04D3B124]kaio_rdwr+000184 (??, ??, ??, ??)
56)> dr r3
r3 : EEEE0000DA278030 <== ESYSERROR_SL_RECURLOCK
(0)> dd vx_fiostats_tunable
vx_fiostats_tunable+000000: 0000000A <== Note the value of "0000000A" when enabled
This issue is tracked via etrack # 2726056.
Different types of statistics are maintained when a file is accessed in Quick Input/Output (QIO) and non-QIO mode. Some common statistics are copied when the file access mode is changed from QIO to non-QIO or vice versa. While switching from QIO mode to non-QIO, the QIO statistics structure is freed and FIO statistics structure is allocated to maintain FIO file-level statistics. There is a race between the thread freeing the QIO statistics which also allocates the FIO statistics and the thread updating the QIO statistics when the file is opened in QIO mode. Thus, the FIO statistics gets corrupted as another thread writes to it assuming that the QIO statistics is allocated.
Symantec has made code changes to protect the allocation/releasing of FIO/QIO statistics using the read-write lock/spin lock for file statistics structure. This change is available from 5.1SP1RP3 patch which is released in October 2012.
Please follow the patch README file for installation details.
Unit the patch is installed, please follow the following workaround to disable FIO stat gathering.
Workaround for AIX
On AIX the vx_fiostats_tunable has to be tuned above 100 to turn off fiostat gathering. Recommendation is this tunable to be tuned to 1000.
So to disable, update /etc/vx/vxfssystem with the below line. If this file doesn't exist then please create one.
Please note a reboot is required for this change to take effect.
Patch links for 5.1SP1RP3
This issue is applicable to systems running:
-AIX, HP-UX, Linux and Solaris
-VxFS 5.1SP1 and above