Sign In
Problem
Unresponsive system (hang) or possible data loss due to an adverse interoperability issue between Qlogic 2GB and 4GB HBAs and Veritas Volume Manager versions: 4.1 MP1 with 122059-02, 4.1 MP2, 5.0, or 5.0 MP1Error Message
qlc: [ID 262021 kern.warning] WARNING: qlc(0): isr, Internal Parity/Pause Error - hccr=0h, stat=428113h, count=710644882-OR-
WARNING: /pci@3,700000/SUNW,qlc@0,1/fp@0,0/ssd@w50060e800327572c,5a (ssd138): undecodable sense information: 0x0 0x0 0x0 0x1 0x0 0x0 0x0 0x2 0xff 0xff 0xff 0xff 0x13 0x7c 0
Solution
This issue
only applies if you have:
Qlogic 2G or 4G
host bus adapters (HBAs) on Solaris 8, 9, or 10 with one of the following
releases of Veritas Volume Manager (VxVM):
Veritas Volume
Manager 4.1 MP1 with patch 122059-02
Veritas Volume
Manager 4.1 MP2 or later
Veritas Volume
Manager 5.0 or later
Detailed
Description:
Due to an issue
in DMP Fast Recovery procedures, interaction between Qlogic 2G and 4G HBAs
and VxVM may cause Solaris systems to become unresponsive (hang) under heavy
load conditions during dynamic multipathing (DMP) Fast Recovery IO failure
analysis.
DMP Fast
Recovery was introduced in the 5.0 release and back-ported to the 4.1 release
through Patch 122059-02 as well as the Maintenance Patch (MP2) patchset
(117080-07).
DMP Fast
Recovery functionality greatly enhances IO failure analysis by communicating
directly with the HBA driver, bypassing the SCSI disk (SD) driver which handles
normal IO traffic. By communicating directly with the HBA, failure
analysis can be conducted much more efficiently without suffering through
backlogged SD driver queues that typically accompany IO path failures during
heavy load.
Incident
e1123248 documents two defects:
- Incorrect Command Descriptor Block (CDB) tagging.
- Failure to reset b_resid (number of bytes not transferred) back to zero upon subsequent attempts to resubmit a given IO.
The
resulting behavior is HBA driver specific. Only Qlogic 2G or 4G HBAs have
been found to exhibit this adverse behavior.
Resolution
for 4.1 MP (x):
A binary hot
fix is available for 4.1 MP2 to fix this issue. The 4.1 MP2 patch
(117080-07) is a prerequisite for the binary hot fix.
A patchadd
patch (128045-01) is available for VxVM 4.1 MP2 RP2. The 4.1 MP2
(117080-07) plus RP2 (124358-04) patches are prerequisites for this patchadd
solution.
Please contact
Symantec Support to obtain either of these patches, referencing this TechNote
292445.
Availability of the DMP_Fast_Recovery tunable on 4.1
MP2:
The 4.1 MP2
Release
Notes http://support.veritas.com/docs/287682
describe a tunable to disable the DMP Fast Recovery functionality:
"The
dmp_fast_recovery tunable controls whether DMP should attempt to obtain SCSI
error information directly from the HBA interface. Setting the value to on can
potentially provide faster error recovery, provided that the HBA interface
supports the error enquiry feature. If set to off, the HBA interface is not
used. The default setting is off. Before enabling this tunable, make sure the
HBA firmware level is supported in the HCL. Enabling this tunable with
unsupported HBA firmware levels may result in a system panic."
There are three
discrepancies in that quote from the MP2 Release Notes:
- While the DMP Fast Recovery feature was included in 4.1 MP2, the dmp_fast_recovery tunable was not exposed.
- DMP Fast Recovery is "on" by default.
- The last two sentences referring to HBA firmware and risk of a system panic actually apply to the 'monitor_fabric' tunable. This tunable is 'off' by default in 4.1 MP2, specifically to protect users against those risks.
In
addition to repairing the two defects outlined above, Incident e1123248 also
exposes the dmp_fast_recovery tunable as documented in the 4.1 MP2 Release
Notes. The 5.0 release does include this tunable by default. The default value
for this tunable remains "on" for both releases.
Resolution for 5.x:
The next rolling patch for 5.0 MP1 will include a permanent fix for these issues. This patch is tentatively scheduled to be released by end of January (08). Until the release of this patch, you can disable DMP Fast Recovery as a temporary workaround as described below:
Workaround for 5.x:
1. Install VxVM 5.0 MP1:
http://support.veritas.com/docs/288505
2. Set dmp_fast_recovery=off:
root# vxdmpadm gettune all |grep fast_recovery
dmp_fast_recovery on on
root#
root# vxdmpadm settune dmp_fast_recovery=off
Tunable value will be changed immediately
root#
root# vxdmpadm gettune all |grep fast_recovery
dmp_fast_recovery off on
root#
root# cat /etc/vx/dmppolicy.info
arraytype
#
arrayname
#
enclosure
#
Tunables
dmp_fast_recovery=off
#
root#
Terms of use for this information are found in Legal Notices.
