Sign In
Problem
Unresponsive system (hang) or possible data loss due to an adverse interoperability issue between Qlogic 2GB and 4GB HBAs and Veritas Volume Manager versions: 4.1 MP1 with 122059-02, 4.1 MP2, 5.0, or 5.0 MP1Error Message
qlc: [ID 262021 kern.warning] WARNING: qlc(0): isr, Internal Parity/Pause Error - hccr=0h, stat=428113h, count=710644882-OR-
WARNING: /pci@3,700000/SUNW,qlc@0,1/fp@0,0/ssd@w50060e800327572c,5a (ssd138): undecodable sense information: 0x0 0x0 0x0 0x1 0x0 0x0 0x0 0x2 0xff 0xff 0xff 0xff 0x13 0x7c 0
Solution
This issue only applies if you have:
Qlogic 2G or 4G host bus adapters (HBAs) on Solaris 8, 9, or 10 with one of the following releases of Veritas Volume Manager (VxVM):
Veritas Volume Manager 4.1 MP1 with patch 122059-02
Veritas Volume Manager 4.1 MP2 or later
Veritas Volume Manager 5.0 or later
Detailed Description:
Due to an issue in DMP Fast Recovery procedures, interaction between Qlogic 2G and 4G HBAs and VxVM may cause Solaris systems to become unresponsive (hang) under heavy load conditions during dynamic multipathing (DMP) Fast Recovery IO failure analysis.
DMP Fast Recovery was introduced in the 5.0 release and back-ported to the 4.1 release through Patch 122059-02 as well as the Maintenance Patch (MP2) patchset (117080-07).
DMP Fast Recovery functionality greatly enhances IO failure analysis by communicating directly with the HBA driver, bypassing the SCSI disk (SD) driver which handles normal IO traffic. By communicating directly with the HBA, failure analysis can be conducted much more efficiently without suffering through backlogged SD driver queues that typically accompany IO path failures during heavy load.
Incident e1123248 documents two defects:
- Incorrect Command Descriptor Block (CDB) tagging.
- Failure to reset b_resid (number of bytes not transferred) back to zero upon subsequent attempts to resubmit a given IO.
The resulting behavior is HBA driver specific. Only Qlogic 2G or 4G HBAs have been found to exhibit this adverse behavior.
Resolution for 4.1 MP (x):
A binary hot fix is available for 4.1 MP2 to fix this issue. The 4.1 MP2 patch (117080-07) is a prerequisite for the binary hot fix.
A patchadd patch (128045-01) is available for VxVM 4.1 MP2 RP2. The 4.1 MP2 (117080-07) plus RP2 (124358-04) patches are prerequisites for this patchadd solution.
Please contact Symantec Support to obtain either of these patches, referencing this TechNote 292445.
Availability of the DMP_Fast_Recovery tunable on 4.1 MP2:
"The dmp_fast_recovery tunable controls whether DMP should attempt to obtain SCSI error information directly from the HBA interface. Setting the value to on can potentially provide faster error recovery, provided that the HBA interface supports the error enquiry feature. If set to off, the HBA interface is not used. The default setting is off. Before enabling this tunable, make sure the HBA firmware level is supported in the HCL. Enabling this tunable with unsupported HBA firmware levels may result in a system panic."
There are three discrepancies in that quote from the MP2 Release Notes:
- While the DMP Fast Recovery feature was included in 4.1 MP2, the dmp_fast_recovery tunable was not exposed.
- DMP Fast Recovery is "on" by default.
- The last two sentences referring to HBA firmware and risk of a system panic actually apply to the 'monitor_fabric' tunable. This tunable is 'off' by default in 4.1 MP2, specifically to protect users against those risks.
In addition to repairing the two defects outlined above, Incident e1123248 also exposes the dmp_fast_recovery tunable as documented in the 4.1 MP2 Release Notes. The 5.0 release does include this tunable by default. The default value for this tunable remains "on" for both releases.
Resolution for 5.x:
The next rolling patch for 5.0 MP1 will include a permanent fix for these issues. This patch is tentatively scheduled to be released by end of January (08). Until the release of this patch, you can disable DMP Fast Recovery as a temporary workaround as described below:
Workaround for 5.x:
1. Install VxVM 5.0 MP1:
https://www.veritas.com/support/en_US/article.TECH51515
2. Set dmp_fast_recovery=off:
root# vxdmpadm gettune all |grep fast_recovery
dmp_fast_recovery on on
root#
root# vxdmpadm settune dmp_fast_recovery=off
Tunable value will be changed immediately
root#
root# vxdmpadm gettune all |grep fast_recovery
dmp_fast_recovery off on
root#
root# cat /etc/vx/dmppolicy.info
arraytype
#
arrayname
#
enclosure
#
Tunables
dmp_fast_recovery=off
#
root#
Terms of use for this information are found in Legal Notices.
|
Translated Content
Please note that this document is a translation from English, and may have been machine-translated. It is possible that updates have been made to the original version after this document was translated and published. Veritas does not guarantee the accuracy regarding the completeness of the translation. You may also refer to the English Version of this knowledge base article for up-to-date information.
