Intermittent failures and possible system crashes are known to occur when using Storage Foundation for UNIX, (SF/HA) Veritas Cluster Server, (VCS) Storage Foundation for Oracle RAC, (SFRAC) and Storage Foundation Cluster File Server (SFCFS) with certain AIX platforms.
Problem: System Crashes/Hangs in one of the drivers of the cluster solutions
System crash/hang will occur with any version of clustering products based on 4.0 or 5.0 releases. The Service Pack releases of AIX with the patch levels specified in the following table are the levels of AIX which result in the crash.
|O/S Level||Storage Foundation||Veritas Cluster Server||Storage Foundation High Availability||Storage Foundation Cluster File Server||Storage Foundation Oracle RAC|
|AIX 5.2 TL10 SP5||NO||YES||YES||YES||YES|
|AIX 5.3 TL6 SP3||NO||YES||YES||YES||YES|
|AIX 5.3 TL7 SP2||NO||YES||YES||YES||YES|
|AIX 6.1 Base SP3||NO||YES||YES||YES||YES|
If your system is upgraded to or installed at the specified levels or above, you are impacted. You must follow the instructions provided below to recover. The specific IBM AIX APAR triggering this LLT panic has the abstract: Bring DLPI driver "to SPEC" applies to AIX - enhance DLPI driver to comply with specifications or normal streams processing. This APAR is installed as:
|AIX OS Level||AIX APAR|
Problem and fix description
Changes to the DLPI driver with the latest TLs on AIX 5.3 and 6.1 result in upcalls to LLT in the interrupt context instead of the process context. Some LLT locks are not interrupt safe which caused panics. Some upcalls panic in GAB where it does an xmalloc in interrupt context.
All LLT locks on AIX are now interrupt safe. Out-of order lock releases are handled correctly with the new interrupt safe locks. Link notifications are provided for the client in the thread or process context instead of the interrupt context. This will ensure that no changes are required for LLT's clients like GAB/LMX thus abstracting all the changes to just one driver.
Impacted releases and corresponding LLT patches
The corresponding LLT patches are found at Patch Central below and under the Related Documents section.
4.0 MP3 https://sort.symantec.com/patch/detail/2106
5.0 MP1 and 5.0 MP1 Update 1 https://sort.symantec.com/patch/detail/2103
Recommended procedure for an existing setup with panics
- Install and configure the appropriate patch for your MP
Recommended procedure for an existing setup without panics
- Install and configure the patch for the appropriate release
(follow directions in the README)
- Upgrade to the latest AIX release when ready.
Recommended procedure for a fresh install
- Use installonly to install the base release
- Upgrade to the latest MP
- Install the required llt patch, then reboot the system.
- Use the configure option to configure the cluster-server.
NOTE# The removal of IBM temporary patch. (Workaround)
Once the LLT patch is installed as above, the workaround patch can be uninstalled, if it was installed.
To uninstall the workaround patch:
For AIX 6.1:
# emgr -rL 61TL00dlpi
For AIX 5.3 TL6:
# emgr -rL 53TL06dlpi
For AIX 5.3 TL7:
# emgr -rL 53TL07dlpi