Veritas Cluster Server (VCS) Asynchronous Monitoring Framework (AMF) causing "ps" commands to hang after upgrading SuSE Linux Enterprise Server (SLES) kernel to a certain patch level

Problem

AMF causing "ps" command to hang after upgrading SLES kernel to a certain patch level.   Commands other than ps which accesses the /proc filesystem will also be affected.

Error Message

Programs trying to retrieve the process information from the /proc filesystem will hang.    From the kernel core, those programs are in uninterruptible state.

  2287      1   0  ffff8104629f77e0  UN   0.0    3564    924  ps          <<< "UN" - UNinterruptible
  2590   2528   0  ffff810432bf17a0  UN   0.0    3568    936  ps
  6423   2824   2  ffff810431cc5080  UN   0.0    3564    936  ps

The following is a sample thread stack of the above processes.

PID: 11951  TASK: ffff8106678c80c0  CPU: 3   COMMAND: "ps"
#0 [ffff810638171d08] schedule at ffffffff802ea7f8
#1 [ffff810638171de0] __down_read at ffffffff802eb937
#2 [ffff810638171e20] access_process_vm at ffffffff8013c3b0
#3 [ffff810638171e90] proc_pid_cmdline at ffffffff801bcff1
#4 [ffff810638171ed0] proc_info_read at ffffffff801bd8af
#5 [ffff810638171f10] vfs_read at ffffffff80186ba0
#6 [ffff810638171f40] sys_read at ffffffff80186f80
#7 [ffff810638171f80] system_call at ffffffff8010ae16
 

Cause

Originally the AMF kernel module was coded for SLES kernel without the above mentioned kernel change.  After the kernel change was introduced, AMF does not work correctly.    

Solution

A hotfix is now available for this issue in the current version(s) of the product(s) mentioned. Refer to the Hotfix link below to obtain the hotfix needed to resolve the issue.

SFHA-SLES10_x86_64-5.1SP1RP2 :https://sort.Veritas.com//patch/detail/5514

VCS-SLES10_x86_64-VRTSamf-5.1SP1RP1P1 https://sort.Veritas.com/patch/detail/4964

The fix is also included in the Storage Foundation High Availability 5.1SP1 RP3.   

Note: After the patch is installed, AMF will work correctly with the following kernel versions.

SLES 10 SP2GA kernel version 2.6.16.60-0.42.11 and above
SLES 10 SP3GA kernel version 2.6.16.60-0.74.7 and above