1. Support
  2. Knowledge base
  3. 100005983

How to collect data related to usage of VxVM, VxFS, & VCS in general

Article: 100005983
Last Published: 2023-11-17
Ratings: 1 1
Product(s): Data Insight

Description

List of the steps for collecting data in the perspective of VxVM, VxFS, & VCS in general.

1) Collect date and time as soon as problem is observed.

2) Check what all operations are going on when performance degradation is observed, check if VERITAS operations are going on (look for VxVM/VxFS/VCS command if running).
 
3) Collect all the process currently running.
 
    # time ps -elf
   
4) Collect kernel threadlist every 2 minutes, when performance degradation is observed. Threadlist will be logged into /var/log/messages file.
 
  To avoid a system hang (it can cause problems with VCS/CFS) while collecting kernel threadlist using the Linux sysrq interface follow these steps.
 
   # cat /proc/sys/kernel/printk > /tmp/printk
   # echo 0 > /proc/sys/kernel/printk
   # echo t > /proc/sysrq-trigger
   # cat /tmp/printk > /proc/sys/kernel/printk
 
5) Collect top, meminfo and vmstat command output at regular interval (top 1 sec, vmstat 5 sec).
 
In one session watch continuously the top 20 process:
 
  # watch -n 1 "top -b -d 1 | head -n 20"
  # top –b –d 1        /* 1 sec interval */
 
If interested to look at any specific process (like vxconfigd) then use the following command
 
   # top -b -d 5 -p `ps -ef |grep vxconfigd |grep -v grep|awk '{print $2}'`
   # vmstat 5      /* 5 seconds interval */
   # while true
    > do
    > date
    > cat /proc/meminfo | egrep 'MemTotal|MemFree|Buffers|Dirty'
    > sleep 2
    > done
 
  6) Collect VxVM i/o statistics if there is an application i/o load.
 
   # vxstat -i 2 –vps
 
7) Collect the output from the following commands at regular intervals (10 seconds) in case if any VxVM command is taking time.
    
  # pmap `ps aux | grep  vxconfigd | grep -v grep | awk '{print $2}'`  | grep -i total
  # cat /proc/`pgrep vxconfigd`/status
  # /usr/sbin/lsof -a -p `pgrep vxconfigd`
  # pstack `pgrep vxconfigd`
 
Collect the stack of vxconfigd using ' gdb' if pstack command doesn’t work.
 
    # gdb -p `pgrep vxconfigd`
    (gdb) bt
    (gdb) info threads
    (gdb) info sharedlibrary
    (gdb) info dcache
    (gdb) quit
 
8) Please run the following command only if any VxVM commands are not working as expected or any VxVM/vxconfigd related issue is observed. Running vxconfigd in debug mode at
   level 9 can slow down the vxconfigd.
 
   How to start vxconfigd in debug mode at various labels (min 1, max 9)
   
    # vxdctl debug <label> /var/tmp/vxconfigd_debug.<label>.log
    
      Once done, disable the logging using following command.
  
     # vxdctl debug 0
  
    Generate a gcore of vxconfigd when high memory usage is observed:
 
     # gcore -o /usr/tmp/vxconfigd.core `pgrep vxconfigd`
 
VxFS specific commands (9,10) :
--------------------------------
 
9) Run the following command on  VxFS mount points and get the "statfile" for each mount point.
   
    # vxfsstat -w "statfile" -t 10 -c 2 /mount_point
 
10) If you experience any performance issues with a VxFS files system, please collect the metasave of the VxFS filesystem in question.
 
   # cd /opt/VRTSspt/FS/MetaSave/
   # ls
      metasave_rhel4_x86     metasave_sles9_x86     README.metasave
     metasave_rhel4_x86_64  metasave_sles9_x86_64
 
     Please follow the steps in README.metasave.
  ---------------------------------------------------------------------------
 
11) Please collect the output of FirstLook
 
    Please see the article for FirstLook (Please skip downloading it):  
 
    To execute FirstLook, invoke start_look. For example, executing FirstLook on Solaris:
 
     # cd /opt/VRTSspt/FirstLook
   # ./start_look -l
 
    Note:
    1) Always use the -l option to get lockstat data unless system is VERY heavily loaded.
    2) Note - do not run more than one hour, the logs will overwrite after 60 minutes.
    3) Try to start the script 5 minutes before the performance issue is expected to be seen.
    4) For example, files will be saved in /opt/VRTSspt/FirstLook/Flook_Apr_8_11:58:00_logs.
 
    Optionally, it is possible to confirm that the FirstLook process is running:
 
    # ps -ef | grep look
       root 19984 1 0 08:58:19 pts/1 0:00 /bin/ksh /vx_install/FirstLook/Flook_sol
 
    After enough time to collect useful information, or if the performance issue is intermittent and has stopped, stop FirstLook:
 
   # ./stop_look
 
    Please compress the output using gzip, e.g., '/opt/VRTSspt/FirstLook/Flook_04_08_11_12:00:07_logs.tar'
 
12) Please collect the output of new version of VRTSexplorer from the affected node, in question:
 
  # cd /opt/VRTSspt/VRTSexplorer
  # ./VRTSexplorer
 
13) Please collect the output of VRTSexplorer from all other nodes in  the VCS cluster.
 
14) Collect system dump from all nodes in the cluster around the same time. Please set peerinact timeout as shown in the article 100029707.
 
 

 

Was this content helpful?

Article Languages