Veritas InfoScale™ 8.0 Troubleshooting Guide - AIX

Last Published:
Product(s): InfoScale & Storage Foundation (8.0)
Platform: AIX
  1. Introduction
    1.  
      About troubleshooting Veritas InfoScale Storage Foundation and High Availability Solutions products
    2.  
      About Veritas Services and Operations Readiness Tools (SORT)
    3. About unique message identifiers
      1.  
        Using Veritas Operations Readiness Tools to find a Unique Message Identifier description and solution
    4. About collecting application and daemon core data for debugging
      1.  
        Letting vxgetcore find debugging data automatically (the easiest method)
      2.  
        Running vxgetcore when you know the location of the core file
      3.  
        Letting vxgetcore prompt you for information
  2. Section I. Troubleshooting Veritas File System
    1. Diagnostic messages
      1. File system response to problems
        1.  
          Recovering a disabled file system
      2.  
        About kernel messages
  3. Section II. Troubleshooting Veritas Volume Manager
    1. Recovering from hardware failure
      1.  
        About recovery from hardware failure
      2.  
        Listing unstartable volumes
      3.  
        Displaying volume and plex states
      4.  
        The plex state cycle
      5.  
        Recovering an unstartable mirrored volume
      6.  
        Recovering an unstartable volume with a disabled plex in the RECOVER state
      7.  
        Forcibly restarting a disabled volume
      8.  
        Clearing the failing flag on a disk
      9.  
        Reattaching failed disks
      10.  
        Recovering from a failed plex attach or synchronization operation
      11. Failures on RAID-5 volumes
        1.  
          System failures
        2.  
          Disk failures
        3.  
          Default startup recovery process for RAID-5
        4. Recovery of RAID-5 volumes
          1.  
            Resynchronizing parity on a RAID-5 volume
          2.  
            Reattaching a failed RAID-5 log plex
          3.  
            Recovering a stale subdisk in a RAID-5 volume
        5.  
          Recovery after moving RAID-5 subdisks
        6. Unstartable RAID-5 volumes
          1.  
            Forcibly starting a RAID-5 volume with stale subdisks
      12.  
        Recovering from an incomplete disk group move
      13.  
        Restarting volumes after recovery when some nodes in the cluster become unavailable
      14. Recovery from failure of a DCO volume
        1.  
          Recovering a version 0 DCO volume
        2.  
          Recovering an instant snap DCO volume (version 20 or later)
    2. Recovering from instant snapshot failure
      1.  
        Recovering from the failure of vxsnap prepare
      2.  
        Recovering from the failure of vxsnap make for full-sized instant snapshots
      3.  
        Recovering from the failure of vxsnap make for break-off instant snapshots
      4.  
        Recovering from the failure of vxsnap make for space-optimized instant snapshots
      5.  
        Recovering from the failure of vxsnap restore
      6.  
        Recovering from the failure of vxsnap refresh
      7.  
        Recovering from copy-on-write failure
      8.  
        Recovering from I/O errors during resynchronization
      9.  
        Recovering from I/O failure on a DCO volume
      10.  
        Recovering from failure of vxsnap upgrade of instant snap data change objects (DCOs)
    3. Recovering from failed vxresize operation
      1.  
        Recovering from a failed vxresize shrink operation
    4. Recovering from boot disk failure
      1.  
        VxVM and boot disk failure
      2. Recovery by reinstallation
        1.  
          General reinstallation information
        2. Reinstalling the system and recovering VxVM
          1.  
            Prepare the system for reinstallation
          2.  
            Reinstall the operating system
          3.  
            Reinstalling the Veritas Volume Manager
          4.  
            Recovering the Veritas Volume Manager configuration
          5.  
            Cleaning up the system configuration
    5. Managing commands, tasks, and transactions
      1.  
        Command logs
      2.  
        Task logs
      3.  
        Transaction logs
      4.  
        Association of command, task, and transaction logs
      5.  
        Associating CVM commands issued from slave to master node
      6.  
        Command completion is not enabled
    6. Backing up and restoring disk group configurations
      1.  
        About disk group configuration backup
      2.  
        Backing up a disk group configuration
      3. Restoring a disk group configuration
        1.  
          Resolving conflicting backups for a disk group
      4.  
        Backing up and restoring Flexible Storage Sharing disk group configuration data
    7. Troubleshooting issues with importing disk groups
      1.  
        Clearing the udid_mismatch flag for non-clone disks
    8. Recovering from CDS errors
      1.  
        CDS error codes and recovery actions
    9. Logging and error messages
      1.  
        About error messages
      2. How error messages are logged
        1.  
          Configuring logging in the startup script
      3. Types of messages
        1.  
          Messages
      4.  
        Collecting log information for troubleshooting
    10. Troubleshooting Veritas Volume Replicator
      1.  
        Recovery from RLINK connect problems
      2. Recovery from configuration errors
        1. Errors during an RLINK attach
          1.  
            Data volume errors during an RLINK attach
          2.  
            Volume set errors during an RLINK attach
        2. Errors during modification of an RVG
          1.  
            Missing data volume error during modifcation of an RVG
          2.  
            Data volume mismatch error during modification of an RVG
          3.  
            Data volume name mismatch error during modification of an RVG
          4. Volume set configuration errors during modification of an RVG
            1.  
              Volume set name mismatch error
            2.  
              Volume index mismatch error
            3.  
              Component volume mismatch error
      3. Recovery on the Primary or Secondary
        1.  
          About recovery from a Primary-host crash
        2. Recovering from Primary data volume error
          1.  
            Example - Recovery with detached RLINKs
          2.  
            Example - Recovery with minimal repair
          3.  
            Example - Recovery by migrating the primary
          4.  
            Example - Recovery from temporary I/O error
        3. Primary SRL volume error cleanup and restart
          1.  
            About RVG PASSTHRU mode
        4.  
          Primary SRL volume error at reboot
        5.  
          Primary SRL volume overflow recovery
        6. Primary SRL header error cleanup and recovery
          1.  
            Recovering from SRL header error
        7. Secondary data volume error cleanup and recovery
          1.  
            Recovery using a Secondary Storage Checkpoint
          2.  
            Cleanup using a Primary Storage Checkpoint
        8.  
          Secondary SRL volume error cleanup and recovery
        9.  
          Secondary SRL header error cleanup and recovery
        10.  
          Secondary SRL header error at reboot
  4. Section III. Troubleshooting Dynamic Multi-Pathing
    1. Dynamic Multi-Pathing troubleshooting
      1.  
        Displaying extended attributes after upgrading to DMP
      2.  
        Recovering from errors when you exclude or include paths to DMP
      3.  
        Downgrading the array support
  5. Section IV. Troubleshooting Storage Foundation Cluster File System High Availability
    1. Troubleshooting Storage Foundation Cluster File System High Availability
      1.  
        About troubleshooting Storage Foundation Cluster File System High Availability
      2. Troubleshooting CFS
        1.  
          Incorrect order in root user's <library> path
        2.  
          CFS commands might hang when run by a non-root user
      3. Troubleshooting fenced configurations
        1.  
          Example of a preexisting network partition (split-brain)
        2. Recovering from a preexisting network partition (split-brain)
          1.  
            Example Scenario I
          2.  
            Example Scenario II
          3.  
            Example Scenario III
      4. Troubleshooting Cluster Volume Manager in Veritas InfoScale products clusters
        1.  
          CVM group is not online after adding a node to the Veritas InfoScale products cluster
        2.  
          Shared disk group cannot be imported in Veritas InfoScale products cluster
        3.  
          Unable to start CVM in Veritas InfoScale products cluster
        4.  
          Removing preexisting keys
        5.  
          CVMVolDg not online even though CVMCluster is online in Veritas InfoScale products cluster
  6. Section V. Troubleshooting Cluster Server
    1. Troubleshooting and recovery for VCS
      1. VCS message logging
        1.  
          Log unification of VCS agent's entry points
        2.  
          Enhancing First Failure Data Capture (FFDC) to troubleshoot VCS resource's unexpected behavior
        3.  
          GAB message logging
        4.  
          Enabling debug logs for agents
        5.  
          Enabling debug logs for IMF
        6.  
          Enabling debug logs for the VCS engine
        7.  
          About debug log tags usage
        8. Gathering VCS information for support analysis
          1.  
            Verifying the metered or forecasted values for CPU, Mem, and Swap
        9.  
          Gathering LLT and GAB information for support analysis
        10.  
          Gathering IMF information for support analysis
        11.  
          Message catalogs
      2. Troubleshooting the VCS engine
        1.  
          HAD diagnostics
        2.  
          HAD restarts continuously
        3.  
          DNS configuration issues cause GAB to kill HAD
        4.  
          Seeding and I/O fencing
        5.  
          Preonline IP check
      3. Troubleshooting Low Latency Transport (LLT)
        1.  
          LLT startup script displays errors
        2.  
          LLT detects cross links usage
        3.  
          LLT link status messages
      4. Troubleshooting Group Membership Services/Atomic Broadcast (GAB)
        1.  
          GAB timer issues
        2.  
          Delay in port reopen
        3.  
          Node panics due to client process failure
      5. Troubleshooting VCS startup
        1.  
          "VCS: 10622 local configuration missing" and "VCS: 10623 local configuration invalid"
        2.  
          "VCS:11032 registration failed. Exiting"
        3.  
          "Waiting for cluster membership."
      6.  
        Troubleshooting Intelligent Monitoring Framework (IMF)
      7. Troubleshooting service groups
        1.  
          VCS does not automatically start service group
        2.  
          System is not in RUNNING state
        3.  
          Service group not configured to run on the system
        4.  
          Service group not configured to autostart
        5.  
          Service group is frozen
        6.  
          Failover service group is online on another system
        7.  
          A critical resource faulted
        8.  
          Service group autodisabled
        9.  
          Service group is waiting for the resource to be brought online/taken offline
        10.  
          Service group is waiting for a dependency to be met.
        11.  
          Service group not fully probed.
        12.  
          Service group does not fail over to the forecasted system
        13.  
          Service group does not fail over to the BiggestAvailable system even if FailOverPolicy is set to BiggestAvailable
        14.  
          Restoring metering database from backup taken by VCS
        15.  
          Initialization of metering database fails
      8. Troubleshooting resources
        1.  
          Service group brought online due to failover
        2.  
          Waiting for service group states
        3.  
          Waiting for child resources
        4.  
          Waiting for parent resources
        5.  
          Waiting for resource to respond
        6. Agent not running
          1.  
            Invalid agent argument list.
        7.  
          The Monitor entry point of the disk group agent returns ONLINE even if the disk group is disabled
      9. Troubleshooting I/O fencing
        1.  
          Node is unable to join cluster while another node is being ejected
        2.  
          The vxfentsthdw utility fails when SCSI TEST UNIT READY command fails
        3.  
          Manually removing existing keys from SCSI-3 disks
        4. System panics to prevent potential data corruption
          1.  
            How I/O fencing works in different event scenarios
        5.  
          Cluster ID on the I/O fencing key of coordinator disk does not match the local cluster's ID
        6. Fencing startup reports preexisting split-brain
          1.  
            Clearing preexisting split-brain condition
        7.  
          Registered keys are lost on the coordinator disks
        8.  
          Replacing defective disks when the cluster is offline
        9.  
          The vxfenswap utility exits if rcp or scp commands are not functional
        10. Troubleshooting CP server
          1.  
            Troubleshooting issues related to the CP server service group
          2.  
            Checking the connectivity of CP server
        11. Troubleshooting server-based fencing on the Veritas InfoScale products cluster nodes
          1.  
            Issues during fencing startup on VCS nodes set up for server-based fencing
        12. Issues during online migration of coordination points
          1.  
            Vxfen service group activity after issuing the vxfenswap command
      10. Troubleshooting notification
        1.  
          Notifier is configured but traps are not seen on SNMP console.
      11. Troubleshooting and recovery for global clusters
        1.  
          Disaster declaration
        2.  
          Lost heartbeats and the inquiry mechanism
        3. VCS alerts
          1.  
            Types of alerts
          2.  
            Managing alerts
          3.  
            Actions associated with alerts
          4.  
            Negating events
          5.  
            Concurrency violation at startup
      12.  
        Troubleshooting the steward process
      13. Troubleshooting licensing
        1.  
          Validating license keys
        2. Licensing error messages
          1.  
            [Licensing] Insufficient memory to perform operation
          2.  
            [Licensing] No valid VCS license keys were found
          3.  
            [Licensing] Unable to find a valid base VCS license key
          4.  
            [Licensing] License key cannot be used on this OS platform
          5.  
            [Licensing] VCS evaluation period has expired
          6.  
            [Licensing] License key can not be used on this system
          7.  
            [Licensing] Unable to initialize the licensing framework
          8.  
            [Licensing] QuickStart is not supported in this release
          9.  
            [Licensing] Your evaluation period for the feature has expired. This feature will not be enabled the next time VCS starts
      14.  
        Verifying the metered or forecasted values for CPU, Mem, and Swap
  7. Section VI. Troubleshooting SFDB
    1. Troubleshooting SFDB
      1.  
        About troubleshooting Storage Foundation for Databases (SFDB) tools

Service group does not fail over to the BiggestAvailable system even if FailOverPolicy is set to BiggestAvailable

Sometimes, a service group might not fail over to the biggest available system even when FailOverPolicy is set to BiggestAvailable.

To troubleshoot this issue, check the engine log located in /var/VRTSvcs/log/engine_A.log to find out the reasons for not failing over to the biggest available system. This may be due to the following reasons:

  • If , one or more of the systems in the service group's SystemList did not have forecasted available capacity, you see the following message in the engine log:

    One of the systems in SystemList of group group_name, system system_name does not have forecasted available capacity updated

  • If the hautil - sys command does not list forecasted available capacity for the systems, you see the following message in the engine log:

    Failed to forecast due to insufficient data

    This message is displayed due to insufficient recent data to be used for forecasting the available capacity.

    The default value for the MeterInterval key of the cluster attribute MeterControl is 120 seconds. There will be enough recent data available for forecasting after 3 metering intervals (6 minutes) from time the VCS engine was started on the system. After this, the forecasted values are updated every ForecastCycle * MeterInterval seconds. The ForecastCycle and MeterInterval values are specified in the cluster attribute MeterControl.

  • If one or more of the systems in the service group's SystemList have stale forecasted available capacity, you can see the following message in the engine log:

    System system_name has not updated forecasted available capacity since last 2 forecast cycles

    This issue is caused when the HostMonitor agent stops functioning. Check if HostMonitor agent process is running by issuing one of the following commands on the system which has stale forecasted values:

    • # ps - aef|grep HostMonitor

    • # haagent -display HostMonitor

      If HostMonitor agent is not running, you can start it by issuing the following command: 

      # hagent -start HostMonitor -sys system_name

      Even if HostMonitor agent is running and you see the above message in the engine log, it means that the HostMonitor agent is not able to forecast, and it will log error messages in the HostMonitor_A.log file in the /var/VRTSvcs/log/ directory.