Veritas InfoScale™ 7.4.2 Troubleshooting Guide - Linux
- Introduction
- Section I. Troubleshooting Veritas File System
- Section II. Troubleshooting Veritas Volume Manager
- Recovering from hardware failure
- About recovery from hardware failure
- Listing unstartable volumes
- Displaying volume and plex states
- The plex state cycle
- Recovering an unstartable mirrored volume
- Recovering an unstartable volume with a disabled plex in the RECOVER state
- Forcibly restarting a disabled volume
- Clearing the failing flag on a disk
- Reattaching failed disks
- Recovering from a failed plex attach or synchronization operation
- Failures on RAID-5 volumes
- Recovering from an incomplete disk group move
- Restarting volumes after recovery when some nodes in the cluster become unavailable
- Recovery from failure of a DCO volume
- Recovering from instant snapshot failure
- Recovering from the failure of vxsnap prepare
- Recovering from the failure of vxsnap make for full-sized instant snapshots
- Recovering from the failure of vxsnap make for break-off instant snapshots
- Recovering from the failure of vxsnap make for space-optimized instant snapshots
- Recovering from the failure of vxsnap restore
- Recovering from the failure of vxsnap refresh
- Recovering from copy-on-write failure
- Recovering from I/O errors during resynchronization
- Recovering from I/O failure on a DCO volume
- Recovering from failure of vxsnap upgrade of instant snap data change objects (DCOs)
- Recovering from failed vxresize operation
- Recovering from boot disk failure
- VxVM and boot disk failure
- Possible root disk configurations
- The boot process
- VxVM boot disk recovery
- Recovery by reinstallation
- Manually unencapsulating a root disk
- Managing commands, tasks, and transactions
- Backing up and restoring disk group configurations
- Troubleshooting issues with importing disk groups
- Recovering from CDS errors
- Logging and error messages
- Troubleshooting Veritas Volume Replicator
- Recovery from RLINK connect problems
- Recovery from configuration errors
- Errors during an RLINK attach
- Errors during modification of an RVG
- Recovery on the Primary or Secondary
- About recovery from a Primary-host crash
- Recovering from Primary data volume error
- Primary SRL volume error cleanup and restart
- Primary SRL volume error at reboot
- Primary SRL volume overflow recovery
- Primary SRL header error cleanup and recovery
- Secondary data volume error cleanup and recovery
- Secondary SRL volume error cleanup and recovery
- Secondary SRL header error cleanup and recovery
- Secondary SRL header error at reboot
- Recovering from hardware failure
- Section III. Troubleshooting Dynamic Multi-Pathing
- Section IV. Troubleshooting Storage Foundation Cluster File System High Availability
- Troubleshooting Storage Foundation Cluster File System High Availability
- About troubleshooting Storage Foundation Cluster File System High Availability
- Troubleshooting CFS
- Troubleshooting fenced configurations
- Troubleshooting Cluster Volume Manager in Veritas InfoScale products clusters
- CVM group is not online after adding a node to the Veritas InfoScale products cluster
- Shared disk group cannot be imported in Veritas InfoScale products cluster
- Unable to start CVM in Veritas InfoScale products cluster
- Removing preexisting keys
- CVMVolDg not online even though CVMCluster is online in Veritas InfoScale products cluster
- Shared disks not visible in Veritas InfoScale products cluster
- Troubleshooting interconnects
- Troubleshooting Storage Foundation Cluster File System High Availability
- Section V. Troubleshooting Cluster Server
- Troubleshooting and recovery for VCS
- VCS message logging
- Log unification of VCS agent's entry points
- Enhancing First Failure Data Capture (FFDC) to troubleshoot VCS resource's unexpected behavior
- GAB message logging
- Enabling debug logs for agents
- Enabling debug logs for IMF
- Enabling debug logs for the VCS engine
- About debug log tags usage
- Gathering VCS information for support analysis
- Gathering LLT and GAB information for support analysis
- Gathering IMF information for support analysis
- Message catalogs
- Troubleshooting the VCS engine
- Troubleshooting Low Latency Transport (LLT)
- Troubleshooting Group Membership Services/Atomic Broadcast (GAB)
- Troubleshooting VCS startup
- Troubleshooting issues with systemd unit service files
- If a unit service has failed and the corresponding module is still loaded, systemd cannot unload it and so its package cannot be removed
- If a unit service is active and the corresponding process is stopped outside of systemd, the service cannot be started again using 'systemctl start'
- If a unit service takes longer than the default timeout to stop or start the corresponding service, it goes into the Failed state
- Troubleshooting Intelligent Monitoring Framework (IMF)
- Troubleshooting service groups
- VCS does not automatically start service group
- System is not in RUNNING state
- Service group not configured to run on the system
- Service group not configured to autostart
- Service group is frozen
- Failover service group is online on another system
- A critical resource faulted
- Service group autodisabled
- Service group is waiting for the resource to be brought online/taken offline
- Service group is waiting for a dependency to be met.
- Service group not fully probed.
- Service group does not fail over to the forecasted system
- Service group does not fail over to the BiggestAvailable system even if FailOverPolicy is set to BiggestAvailable
- Restoring metering database from backup taken by VCS
- Initialization of metering database fails
- Troubleshooting resources
- Troubleshooting I/O fencing
- Node is unable to join cluster while another node is being ejected
- The vxfentsthdw utility fails when SCSI TEST UNIT READY command fails
- Manually removing existing keys from SCSI-3 disks
- System panics to prevent potential data corruption
- Cluster ID on the I/O fencing key of coordinator disk does not match the local cluster's ID
- Fencing startup reports preexisting split-brain
- Registered keys are lost on the coordinator disks
- Replacing defective disks when the cluster is offline
- The vxfenswap utility exits if rcp or scp commands are not functional
- Troubleshooting CP server
- Troubleshooting server-based fencing on the Veritas InfoScale products cluster nodes
- Issues during online migration of coordination points
- Troubleshooting notification
- Troubleshooting and recovery for global clusters
- Troubleshooting the steward process
- Troubleshooting licensing
- Validating license keys
- Licensing error messages
- [Licensing] Insufficient memory to perform operation
- [Licensing] No valid VCS license keys were found
- [Licensing] Unable to find a valid base VCS license key
- [Licensing] License key cannot be used on this OS platform
- [Licensing] VCS evaluation period has expired
- [Licensing] License key can not be used on this system
- [Licensing] Unable to initialize the licensing framework
- [Licensing] QuickStart is not supported in this release
- [Licensing] Your evaluation period for the feature has expired. This feature will not be enabled the next time VCS starts
- Verifying the metered or forecasted values for CPU, Mem, and Swap
- VCS message logging
- Troubleshooting and recovery for VCS
- Section VI. Troubleshooting SFDB
Replacing a failed root disk
To replace a failed root disk
- Use the vxplex command to remove the plex records that were on the failed disk:
# vxplex -g bootdg -o rm dis rootvol-01 swapvol-01
This example removes the plexes rootvol-01, and swapvol-01 that are configured on the failed root disk. You may need to modify the list of plexes according to your system configuration.
- Shut down the system, and then power it down.
- Replace the failed disk with a disk of the same or larger capacity.
- Power up the system, and boot it from Linux installation CD number 1.
- On a Red Hat system, run the following command at the boot prompt to put the system in rescue mode:
boot: linux rescue
On a SUSE system, choose the Rescue option from the menu.
Log in as root, select your language and keyboard, and choose to skip finding your installation.
- Use the fdisk command to ensure that the root mirror disk (sdb) and the replacement root disk (sda) have the same geometry:
# fdisk -l /dev/sdb # fdisk -l /dev/sda
See the fdisk(8) manual page.
- If the replacement disk already contains a VxVM private region, use the fdisk command to change the partition type for the private region partition to a value other than 7f.
# fdisk /dev/sda
- Make a temporary mount point, /vxvm, and mount the root partition on it:
# mkdir /vxvm # mount -t ext3 /dev/sdb1 /vxvm
In this example, the mirror of the root partition is /dev/sdb1, and the root file system type is ext3. You may need to modify this command according to your system configuration. For example, the root file system may be configured as a reiserfs file system.
- If the disk has a separate boot partition, mount this on /vxvm/boot:
# mount -t ext3 /dev/sdb2 /vxvm/boot
In this example, the mirror of the boot partition is /dev/sdb2, and the boot file system type is ext3. You may need to modify this command according to your system configuration.
- Install the master boot record (MBR) on the replacement disk (in this example, sda).
For the GRUB boot loader:
Create a backup copy of the GRUB configuration file (/vxvm/boot/grub/menu.lst or /vxvm/etc/grub.conf as appropriate), for example:
# cp /vxvm/etc/grub.conf /vxvm/etc/grub.conf.b4repldisk
Run the sync command:
# sync
In the configuration file, change all occurrences of sda to sdb, except for the boot= statement.
In the configuration file, change all occurrences of hd0 to hd1.
After saving your changes to the configuration file, run the following commands to install the boot loader:
# /vxvm/sbin/grub grub> root (hd1,1) grub> setup (hd0) grub> quitFor the LILO boot loader:
Create a backup copy of the LILO configuration file, for example:
# cp /vxvm/etc/lilo.conf /vxvm/etc/lilo.conf.b4repldisk
Run the sync command:
# sync
In the configuration file, change all occurrences of sda to sdb, except for the boot= statement.
In the configuration file, add a root= statement to the boot entries where this is missing. This statement specifies the device that is to be mounted as root, for example, /dev/sdb1. The following example is for the vxvm_root entry:
image=/boot/vmlinuz-2.4.21-4.ELsmp label=vxvm_root initrd=/boot/VxVM_initrd.img root=/dev/sdb1After saving your changes to the configuration file, run the following command to install the boot loader:
# /vxvm/sbin/lilo -r /vxvm
- Unmount the partitions, run sync, and then exit the rescue shell:
# cd / # umount /vxvm/boot # umount /vxvm # sync # exit
- Shut down and power cycle the system. Enter the system's BIOS settings mode (this is usually achieved by pressing a key such as Esc, F2 or F12 on the console keyboard). Verify in the BIOS settings that the system is set to boot from the new root disk (in this example, sdb). Otherwise the system may not be bootable.
- Reboot the system, selecting vxvm_root at the GRUB or LILO boot prompt as appropriate.
- Run the following command to mirror the volumes from the root mirror disk onto the replacement disk:
# /etc/vx/bin/vxrootmir sda rootdisk
This example assumes that the disk media name of the replacement root disk is sda. You may need to modify this name according to your system configuration.
- Restore the contents of the boot loader configuration file, and recreate the original MBR on the root disk (in this example, sda).
For the GRUB boot loader:
Restore the original boot loader configuration file:
# mv /etc/grub.conf.b4repldisk /etc/grub.conf
Run the sync command:
# sync
Run the following commands to recreate the boot loader:
# /sbin/grub grub> root (hd0,1) grub> setup --stage2=/boot/grub/stage2 (hd0) grub> quitFor the LILO boot loader:
Restore the original boot loader configuration file:
# mv /etc/lilo.conf.b4repldisk /etc/lilo.conf
Run the sync command:
# sync
Run the following command to recreate the boot loader:
# /sbin/lilo