Storage Foundation for Oracle® RAC 7.3.1 Administrator's Guide - Linux
- Section I. SF Oracle RAC concepts and administration
- Overview of Storage Foundation for Oracle RAC
- About Storage Foundation for Oracle RAC
- How SF Oracle RAC works (high-level perspective)
- Component products and processes of SF Oracle RAC
- Periodic health evaluation of SF Oracle RAC clusters
- About Virtual Business Services
- About Veritas InfoScale Operations Manager
- About Veritas Services and Operations Readiness Tools (SORT)
- Administering SF Oracle RAC and its components
- Administering SF Oracle RAC
- Setting the environment variables for SF Oracle RAC
- Starting or stopping SF Oracle RAC on each node
- Applying Oracle patches on SF Oracle RAC nodes
- Migrating Pluggable Databases (PDB) between Container Databases (CDB)
- Installing Veritas Volume Manager, Veritas File System, or ODM patches on SF Oracle RAC nodes
- Applying operating system updates on SF Oracle RAC nodes
- Adding storage to an SF Oracle RAC cluster
- Recovering from storage failure
- Backing up and restoring Oracle database using Veritas NetBackup
- Enhancing the performance of SF Oracle RAC clusters
- Administering SmartIO
- Creating snapshots for offhost processing
- Managing database storage efficiently using SmartTier
- Optimizing database storage using Thin Provisioning and SmartMove
- Scheduling periodic health checks for your SF Oracle RAC cluster
- Using environment variables to start and stop VCSMM modules
- Verifying the nodes in an SF Oracle RAC cluster
- Administering VCS
- About managing VCS modules
- Viewing available Veritas device drivers
- Starting and stopping VCS
- Environment variables to start and stop VCS modules
- Adding and removing LLT links
- Configuring aggregated interfaces under LLT
- Displaying the cluster details and LLT version for LLT links
- Configuring destination-based load balancing for LLT
- Enabling and disabling intelligent resource monitoring for agents manually
- Administering the AMF kernel driver
- Administering I/O fencing
- About administering I/O fencing
- About the vxfentsthdw utility
- General guidelines for using the vxfentsthdw utility
- About the vxfentsthdw command options
- Testing the coordinator disk group using the -c option of vxfentsthdw
- Performing non-destructive testing on the disks using the -r option
- Testing the shared disks using the vxfentsthdw -m option
- Testing the shared disks listed in a file using the vxfentsthdw -f option
- Testing all the disks in a disk group using the vxfentsthdw -g option
- Testing a disk with existing keys
- About the vxfenadm utility
- About the vxfenclearpre utility
- About the vxfenswap utility
- Enabling or disabling the preferred fencing policy
- About I/O fencing log files
- Migrating from disk-based fencing to server-based fencing using the installer
- Migrating from server-based fencing to disk-based fencing using the installer
- Administering the CP server
- Administering CFS
- Administering CVM
- Listing all the CVM shared disks
- Establishing CVM cluster membership manually
- Changing the CVM master manually
- Importing a shared disk group manually
- Deporting a shared disk group manually
- Starting shared volumes manually
- Verifying if CVM is running in an SF Oracle RAC cluster
- Verifying CVM membership state
- Verifying the state of CVM shared disk groups
- Verifying the activation mode
- Administering Flexible Storage Sharing
- About Flexible Storage Sharing disk support
- About the volume layout for Flexible Storage Sharing disk groups
- Setting the host prefix
- Exporting a disk for Flexible Storage Sharing
- Setting the Flexible Storage Sharing attribute on a disk group
- Using the host disk class and allocating storage
- Administering mirrored volumes using vxassist
- Displaying exported disks and network shared disk groups
- Tuning LLT for memory and performance in FSS environments
- Backing up and restoring disk group configuration data
- Administering SF Oracle RAC global clusters
- Administering SF Oracle RAC
- Overview of Storage Foundation for Oracle RAC
- Section II. Performance and troubleshooting
- Troubleshooting SF Oracle RAC
- About troubleshooting SF Oracle RAC
- Restarting the installer after a failed network connection
- Installer cannot create UUID for the cluster
- Troubleshooting SF Oracle RAC pre-installation check failures
- Troubleshooting LLT health check warning messages
- Troubleshooting I/O fencing
- SCSI reservation errors during bootup
- The vxfentsthdw utility fails when SCSI TEST UNIT READY command fails
- Node is unable to join cluster while another node is being ejected
- System panics to prevent potential data corruption
- Cluster ID on the I/O fencing key of coordinator disk does not match the local cluster's ID
- Fencing startup reports preexisting split-brain
- Registered keys are lost on the coordinator disks
- Replacing defective disks when the cluster is offline
- Troubleshooting I/O fencing health check warning messages
- Troubleshooting CP server
- Troubleshooting server-based fencing on the SF Oracle RAC cluster nodes
- Issues during online migration of coordination points
- Troubleshooting Cluster Volume Manager in SF Oracle RAC clusters
- Restoring communication between host and disks after cable disconnection
- Shared disk group cannot be imported in SF Oracle RAC cluster
- Error importing shared disk groups in SF Oracle RAC cluster
- Unable to start CVM in SF Oracle RAC cluster
- CVM group is not online after adding a node to the SF Oracle RAC cluster
- CVMVolDg not online even though CVMCluster is online in SF Oracle RAC cluster
- Shared disks not visible in SF Oracle RAC cluster
- Troubleshooting CFS
- Troubleshooting interconnects
- Troubleshooting Oracle
- Error when starting an Oracle instance in SF Oracle RAC
- Clearing Oracle group faults
- Oracle log files show shutdown called even when not shutdown manually
- DBCA fails while creating an Oracle RAC database
- Oracle's clusterware processes fail to start
- Oracle Clusterware fails after restart
- Troubleshooting the Virtual IP (VIP) configuration in an SF Oracle RAC cluster
- Troubleshooting Oracle Clusterware health check warning messages in SF Oracle RAC clusters
- Troubleshooting ODM in SF Oracle RAC clusters
- Troubleshooting Flex ASM in SF Oracle RAC clusters
- Prevention and recovery strategies
- Tunable parameters
- Troubleshooting SF Oracle RAC
- Section III. Reference
LLT health checks
This section lists the health checks performed for LLT, the messages displayed for each check, and a brief description of the check.
Note:
Warning messages indicate issues in the components or the general health of the cluster. Follow the troubleshooting recommendations to resolve the issues.
Table: List of health checks for LLT lists the health checks performed for LLT.
Table: List of health checks for LLT
List of health checks | Message | Description |
|---|---|---|
LLT timer subsystem scheduling check | Warning: OS timer is not called for num seconds | Checks whether the LLT module runs in accordance with the scheduled operating system timer. The message indicates that the operating system timer is not called for the specified interval. The parameter timer_threshold contains the optimum threshold for this check. |
OS memory availability check for the packet transmission | Warning: Kernel failed to allocate memory num time(s) | Checks whether the kernel has allocated sufficient memory to LLT for cluster communication. The message indicates that the kernel attempts at allocating the requisite memory has failed (num) times. The parameter no_mem_to_xmit_allow contains the optimum threshold for this check. |
Flow control status monitoring | Flow-control occurred num time(s)and back-enabled num time(s) on port port number for node node number | Checks whether LLT has sufficient bandwidth to accept incoming data packets and transmit the data packets. Flow control also depends on the peer nodes' ability to accept incoming data traffic. The message indicates that packet transmission and reception was controlled (num) and normalized (num) times. The parameter max_canput contains the optimum threshold for this check. |
Flaky link monitoring | Warning: Connectivity with node node id on link link id is flaky num time(s). The distribution is: (0-4 s) <num> (4-8 s) <num> (8-12 s) <num> (12-16 s) <num> (>=16 s) | Checks whether the private interconnects are stable. The message indicates that connectivity (link) with the peer node (node id) is monitored (num) times within a stipulated duration (for example, 0-4 seconds). The parameter flakylink_allow contains the optimum threshold for this check. |
Node status check | One or more link connectivity with peer node(s) node name is in trouble. | Checks the current node membership. The message indicates connectivity issues with the peer node (node name). This check does not require a threshold. |
Link status check | Link connectivity with node name is on only one link. Veritas recommends configuring a minimum of 2 links. | Checks the number of private interconnects that are currently active. The message indicates that only one private interconnect exists or is operational between the local node and the peer node node id. This check does not require a threshold. |
Connectivity check | Only one link is configured under LLT. Veritas recommends configuring a minimum of 2 links. | Checks the number of private interconnects that were configured under LLT during configuration. The message indicates that only one private interconnect exists or is configured under LLT. This check does not require a threshold. |
LLT packet related checks | Retransmitted % percentage of total transmitted packets. Sent % percentage of total transmitted packet when no link is up. % percentage of total received packets are with bad checksum. % percentage of total received packets are out of window. % percentage of total received packets are misaligned. | Checks whether the data packets transmitted by LLT reach peer nodes without any error. If there is an error in packet transmission, it indicates an error in the private interconnects. The message indicates the percentage of data packets transmitted or received and the associated transmission errors, such as bad checksum and failed link.
The following parameters contain the optimum thresholds for this check: retrans_pct, send_nolinkup_pct, recv_oow_pct, recv_misaligned_pct, recv_badcksum_pct |
DLPI layer related checks | % percentage of total received packets are with DLPI error. | Checks whether the data link layer (DLP) is causing errors in packet transmission. The message indicates that some of the received packets (%) contain DLPI error. You can set a desired percentage value in the configuration file. The parameter recv_dlpierror_pct contains the optimum threshold for this check. |
Traffic distribution over the links | Traffic distribution over links: %% Send data on linknum percentage %% Recv data on linknum percentage | Checks the distribution of traffic over all the links configured under LLT. The message displays the percentage of data (%) sent and recd on a particular link (num) This check does not require a threshold. |
LLT Ports status check | % per of total transmitted packets are with large xmit latency (>16ms) for port port id %per received packets are with large recv latency (>16ms) for port port id. | Checks the latency period for transmitting or receiving packets. The message indicates that some percentage (%) of the transmitted/received packets exceed the stipulated latency time. The following parameters contain the optimum thresholds for this check: hirecvlatencycnt_pct, hixmitlatencycnt_pct. |
System load monitoring | Load Information: Average : num,num,num | Monitors the system workload at the stipulated periodicity (1 second, 5 seconds, 15 seconds) This check does not require a threshold. |