Storage Foundation and High Availability 8.0.2 Configuration and Upgrade Guide - Linux
- Section I. Introduction to SFHA
- Introducing Storage Foundation and High Availability
- Section II. Configuration of SFHA
- Preparing to configure
- Preparing to configure SFHA clusters for data integrity
- About planning to configure I/O fencing
- Setting up the CP server
- Planning your CP server setup
- Installing the CP server using the installer
- Configuring the CP server cluster in secure mode
- Setting up shared storage for the CP server database
- Configuring the CP server using the installer program
- Configuring the CP server manually
- Configuring CP server using response files
- Verifying the CP server configuration
- Configuring SFHA
- Configuring Storage Foundation High Availability using the installer
- Overview of tasks to configure SFHA using the product installer
- Required information for configuring Storage Foundation and High Availability Solutions
- Starting the software configuration
- Specifying systems for configuration
- Configuring the cluster name
- Configuring private heartbeat links
- Configuring the virtual IP of the cluster
- Configuring SFHA in secure mode
- Configuring a secure cluster node by node
- Adding VCS users
- Configuring SMTP email notification
- Configuring SNMP trap notification
- Configuring global clusters
- Completing the SFHA configuration
- About Veritas License Audit Tool
- Verifying and updating licenses on the system
- Configuring SFDB
- Configuring Storage Foundation High Availability using the installer
- Configuring SFHA clusters for data integrity
- Setting up disk-based I/O fencing using installer
- Setting up server-based I/O fencing using installer
- Setting up non-SCSI-3 I/O fencing in virtual environments using installer
- Setting up majority-based I/O fencing using installer
- Enabling or disabling the preferred fencing policy
- Manually configuring SFHA clusters for data integrity
- Setting up disk-based I/O fencing manually
- Setting up server-based I/O fencing manually
- Preparing the CP servers manually for use by the SFHA cluster
- Generating the client key and certificates manually on the client nodes
- Configuring server-based fencing on the SFHA cluster manually
- Configuring CoordPoint agent to monitor coordination points
- Verifying server-based I/O fencing configuration
- Setting up non-SCSI-3 fencing in virtual environments manually
- Setting up majority-based I/O fencing manually
- Performing an automated SFHA configuration using response files
- Performing an automated I/O fencing configuration using response files
- Configuring I/O fencing using response files
- Response file variables to configure disk-based I/O fencing
- Sample response file for configuring disk-based I/O fencing
- Response file variables to configure server-based I/O fencing
- Sample response file for configuring server-based I/O fencing
- Response file variables to configure non-SCSI-3 I/O fencing
- Sample response file for configuring non-SCSI-3 I/O fencing
- Response file variables to configure majority-based I/O fencing
- Sample response file for configuring majority-based I/O fencing
- Section III. Upgrade of SFHA
- Planning to upgrade SFHA
- About the upgrade
- Supported upgrade paths
- Considerations for upgrading SFHA to 8.0.2 on systems configured with an Oracle resource
- Preparing to upgrade SFHA
- Considerations for upgrading REST server
- Using Install Bundles to simultaneously install or upgrade full releases (base, maintenance, rolling patch), and individual patches
- Upgrading Storage Foundation and High Availability
- Performing a rolling upgrade of SFHA
- Performing a phased upgrade of SFHA
- About phased upgrade
- Performing a phased upgrade using the product installer
- Moving the service groups to the second subcluster
- Upgrading the operating system on the first subcluster
- Upgrading the first subcluster
- Preparing the second subcluster
- Activating the first subcluster
- Upgrading the operating system on the second subcluster
- Upgrading the second subcluster
- Finishing the phased upgrade
- Performing an automated SFHA upgrade using response files
- Performing post-upgrade tasks
- Optional configuration steps
- Re-joining the backup boot disk group into the current disk group
- Reverting to the backup boot disk group after an unsuccessful upgrade
- Recovering VVR if automatic upgrade fails
- Post-upgrade tasks when VCS agents for VVR are configured
- Resetting DAS disk names to include host name in FSS environments
- Upgrading disk layout versions
- Upgrading VxVM disk group versions
- Updating variables
- Setting the default disk group
- About enabling LDAP authentication for clusters that run in secure mode
- Verifying the Storage Foundation and High Availability upgrade
- Planning to upgrade SFHA
- Section IV. Post-installation tasks
- Section V. Adding and removing nodes
- Adding a node to SFHA clusters
- About adding a node to a cluster
- Before adding a node to a cluster
- Adding a node to a cluster using the Veritas InfoScale installer
- Adding the node to a cluster manually
- Adding a node using response files
- Configuring server-based fencing on the new node
- After adding the new node
- Adding nodes to a cluster that is using authentication for SFDB tools
- Updating the Storage Foundation for Databases (SFDB) repository after adding a node
- Removing a node from SFHA clusters
- Removing a node from a SFHA cluster
- Verifying the status of nodes and service groups
- Deleting the departing node from SFHA configuration
- Modifying configuration files on each remaining node
- Removing the node configuration from the CP server
- Removing security credentials from the leaving node
- Unloading LLT and GAB and removing Veritas InfoScale Availability or Enterprise on the departing node
- Updating the Storage Foundation for Databases (SFDB) repository after removing a node
- Removing a node from a SFHA cluster
- Adding a node to SFHA clusters
- Section VI. Configuration and upgrade reference
- Appendix A. Installation scripts
- Appendix B. SFHA services and ports
- Appendix C. Configuration files
- Appendix D. Configuring the secure shell or the remote shell for communications
- About configuring secure shell or remote shell communication modes before installing products
- Manually configuring passwordless ssh
- Setting up ssh and rsh connection using the installer -comsetup command
- Setting up ssh and rsh connection using the pwdutil.pl utility
- Restarting the ssh session
- Enabling rsh for Linux
- Appendix E. Sample SFHA cluster setup diagrams for CP server-based I/O fencing
- Appendix F. Configuring LLT over UDP
- Using the UDP layer for LLT
- Manually configuring LLT over UDP using IPv4
- Broadcast address in the /etc/llttab file
- The link command in the /etc/llttab file
- The set-addr command in the /etc/llttab file
- Selecting UDP ports
- Configuring the netmask for LLT
- Configuring the broadcast address for LLT
- Sample configuration: direct-attached links
- Sample configuration: links crossing IP routers
- Using the UDP layer of IPv6 for LLT
- Manually configuring LLT over UDP using IPv6
- About configuring LLT over UDP multiport
- Appendix G. Using LLT over RDMA
- Using LLT over RDMA
- About RDMA over RoCE or InfiniBand networks in a clustering environment
- How LLT supports RDMA capability for faster interconnects between applications
- Using LLT over RDMA: supported use cases
- Configuring LLT over RDMA
- Choosing supported hardware for LLT over RDMA
- Installing RDMA, InfiniBand or Ethernet drivers and utilities
- Configuring RDMA over an Ethernet network
- Configuring RDMA over an InfiniBand network
- Tuning system performance
- Manually configuring LLT over RDMA
- LLT over RDMA sample /etc/llttab
- Verifying LLT configuration
- Troubleshooting LLT over RDMA
- IP addresses associated to the RDMA NICs do not automatically plumb on node restart
- Ping test fails for the IP addresses configured over InfiniBand interfaces
- After a node restart, by default the Mellanox card with Virtual Protocol Interconnect (VPI) gets configured in InfiniBand mode
- The LLT module fails to start
Setting up server-based I/O fencing using installer
You can configure server-based I/O fencing for the SFHA cluster using the installer.
With server-based fencing, you can have the coordination points in your configuration as follows:
Combination of CP servers and SCSI-3 compliant coordinator disks
CP servers only
Veritas also supports server-based fencing with a single highly available CP server that acts as a single coordination point.
See About planning to configure I/O fencing.
See Recommended CP server configurations.
This section covers the following example procedures:
Mix of CP servers and coordinator disks | See “To configure server-based fencing for the SFHA cluster (one CP server and two coordinator disks)”. |
Single CP server | See “To configure server-based fencing for the SFHA cluster”. |
To configure server-based fencing for the SFHA cluster (one CP server and two coordinator disks)
Depending on the server-based configuration model in your setup, make sure of the following:
CP servers are configured and are reachable from the SFHA cluster. The SFHA cluster is also referred to as the application cluster or the client cluster.
The coordination disks are verified for SCSI3-PR compliance.
- Start the installer with the -fencing option.
# /opt/VRTS/install/installer -fencing
The installer starts with a copyright message and verifies the cluster information.
Note the location of log files which you can access in the event of any problem with the configuration process.
- Confirm that you want to proceed with the I/O fencing configuration at the prompt.
The program checks that the local node running the script can communicate with remote nodes and checks whether SFHA 8.0.2 is configured properly.
- Review the I/O fencing configuration options that the program presents. Type 1 to configure server-based I/O fencing.
Select the fencing mechanism to be configured in this Application Cluster [1-3,b,q] 1
- Make sure that the storage supports SCSI3-PR, and answer y at the following prompt.
Does your storage environment support SCSI3 PR? [y,n,q] (y)
Provide the following details about the coordination points at the installer prompt:
Enter the total number of coordination points including both servers and disks. This number should be at least 3.
Enter the total number of co-ordination points including both Coordination Point servers and disks: [b] (3)
Enter the total number of coordinator disks among the coordination points.
Enter the total number of disks among these: [b] (0) 2
Provide the following CP server details at the installer prompt:
Enter the total number of virtual IP addresses or the total number of fully qualified host names for each of the CP servers.
How many IP addresses would you like to use to communicate to Coordination Point Server #1?: [b,q,?] (1) 1
Enter the virtual IP addresses or the fully qualified host name for each of the CP servers. The installer assumes these values to be identical as viewed from all the application cluster nodes.
Enter the Virtual IP address or fully qualified host name #1 for the HTTPS Coordination Point Server #1: [b] 10.209.80.197
The installer prompts for this information for the number of virtual IP addresses you want to configure for each CP server.
Enter the port that the CP server would be listening on.
Enter the port that the coordination point server 10.209.80.197 would be listening on or accept the default port suggested: [b] (443)
Provide the following coordinator disks-related details at the installer prompt:
Choose the coordinator disks from the list of available disks that the installer displays. Ensure that the disk you choose is available from all the SFHA (application cluster) nodes.
The number of times that the installer asks you to choose the disks depends on the information that you provided in step 6. For example, if you had chosen to configure two coordinator disks, the installer asks you to choose the first disk and then the second disk:
Select disk number 1 for co-ordination point 1) sdx 2) sdy 3) sdz Please enter a valid disk which is available from all the cluster nodes for co-ordination point [1-3,q] 1
If you have not already checked the disks for SCSI-3 PR compliance in step 1, check the disks now.
The installer displays a message that recommends you to verify the disks in another window and then return to this configuration procedure.
Press Enter to continue, and confirm your disk selection at the installer prompt.
Enter a disk group name for the coordinator disks or accept the default.
Enter the disk group name for coordinating disk(s): [b] (vxfencoorddg)
- Verify and confirm the coordination points information for the fencing configuration.
For example:
Total number of coordination points being used: 3 Coordination Point Server ([VIP or FQHN]:Port): 1. 10.209.80.197 ([10.209.80.197]:443) SCSI-3 disks: 1. sdx 2. sdy Disk Group name for the disks in customized fencing: vxfencoorddg Disk policy used for customized fencing: dmpThe installer initializes the disks and the disk group and deports the disk group on the SFHA (application cluster) node.
- Verify and confirm the I/O fencing configuration information.
CPS Admin utility location: /opt/VRTScps/bin/cpsadm Cluster ID: 2122 Cluster Name: clus1 UUID for the above cluster: {ae5e589a-1dd1-11b2-dd44-00144f79240c} - Review the output as the installer updates the application cluster information on each of the CP servers to ensure connectivity between them. The installer then populates the
/etc/vxfenmodefile with the appropriate details in each of the application cluster nodes.Updating client cluster information on Coordination Point Server 10.209.80.197 Adding the client cluster to the Coordination Point Server 10.209.80.197 .......... Done Registering client node sys1 with Coordination Point Server 10.209.80.197...... Done Adding CPClient user for communicating to Coordination Point Server 10.209.80.197 .... Done Adding cluster clus1 to the CPClient user on Coordination Point Server 10.209.80.197 .. Done Registering client node sys2 with Coordination Point Server 10.209.80.197 ..... Done Adding CPClient user for communicating to Coordination Point Server 10.209.80.197 .... Done Adding cluster clus1 to the CPClient user on Coordination Point Server 10.209.80.197 ..Done Updating /etc/vxfenmode file on sys1 .................................. Done Updating /etc/vxfenmode file on sys2 ......... ........................ Done
- Review the output as the installer stops and restarts the VCS and the fencing processes on each application cluster node, and completes the I/O fencing configuration.
- Configure the CP agent on the SFHA (application cluster). The Coordination Point Agent monitors the registrations on the coordination points.
Do you want to configure Coordination Point Agent on the client cluster? [y,n,q] (y) Enter a non-existing name for the service group for Coordination Point Agent: [b] (vxfen)
- Additionally the coordination point agent can also monitor changes to the Coordinator Disk Group constitution such as a disk being accidently deleted from the Coordinator Disk Group. The frequency of this detailed monitoring can be tuned with the LevelTwoMonitorFreq attribute. For example, if you set this attribute to 5, the agent will monitor the Coordinator Disk Group constitution every five monitor cycles.
Note that for the LevelTwoMonitorFreq attribute to be applicable there must be disks as part of the Coordinator Disk Group.
Enter the value of the LevelTwoMonitorFreq attribute: (5)
- Enable auto refresh of coordination points.
Do you want to enable auto refresh of coordination points if registration keys are missing on any of them? [y,n,q,b,?] (n)
- Note the location of the configuration log files, summary files, and response files that the installer displays for later use.
- Verify the fencing configuration using:
# vxfenadm -d
- Verify the list of coordination points.
# vxfenconfig -l
To configure server-based fencing for the SFHA cluster
- Make sure that the CP server is configured and is reachable from the SFHA cluster. The SFHA cluster is also referred to as the application cluster or the client cluster.
- See Setting up the CP server.
- Start the installer with -fencing option.
# /opt/VRTS/install/installer -fencing
The installer starts with a copyright message and verifies the cluster information.
Note the location of log files which you can access in the event of any problem with the configuration process.
- Confirm that you want to proceed with the I/O fencing configuration at the prompt.
The program checks that the local node running the script can communicate with remote nodes and checks whether SFHA 8.0.2 is configured properly.
- Review the I/O fencing configuration options that the program presents. Type 1 to configure server-based I/O fencing.
Select the fencing mechanism to be configured in this Application Cluster [1-7,q] 1
- Make sure that the storage supports SCSI3-PR, and answer y at the following prompt.
Does your storage environment support SCSI3 PR? [y,n,q] (y)
- Enter the total number of coordination points as 1.
Enter the total number of co-ordination points including both Coordination Point servers and disks: [b] (3) 1
Read the installer warning carefully before you proceed with the configuration.
Provide the following CP server details at the installer prompt:
Enter the total number of virtual IP addresses or the total number of fully qualified host names for each of the CP servers.
How many IP addresses would you like to use to communicate to Coordination Point Server #1? [b,q,?] (1) 1
Enter the virtual IP address or the fully qualified host name for the CP server. The installer assumes these values to be identical as viewed from all the application cluster nodes.
Enter the Virtual IP address or fully qualified host name #1 for the Coordination Point Server #1: [b] 10.209.80.197
The installer prompts for this information for the number of virtual IP addresses you want to configure for each CP server.
Enter the port that the CP server would be listening on.
Enter the port in the range [49152, 65535] which the Coordination Point Server 10.209.80.197 would be listening on or simply accept the default port suggested: [b] (443)
- Verify and confirm the coordination points information for the fencing configuration.
For example:
Total number of coordination points being used: 1 Coordination Point Server ([VIP or FQHN]:Port): 1. 10.209.80.197 ([10.209.80.197]:443) - Verify and confirm the I/O fencing configuration information.
CPS Admin utility location: /opt/VRTScps/bin/cpsadm Cluster ID: 2122 Cluster Name: clus1 UUID for the above cluster: {ae5e589a-1dd1-11b2-dd44-00144f79240c} - Review the output as the installer updates the application cluster information on each of the CP servers to ensure connectivity between them. The installer then populates the
/etc/vxfenmodefile with the appropriate details in each of the application cluster nodes.The installer also populates the
/etc/vxfenmodefile with the entry single_cp=1 for such single CP server fencing configuration.Updating client cluster information on Coordination Point Server 10.209.80.197 Adding the client cluster to the Coordination Point Server 10.209.80.197 .......... Done Registering client node sys1 with Coordination Point Server 10.209.80.197...... Done Adding CPClient user for communicating to Coordination Point Server 10.209.80.197 .... Done Adding cluster clus1 to the CPClient user on Coordination Point Server 10.209.80.197 .. Done Registering client node sys2 with Coordination Point Server 10.209.80.197 ..... Done Adding CPClient user for communicating to Coordination Point Server 10.209.80.197 .... Done Adding cluster clus1 to the CPClient user on Coordination Point Server 10.209.80.197 .. Done Updating /etc/vxfenmode file on sys1 .................................. Done Updating /etc/vxfenmode file on sys2 ......... ........................ Done
- Review the output as the installer stops and restarts the VCS and the fencing processes on each application cluster node, and completes the I/O fencing configuration.
- Configure the CP agent on the SFHA (application cluster).
Do you want to configure Coordination Point Agent on the client cluster? [y,n,q] (y) Enter a non-existing name for the service group for Coordination Point Agent: [b] (vxfen)
- Enable auto refresh of coordination points.
Do you want to enable auto refresh of coordination points if registration keys are missing on any of them? [y,n,q,b,?] (n)
- Note the location of the configuration log files, summary files, and response files that the installer displays for later use.