Introduction to SUSE Linux Enterprise High Availability

SUSE Linux Enterprise High Availability is available with the following products:

SUSE Linux Enterprise High Availability eliminates single points of failure to improve the availability and manageability of critical resources. This helps you maintain business continuity, protect data integrity, and reduce unplanned downtime for critical workloads.

The following figure shows a three-node cluster. Each of the nodes has a Web server installed and hosts two Web sites. All the data, graphics and Web page content for each Web site are stored on a shared disk subsystem connected to each of the nodes.

Figure 1: Three-node cluster #

 

During normal cluster operation, each node is in constant communication with the other nodes in the cluster and periodically checks resources to detect failure.

The following figure shows how the cluster moves resources when Web server 1 fails due to hardware or software problems.

Figure 2: Three-node cluster after one node fails #

 

Web site A moves to Web server 2 and Web site B moves to Web server 3. The Web sites continue to be available and are evenly distributed between the remaining cluster nodes.

When Web server 1 failed, the High Availability software performed the following tasks:

In this example, failover happened quickly and users regained access to the Web sites within seconds, usually without needing to log in again.

When Web server 1 returns to a normal operating state, Web site A and Web site B can move back (“fail back”) to Web server 1 automatically. This incurs some downtime, so alternatively you can configure resources to only fail back at a specified time that causes minimal service interruption.

A high-availability cluster is a group of servers that work together to ensure the availability of applications or services. Servers that are configured as members of the cluster are called nodes. If one node fails, the resources running on it can move to another node in the cluster. SUSE Linux Enterprise High Availability supports clusters with up to 32 nodes. However, clusters usually fall into one of two categories: two-node clusters, or clusters with an odd number of nodes (typically three or five).

Internal cluster communication is handled by Corosync. The Corosync Cluster Engine is a group communication system that provides messaging, membership and quorum information about the cluster. In SUSE Linux Enterprise High Availability 16, Corosync uses kronosnet (knet) as the default transport protocol.

We highly recommend configuring at least two communication channels for the cluster. The preferred method is to use network device bonding. If you cannot use a network bond, you can set up a redundant communication channel in Corosync (also known as a second “ring”).

In a High Availability cluster, the applications and services that need to be highly available are called resources. Cluster resources can include Web sites, e-mail servers, databases, file systems, virtual machines and any other server-based applications or services you want to make available to users at all times. You can start, stop, monitor and move resources as needed. You can also specify whether specific resources should run together on the same node, or start and stop in sequential order. If a cluster node fails, the resources running on it fail over (move) to another node instead of being lost.

In SUSE Linux Enterprise High Availability, the cluster resource manager (CRM) is Pacemaker, which manages and coordinates all cluster services. Pacemaker uses resource agents (RAs) to start, stop and monitor resources. A resource agent abstracts the resource it manages and presents its status to the cluster. SUSE Linux Enterprise High Availability supports many different resource agents that are designed to manage specific types of applications or services.

In a split-brain scenario, cluster nodes are divided into two or more groups (or partitions) that do not know about each other. This might be because of a hardware or software failure, or a failed network connection, for example. A split-brain scenario can be resolved by fencing (resetting or powering off) one or more of the nodes. Node fencing prevents a failed node from accessing shared resources and prevents cluster resources from running on a node with an uncertain status. This helps protect the cluster from data corruption.

SUSE Linux Enterprise High Availability uses STONITH as the node fencing mechanism. To be supported, all SUSE Linux Enterprise High Availability clusters must have at least one STONITH device. For critical workloads, we recommend using two or three STONITH devices. A STONITH device can be either a physical device (a power switch) or a software mechanism (SBD in combination with a watchdog).

When communication fails between one or more nodes and the rest of the cluster (a split-brain scenario), a cluster partition occurs. The nodes can only communicate with other nodes in the same partition and are unaware of the separated nodes. A cluster partition has quorum (or is “quorate”) if it has the majority of nodes (or “votes”). This is determined by quorum calculation. Quorum must be calculated so the non-quorate nodes can be fenced.

Corosync calculates quorum based on the following formula:

N ≥ C/2 + 1

N = minimum number of operational nodes
C = number of cluster nodes

For example, a five-node cluster needs a minimum of three operational nodes to maintain quorum.

Clusters with an even number of nodes, especially two-node clusters, might have equal numbers of nodes in each partition and therefore no majority. To avoid this situation, you can configure the cluster to use QDevice in combination with QNetd. QNetd is an arbitrator running outside the cluster. It communicates with the QDevice daemon running on the cluster nodes to provide a configurable number of votes for quorum calculation. This lets a cluster sustain more node failures than the standard quorum rules allow.

High Availability clusters might include a shared disk subsystem connected via Fibre Channel or iSCSI. If a node fails, another node in the cluster automatically mounts the shared disk directories that were previously mounted on the failed node. This gives users continuous access to the directories on the shared disk subsystem. Content stored on the shared disk might include data, applications and services.

The following figure shows what a typical Fibre Channel cluster configuration might look like. The green lines depict connections to an Ethernet power switch, which can reboot a node if a ping request fails.

Figure 3: Typical Fibre Channel cluster configuration #

 

The following figure shows what a typical iSCSI cluster configuration might look like.

Figure 4: Typical iSCSI cluster configuration #

 

Although most clusters include a shared disk subsystem, you can also create a cluster without a shared disk subsystem. The following figure shows what a cluster without a shared disk subsystem might look like.

Figure 5: Typical cluster configuration without shared storage #

 

SUSE Linux Enterprise High Availability has a layered architecture. The following figure shows the different layers and their associated components.

Figure 6: Architecture of a SUSE Linux Enterprise High Availability cluster #

 

Many actions performed in the cluster cause a cluster-wide change, for example, adding or removing a cluster resource or changing resource constraints. The following example explains what happens in the cluster when you perform such an action: