Applies to SUSE Linux Enterprise High Availability 15 SP5

8 Managing cluster resources #

After configuring the resources in the cluster, use the cluster management tools to start, stop, clean up, remove or migrate the resources. This chapter describes how to use Hawk2 or crmsh for resource management tasks.

8.1 Showing cluster resources #

8.1.1 Showing cluster resources with `crmsh` #

When administering a cluster the command crm configure show lists the current CIB objects like cluster configuration, global options, primitives, and others:

# crm configure show
node 178326192: alice
node 178326448: bob
primitive admin_addr IPaddr2 \
       params ip=192.168.2.1 \
       op monitor interval=10 timeout=20
primitive stonith-sbd stonith:external/sbd \
       params pcmk_delay_max=30
property cib-bootstrap-options: \
       have-watchdog=true \
       dc-version=1.1.15-17.1-e174ec8 \
       cluster-infrastructure=corosync \
       cluster-name=hacluster \
       stonith-enabled=true \
       placement-strategy=balanced \
       standby-mode=true
rsc_defaults rsc-options: \
       resource-stickiness=1 \
       migration-threshold=3
op_defaults op-options: \
       timeout=600 \
       record-pending=true

If you have lots of resources, the output of show is too verbose. To restrict the output, use the name of the resource. For example, to list the properties of the primitive admin_addr only, append the resource name to show:

# crm configure show admin_addr
primitive admin_addr IPaddr2 \
       params ip=192.168.2.1 \
       op monitor interval=10 timeout=20

However, in some cases, you want to limit the output of specific resources even more. This can be achieved with filters. Filters limit the output to specific components. For example, to list the nodes only, use type:node:

# crm configure show type:node
node 178326192: alice
node 178326448: bob

If you are also interested in primitives, use the or operator:

# crm configure show type:node or type:primitive
node 178326192: alice
node 178326448: bob
primitive admin_addr IPaddr2 \
       params ip=192.168.2.1 \
       op monitor interval=10 timeout=20
primitive stonith-sbd stonith:external/sbd \
       params pcmk_delay_max=30

Furthermore, to search for an object that starts with a certain string, use this notation:

# crm configure show type:primitive and 'admin*'
primitive admin_addr IPaddr2 \
       params ip=192.168.2.1 \
       op monitor interval=10 timeout=20

To list all available types, enter crm configure show type: and press the →| key. The Bash completion gives you a list of all types.

8.2 Editing resources and groups #

You can edit resources or groups using either Hawk2 or crmsh.

8.2.1 Editing resources and groups with Hawk2 #

If you have created a resource, you can edit its configuration at any time by adjusting parameters, operations or meta attributes as needed.

Procedure 8.1: Modifying a resource or group #

Log in to Hawk2:
```
https://HAWKSERVER:7630/
```
On the Hawk2 Status screen, go to the Resources list.
In the Operations column, click the arrow down icon next to the resource or group you want to modify and select Edit.
The resource configuration screen opens.
Figure 8.1: Hawk2—editing a primitive resource #
At the top of the configuration screen, you can select operations to perform.
If you edit a primitive resource, the following operations are available:
- Copying the resource
- Renaming the resource (changing its ID)
- Deleting the resource
If you edit a group, the following operations are available:
- Creating a new primitive to be added to this group
- Renaming the group (changing its ID)
- Dragging and dropping group members into a new order
To add a new parameter, operation or meta attribute, select an entry from the empty drop-down box.
To edit any values in the Operations category, click the Edit icon of the respective entry, enter a different value for the operation, and click Apply.
When you are finished, click the Apply button in the resource configuration screen to confirm your changes to the parameters, operations or meta attributes.
A message at the top of the screen shows if the action has been successful.

8.2.2 Editing groups with `crmsh` #

To change the order of a group member, use the modgroup command from the configure subcommand. For example, use the following command to move the primitive Email before Public-IP:

crm(live)configure# modgroup g-mailsvc add Email before Public-IP

To remove a resource from a group (for example, Email), use this command:

crm(live)configure# modgroup g-mailsvc remove Email

8.3 Starting cluster resources #

Before you start a cluster resource, make sure it is set up correctly. For example, if you use an Apache server as a cluster resource, set up the Apache server first. Complete the Apache configuration before starting the respective resource in your cluster.

Note: Do not touch services managed by the cluster

When managing a resource via the High Availability software, the resource must not be started or stopped otherwise (outside the cluster, for example manually or on boot or reboot). The High Availability software is responsible for all service start or stop actions.

However, if you want to check if the service is configured properly, start it manually, but make sure that it is stopped again before the High Availability software takes over.

For interventions in resources that are currently managed by the cluster, set the resource to maintenance mode first. For details, see Procedure 28.5, “Putting a resource into maintenance mode with Hawk2”.

You can start a cluster resource using either Hawk2 or crmsh.

8.3.1 Starting cluster resources with Hawk2 #

When creating a resource with Hawk2, you can set its initial state with the target-role meta attribute. If you set its value to stopped, the resource does not start automatically after being created.

Procedure 8.2: Starting a new resource #

Log in to Hawk2:
```
https://HAWKSERVER:7630/
```
From the left navigation bar, select Monitoring › Status. The list of Resources also shows the Status.
Select the resource to start. In its Operations column, click the Start icon. To continue, confirm the message that appears.

When the resource has started, Hawk2 changes the resource's Status to green and shows on which node it is running.

8.3.2 Starting cluster resources with `crmsh` #

To start a new cluster resource, you need the respective identifier.

Procedure 8.3: Starting a cluster resource with crmsh #

Log in as root and start the crm interactive shell:
```
# crm
```
Switch to the resource level:
```
crm(live)# resource
```
Start the resource with start and press the →| key to show all known resources:
```
crm(live)resource# start ID
```

8.4 Stopping cluster resources #

8.4.1 Stopping cluster resources with `crmsh` #

To stop one or more existing cluster resources, you need the respective identifiers.

Procedure 8.4: Stopping cluster resources with crmsh #

Log in as root and start the crm interactive shell:
```
# crm
```
Switch to the resource level:
```
crm(live)# resource
```
Stop the resource with stop and press the →| key to show all known resources:
```
crm(live)resource# stop ID
```
You can stop multiple resources at once:
```
crm(live)resource# stop ID1 ID2 ...
```

8.5 Cleaning up cluster resources #

A resource is automatically restarted if it fails, but each failure increases the resource's fail count.

If a migration-threshold has been set for the resource, the node will no longer run the resource when the number of failures reaches the migration threshold.

By default, fail counts are not automatically reset. You can configure a fail count to be reset automatically by setting a failure-timeout option for the resource, or you can manually reset the fail count using either Hawk2 or crmsh.

8.5.1 Cleaning up cluster resources with Hawk2 #

Procedure 8.5: Cleaning up a resource #

Log in to Hawk2:
```
https://HAWKSERVER:7630/
```
From the left navigation bar, select Status. The list of Resources also shows the Status.
Go to the resource to clean up. In the Operations column, click the arrow down button and select Cleanup. To continue, confirm the message that appears.
This executes the command crm resource cleanup and cleans up the resource on all nodes.

8.5.2 Cleaning up cluster resources with `crmsh` #

Procedure 8.6: Cleaning up a resource with crmsh #

Open a shell and log in as user root.

Get a list of all your resources:

# crm resource status
Full List of Resources
   * admin-ip      (ocf:heartbeat:IPaddr2):    Started
   * stonith-sbd   (stonith:external/sbd):     Started
   * Resource Group: dlm-clvm:
     * dlm:        (ocf:pacemaker:controld)    Started
     * clvm:       (ocf:heartbeat:lvmlockd)    Started

Show the fail count of a resource:

# crm resource failcount RESOURCE show NODE

For example, to show the fail count of the resource dlm on node alice:

# crm resource failcount dlm show alice
scope=status name=fail-count-dlm value=2

Clean up the resource:
```
# crm resource cleanup RESOURCE
```
This command cleans up the resource on all nodes. If the resource is part of a group, crmsh also cleans up the other resources in the group.

8.6 Removing cluster resources #

To remove a resource from the cluster, follow either the Hawk2 or crmsh procedure below to avoid configuration errors.

8.6.1 Removing cluster resources with Hawk2 #

Procedure 8.7: Removing a cluster resource #

Log in to Hawk2:
```
https://HAWKSERVER:7630/
```
Clean up the resource on all nodes as described in Procedure 8.5, “Cleaning up a resource”.
Stop the resource:
1. From the left navigation bar, select Monitoring › Status. The list of Resources also shows the Status.
2. In the Operations column, click the Stop button next to the resource.
3. To continue, confirm the message that appears.
  The Status column will reflect the change when the resource is stopped.
Delete the resource:
1. From the left navigation bar, select Configuration › Edit Configuration.
2. In the list of Resources, go to the respective resource. From the Operations column, click the Delete icon next to the resource.
3. To continue, confirm the message that appears.

8.6.2 Removing cluster resources with `crmsh` #

Procedure 8.8: Removing a cluster resource with crmsh #

Get a list of your resources:

crm(live)# resource status
Full List of Resources:
  * admin-ip     (ocf:heartbeat:IPaddr2):     Started
  * stonith-sbd  (stonith:external/sbd):      Started
  * nfsserver    (ocf:heartbeat:nfsserver):   Started

Stop the resource you want to remove:
```
crm(live)# resource stop RESOURCE
```
Delete the resource:
```
crm(live)# configure delete RESOURCE
```

8.7 Migrating cluster resources #

The cluster will fail over (migrate) resources automatically in case of software or hardware failures, according to certain parameters you can define (for example, migration threshold or resource stickiness). You can also manually migrate a resource to another node in the cluster, or move the resource away from the current node and let the cluster decide where to put it.

You can migrate a cluster resource using either Hawk2 or crmsh.

8.7.1 Migrating cluster resources with Hawk2 #

Procedure 8.9: Manually migrating a resource #

Log in to Hawk2:
```
https://HAWKSERVER:7630/
```
From the left navigation bar, select Monitoring › Status. The list of Resources also shows the Status.
In the list of Resources, select the respective resource.
In the Operations column, click the arrow down button and select Migrate.
In the window that opens, you have the following choices:
- Away from current node: This creates a location constraint with a -INFINITY score for the current node.
- Alternatively, you can move the resource to another node. This creates a location constraint with an INFINITY score for the destination node.
Confirm your choice.

To allow a resource to move back again, proceed as follows:

Procedure 8.10: Unmigrating a resource #

Log in to Hawk2:
```
https://HAWKSERVER:7630/
```
From the left navigation bar, select Monitoring › Status. The list of Resources also shows the Status.
In the list of Resources, go to the respective resource.
In the Operations column, click the arrow down button and select Clear. To continue, confirm the message that appears.
Hawk2 uses the crm_resource --clear command. The resource can move back to its original location or it may stay where it is (depending on resource stickiness).

For more information, see Pacemaker Explained, available from https://www.clusterlabs.org/pacemaker/doc/. Refer to section Resource Migration.

8.7.2 Migrating cluster resources with `crmsh` #

Use the move command for this task. For example, to migrate the resource ipaddress1 to a cluster node named bob, use these commands:

# crm resource
 crm(live)resource# move ipaddress1 bob

8.8 Grouping resources by using tags #

Tags are a way to refer to multiple resources at once, without creating any colocation or ordering relationship between them. This can be useful for grouping conceptually related resources. For example, if you have several resources related to a database, create a tag called databases and add all resources related to the database to this tag. This allows you to stop or start them all with a single command.

Tags can also be used in constraints. For example, the following location constraint loc-db-prefer applies to the set of resources tagged with databases:

location loc-db-prefer databases 100: alice

You can create tags using either Hawk2 or crmsh.

8.8.1 Grouping resources by using tags with Hawk2 #

Procedure 8.11: Adding a tag #

Log in to Hawk2:
```
https://HAWKSERVER:7630/
```
From the left navigation bar, select Configuration › Add Resource › Tag.
Enter a unique Tag ID.
From the Objects list, select the resources you want to refer to with the tag.
Click Create to finish the configuration. A message at the top of the screen shows if the action has been successful.

Figure 8.2: Hawk2—tag #

8.8.2 Grouping resources by using tags with `crmsh` #

For example, if you have several resources related to a database, create a tag called databases and add all resources related to the database to this tag:

# crm configure tag databases: db1 db2 db3

This allows you to start them all with a single command:

# crm resource start databases

Similarly, you can stop them all too:

# crm resource stop databases