Procedure 1.1. Procedure for Recovering from Compute Node Failure

1. If the Compute Node failed, it should have been fenced. Verify that this is the case. Otherwise, check /var/log/pacemaker.log on the Designated Coordinator to determine why the Compute Node was not fenced. The most likely reason is a problem with STONITH devices.

2. Determine the cause of the Compute Node's failure.

3. Rectify the root cause.

4. Boot the Compute Node again.

5. Check whether the crowbar_join script ran successfully on the Compute Node. If this is not the case, check the log files to find out the reason. Refer to the section called “On All Other Crowbar Nodes” to find the exact location of the log file.

6. If the chef-client agent triggered by crowbar_join succeeded, confirm that the pacemaker_remote service is up and running.

7. Check whether the remote node is registered and considered healthy by the core cluster. If this is not the case check /var/log/pacemaker.log on the Designated Coordinator to determine the cause. There should be a remote primitive running on the core cluster (active/passive). This primitive is responsible for establishing a TCP connection to the pacemaker_remote service on port 3121 of the Compute Node. Ensure that nothing is preventing this particular TCP connection from being established (for example, problems with NICs, switches, firewalls etc.). One way to do this is to run the following commands:

   tux > lsof -i tcp:3121
             tux > tcpdump tcp port 3121
           

8. If Pacemaker can communicate with the remote node, it should start the nova-compute service on it as part of the cloned group cl-g-nova-compute using the NovaCompute OCF resource agent. This cloned group will block startup of nova-evacuate until at least one clone is started.

   A necessary, related but different procedure is described in the section called “Bootstrapping the Compute Plane”.

9. It may happen that novaCompute has been launched correctly on the Compute Node by lrmd, but the openstack-nova-compute service is still not running. This usually happens when nova-evacuate did not run correctly.

   If nova-evacuate is not running on one of the core cluster nodes, make sure that the service is marked as started (target-role="Started"). If this is the case, then your cloud does not have any Compute Nodes already running as assumed by this procedure.

   If nova-evacuate is started but it is failing, check the Pacemaker logs to determine the cause.

   If nova-evacuate is started and functioning correctly, it should call nova's evacuate API to release resources used by the Compute Node and resurrect elsewhere any VMs that died when it failed.

10. If openstack-nova-compute is running, but VMs are not booted on the node, check that the service is not disabled or forced down using the openstack compute service list command. In case the service is disabled, run the openstack compute service set –enable SERVICE_ID command. If the service is forced down, run the following commands:

    tux > fence_nova_param () {
              key="$1"
              cibadmin -Q -A "//primitive[@id="fence-nova"]//nvpair[@name='$key']" | \
              sed -n '/.*value="/{s///;s/".*//;p}'
              }
              tux > fence_compute \
              --auth-url=`fence_nova_param auth-url` \
              --endpoint-type=`fence_nova_param endpoint-type` \
              --tenant-name=`fence_nova_param tenant-name` \
              --domain=`fence_nova_param domain` \
              --username=`fence_nova_param login` \
              --password=`fence_nova_param passwd` \
              -n COMPUTE_HOSTNAME \
              --action=on