9 Customizing Crowbar #

In large deployments with many nodes, there are always some nodes that are in a fail or unknown state. New barclamps cannot be applied to them and values cannot be updated in some barclamps that are already deployed. This happens because Crowbar will refuse to apply a barclamp to a list of nodes if they are not all in ready state.

To avoid having to manually take out nodes that are not ready, there is a feature called skip unready nodes. Instead of refusing to apply the barclamp, it will skip the nodes that it finds in any other state than ready.

Enabling the Feature

In /opt/dell/crowbar_framework/config/crowbar.yml, set the option skip_unready_nodes to true.

default: &default
skip_unready_nodes:
  enabled: false <<< change to true

Roles Affected

All Barclamp roles are affected. The default config file includes all the roles that have been tested and found to be working. Adding roles to the default list is not supported for the skip_unready_nodes feature. Removing default roles is supported.

The list of currently supported roles to skip:

- bmc-nat-client
- ceilometer-agent
- deployer-client
- dns-client
- ipmi
- logging-client
- nova-compute-ironic
- nova-compute-kvm
- nova-compute-qemu
- nova-compute-vmware
- ntp-client
- provisioner-base
- suse-manager-client
- swift-storage
- updater

Determining Which Nodes Were Skipped

Skipped nodes are logged to the Crowbar log (/var/log/crowbar/production.log) where you can search for the text skipped until next chef run. This will print the log lines where nodes were skipped, the name of the node, and the barclamp which was being applied.

tux > grep "skipped until next chef run" /var/log/crowbar/production.log

Important

After enabling/disabling the skip_unready_nodes feature or adding/removing roles, the Crowbar framework service must be restarted (systemctl restart crowbar) in order to use the updated settings.

When a barclamp is applied, all nodes will run chef-client. Sometimes it is not necessary to run chef for each node as attributes or roles may have not changed for some of the nodes.

The skip unchanged nodes feature allows nodes to be skipped if their attributes or roles have not changed. This can help speed up applying a barclamp, especially in large deployments and with roles that are usually applied to a lot of nodes, such as nova.

Enabling the Feature

In /opt/dell/crowbar_framework/config/crowbar.yml, set the option skip_unchanged_nodes to true.

default: &default
skip_unchanged_nodes:
  enabled: false <<< change to true

When a service configuration has changed, Chef forces this service to restart. Sometimes it is useful to have manual control of these restarts. This feature supports avoiding automatic restart of services and including them in a pending restart list. Disabling restarts is enabled/disabled by barclamp and cannot be done on a service level. In other words, enabling this feature on the cinder barclamp will disable automatic restarts for all cinder-* services.

Two steps are necessary to activate this feature:

Enabling the Feature

Sometimes a change in a proposal requires services to be restarted on all implicated nodes. This could be a problem in a production environment where interrupting a service completely is not an option, and where manual restart control is needed. With the service disable_restart feature in crowbarctl, you can set a flag for certain proposals to prevent the automatic restart.

Enabling the Feature

Check Pending Restarts

When a proposal is applied with the disable_restart flag enabled, the implicated nodes will not restart. They will be listed as pending restarts. To check this list, run the command crowbarctl services list_restarts.

In the following example, disable_restart is enabled for RabbitMQ. After applying rabbitmq, list_restarts will show the affected nodes, the proposal, and the services to restart.

crowbarctl services list_restarts
{
   "d52-54-77-77-01-01.vo5.cloud.suse.de": {
    "alias": "controller1",
    "rabbitmq": {
      "rabbitmq-server": {
        "pacemaker_service": false,
        "timestamp": "2018-03-07 15:30:30 UTC"
      }
    }
  }
}

After a manual restart of the service, the flags should be cleaned. The following command will clean the flag for a specific node, for a specific proposal or all proposals, and for a specific service.

crowbarctl services clear_restart NODE [COOKBOOK [SERVICE]]

For example, to clean the RabbitMQ flag from the controller1 node, run the command:

crowbarctl service clear_restart controller1 rabbitmq

To clean the controller1 node, run the command:

crowbarctl service clear_restart controller1