These commands help you to deploy or upgrade the Ceph cluster, run commands on several (or all) cluster nodes at the same time, or assist you when adding or removing cluster nodes. The most frequently used are salt, salt-run, and deepsea. You need to run Salt commands on the Salt master node (refer to Book “Deployment Guide”, Chapter 5 “Deploying with DeepSea/Salt”, Section 5.2 “Introduction to DeepSea” for details) as root. These commands are introduced with the following prompt:

root@master # 

For example:

root@master # salt '*.example.net' test.ping

These are lower level commands to configure and fine tune all aspects of the cluster and its gateways on the command line, for example ceph, rbd, radosgw-admin, or crushtool.

To run Ceph related commands, you need to have read access to a Ceph key. The key's capabilities then define your privileges within the Ceph environment. One option is to run Ceph commands as root (or via sudo) and use the unrestricted default keyring 'ceph.client.admin.key'.

Safer and recommended option is to create a more restrictive individual key for each administrator user and put it in a directory where the users can read it, for example:

~/.ceph/ceph.client.USERNAME.keyring

Tip: Path to Ceph Keys

To use a custom admin user and keyring, you need to specify the user name and path to the key each time you run the ceph command using the -n client.USER_NAME and --keyring PATH/TO/KEYRING options.

To avoid this, include these options in the CEPH_ARGS variable in the individual users' ~/.bashrc files.

Although you can run Ceph related commands on any cluster node, we recommend running them on the Admin Node. This documentation uses the cephadm user to run the commands, therefore they are introduced with the following prompt:

cephadm@adm > 

For example:

cephadm@adm > ceph auth list

Tip: Commands for Specific Nodes

If the documentation instructs you to run a command on a cluster node with a specific role, it will be addressed by the prompt. For example:

cephadm@mon > 

Linux commands not related to Ceph or DeepSea, such as mount, cat, or openssl, are introduced either with the cephadm@adm > or root # prompts, depending on which privileges the related command requires.

For more information on Ceph key management, refer to Section 19.2, “Key Management”.

The procedure of adding new nodes to the cluster is almost identical to the initial cluster node deployment described in Book “Deployment Guide”, Chapter 5 “Deploying with DeepSea/Salt”:

Tip: Prevent Rebalancing

When adding an OSD to the existing cluster, bear in mind that the cluster will be rebalancing for some time afterward. To minimize the rebalancing periods, add all OSDs you intend to add at the same time.

An additional way is to set the osd crush initial weight = 0 option in the ceph.conf file before adding the OSDs:

1. Add osd crush initial weight = 0 to /srv/salt/ceph/configuration/files/ceph.conf.d/global.conf.

2. Create the new configuration on the Salt master node:

   root@master # salt 'SALT_MASTER_NODE' state.apply ceph.configuration.create

   Or:

   root@master # salt-call state.apply ceph.configuration.create

3. Apply the new configuration to the targeted OSD minions:

   root@master # salt 'OSD_MINIONS' state.apply ceph.configuration

   

   Note

   If this is not a new node, but you want to proceed as if it were, ensure you remove the /etc/ceph/destroyedOSDs.yml file from the node. Otherwise, any devices from the first attempt will be restored with their previous OSD ID and reweight.

   Run the following commands:

   root@master # salt-run state.orch ceph.stage.1
   root@master # salt-run state.orch ceph.stage.2
   root@master # salt 'node*' state.apply ceph.osd

4. After the new OSDs are added, adjust their weights as required with the ceph osd crush reweight command in small increments. This allows the cluster to rebalance and become healthy between increasing increments so it does not overwhelm the cluster and clients accessing the cluster.

You can deploy all types of supported roles with DeepSea. See Book “Deployment Guide”, Chapter 5 “Deploying with DeepSea/Salt”, Section 5.5.1.2 “Role Assignment” for more information on supported role types and examples of matching them.

To add a new service to an existing node, follow these steps:

Tip: Removing a Cluster Node Temporarily

The Salt master expects all minions to be present in the cluster and responsive. If a minion breaks and is not responsive anymore, it causes problems to the Salt infrastructure, mainly to DeepSea and Ceph Dashboard.

Before you fix the minion, delete its key from the Salt master temporarily:

root@master # salt-key -d MINION_HOST_NAME

After the minion is fixed, add its key to the Salt master again:

root@master # salt-key -a MINION_HOST_NAME

To remove a role from a cluster, edit /srv/pillar/ceph/proposals/policy.cfg and remove the corresponding line(s). Then run stages 2 and 5 as described in Book “Deployment Guide”, Chapter 5 “Deploying with DeepSea/Salt”, Section 5.3 “Cluster Deployment”.

Note: Removing OSDs from Cluster

In case you need to remove a particular OSD node from your cluster, ensure that your cluster has more free disk space than the disk you intend to remove. Bear in mind that removing an OSD results in rebalancing of the whole cluster.

Before running stage 5 to do the actual removal, always check which OSDs are going to be removed by DeepSea:

root@master # salt-run rescinded.ids

When a role is removed from a minion, the objective is to undo all changes related to that role. For most of the roles, the task is simple, but there may be problems with package dependencies. If a package is uninstalled, its dependencies are not.

Removed OSDs appear as blank drives. The related tasks overwrite the beginning of the file systems and remove backup partitions in addition to wiping the partition tables.

Note: Preserving Partitions Created by Other Methods

Disk drives previously configured by other methods, such as ceph-deploy, may still contain partitions. DeepSea will not automatically destroy these. The administrator must reclaim these drives manually.

[...]
# Hardware Profile
role-storage/cluster/data*.sls
[...]

[...]
# Hardware Profile
role-storage/cluster/data[1,3-6]*.sls
[...]

    [...]
    drive_group_name:
      target: 'data[1,3-6]*'
    [...]

root@master # salt-run state.orch ceph.stage.2
root@master # salt-run rescinded.ids
root@master # salt-run state.orch ceph.stage.5

# Hardware Profile
role-storage/cluster/data[1-7]*.sls

# Roles
role-rgw/cluster/data8*.sls

# Hardware Profile
role-storage/cluster/data[1-8]*.sls

# Roles
role-rgw/cluster/rgw1*.sls

root@master # salt-run state.orch ceph.stage.2
root@master # salt-run state.orch ceph.stage.3
root@master # salt-run state.orch ceph.stage.4
root@master # salt-run rescinded.ids
root@master # salt-run state.orch ceph.stage.5

root@master # salt-key -d MINION_ID

cephadm@adm > ceph osd purge-actual $id --yes-i-really-mean-it
cephadm@adm > ceph auth del osd.$id

cephadm@adm > ceph destroy $id
cephadm@adm > ceph osd rm $id
cephadm@adm > ceph osd crush remove osd.$id

When one or more of your monitor nodes fail and are not responding, you need to remove the failed monitors from the cluster and possibly then re-add them back in the cluster.

Important: The Minimum Is Three Monitor Nodes

The number of monitor nodes must not be less than three. If a monitor node fails, and as a result your cluster has only two monitor nodes, you need to temporarily assign the monitor role to other cluster nodes before you redeploy the failed monitor nodes. After you redeploy the failed monitor nodes, you can uninstall the temporary monitor roles.

For more information on adding new nodes/roles to the Ceph cluster, see Section 2.1, “Adding New Cluster Nodes” and Section 2.2, “Adding New Roles to Nodes”.

For more information on removing cluster nodes, refer to Section 2.3, “Removing and Reinstalling Cluster Nodes”.

There are two basic degrees of a Ceph node failure:

After using DeepSea to deploy an OSD, you may want to verify that the OSD is encrypted.

To add a disk to an existing OSD node, verify that any partition on the disk was removed and wiped. Refer to Step 12 in Book “Deployment Guide”, Chapter 5 “Deploying with DeepSea/Salt”, Section 5.3 “Cluster Deployment” for more details. Adapt /srv/salt/ceph/configuration/files/drive_groups.yml accordingly (refer to Book “Deployment Guide”, Chapter 5 “Deploying with DeepSea/Salt”, Section 5.5.2 “DriveGroups” for details). After saving the file, run DeepSea's stage 3:

root@master # deepsea stage run ceph.stage.3

You can remove a Ceph OSD from the cluster by running the following command:

root@master # salt-run osd.remove OSD_ID

OSD_ID needs to be a number of the OSD without the osd. prefix. For example, from osd.3 only use the digit 3.

root@master # salt-run osd.remove 2 6 11 15
Removing osd 2 on host data1
Draining the OSD
Waiting for ceph to catch up.
osd.2 is safe to destroy
Purging from the crushmap
Zapping the device


Removing osd 6 on host data1
Draining the OSD
Waiting for ceph to catch up.
osd.6 is safe to destroy
Purging from the crushmap
Zapping the device


Removing osd 11 on host data1
Draining the OSD
Waiting for ceph to catch up.
osd.11 is safe to destroy
Purging from the crushmap
Zapping the device


Removing osd 15 on host data1
Draining the OSD
Waiting for ceph to catch up.
osd.15 is safe to destroy
Purging from the crushmap
Zapping the device


2:
True
6:
True
11:
True
15:
True

root@master # salt-run osd.remove OSD_HOST_NAME

2.7.3 Removing Broken OSDs Forcefully #

There are cases when removing an OSD gracefully (see Section 2.7, “Removing an OSD”) fails. This may happen, for example, if the OSD or its journal, WAL or DB are broken, when it suffers from hanging I/O operations, or when the OSD disk fails to unmount.

root@master # salt-run osd.remove OSD_ID force=True

Tip: Hanging Mounts

If a partition is still mounted on the disk being removed, the command will exit with the 'Unmount failed - check for processes on DEVICE' message. You can then list all processes that access the file system with the fuser -m DEVICE. If fuser returns nothing, try manual unmount DEVICE and watch the output of dmesg or journalctl commands.

There are several reasons why you may need to replace an OSD disk, for example:

The replacement procedure is the same for both cases. It is also valid for both default and customized CRUSH Maps.

Important: Shared device failure

If a shared device for DB/WAL fails, you need to perform the replacement procedure for all OSDs that share the failed device.

If the operating system breaks and is not recoverable on one of your OSD nodes, follow these steps to recover it and redeploy its OSD role with cluster data untouched:

If you need to replace the Admin Node host with a new one, you need to move the Salt master and DeepSea files. Use your favorite synchronization tool for transferring the files. In this procedure, we use rsync because it is a standard tool available in SUSE Linux Enterprise Server 15 SP1 software repositories.

The installation can be automated by using the Salt reactor. For virtual environments or consistent hardware environments, this configuration will allow the creation of a Ceph cluster with the specified behavior.

Warning

Salt cannot perform dependency checks based on reactor events. There is a real risk of putting your Salt master into a death spiral.

The automated installation requires the following:

The default reactor configuration will only run stages 0 and 1. This allows testing of the reactor without waiting for subsequent stages to complete.

When the first salt-minion starts, stage 0 will begin. A lock prevents multiple instances. When all minions complete stage 0, stage 1 will begin.

If the operation is performed properly, edit the file

/etc/salt/master.d/reactor.conf

and replace the following line

- /srv/salt/ceph/reactor/discovery.sls

with

- /srv/salt/ceph/reactor/all_stages.sls

Verify that the line is not commented out.

Keep the Ceph cluster nodes up-to-date by applying rolling updates regularly.

update_method_init: zypper-patch

In some cases it may be necessary to halt or reboot the whole cluster. We recommended carefully checking for dependencies of running services. The following steps provide an outline for stopping and starting the cluster:

If you need to put custom settings into the ceph.conf file, you can do so by modifying the configuration files in the /srv/salt/ceph/configuration/files/ceph.conf.d directory:

Note: Unique rgw.conf

The Object Gateway offers a lot of flexibility and is unique compared to the other ceph.conf sections. All other Ceph components have static headers such as [mon] or [osd]. The Object Gateway has unique headers such as [client.rgw.rgw1]. This means that the rgw.conf file needs a header entry. For examples, see

/srv/salt/ceph/configuration/files/rgw.conf

or

/srv/salt/ceph/configuration/files/rgw-ssl.conf

See Section 26.7, “Enabling HTTPS/SSL for Object Gateways” for more examples.

Important: Run stage 3

After you make custom changes to the above mentioned configuration files, run stages 3 and 4 to apply these changes to the cluster nodes:

root@master # salt-run state.orch ceph.stage.3
root@master # salt-run state.orch ceph.stage.4

These files are included from the /srv/salt/ceph/configuration/files/ceph.conf.j2 template file, and correspond to the different sections that the Ceph configuration file accepts. Putting a configuration snippet in the correct file enables DeepSea to place it into the correct section. You do not need to add any of the section headers.

Tip

To apply any configuration options only to specific instances of a daemon, add a header such as [osd.1]. The following configuration options will only be applied to the OSD daemon with the ID 1.

auth cluster required = none
auth service required = none
auth client required = none

root@master # salt-run state.orch ceph.configuration.create

2.14.2 Including Configuration Files #

If you need to apply a lot of custom configurations, use the following include statements within the custom configuration files to make file management easier. Following is an example of the osd.conf file:

[osd.1]
{% include "ceph/configuration/files/ceph.conf.d/osd1.conf" ignore missing %}
[osd.2]
{% include "ceph/configuration/files/ceph.conf.d/osd2.conf" ignore missing %}
[osd.3]
{% include "ceph/configuration/files/ceph.conf.d/osd3.conf" ignore missing %}
[osd.4]
{% include "ceph/configuration/files/ceph.conf.d/osd4.conf" ignore missing %}

In the previous example, the osd1.conf, osd2.conf, osd3.conf, and osd4.conf files contain the configuration options specific to the related OSD.

Tip: Runtime Configuration

Changes made to Ceph configuration files take effect after the related Ceph daemons restart. See Section 25.1, “Runtime Configuration” for more information on changing the Ceph runtime configuration.

AppArmor is a security solution that confines programs by a specific profile. For more details, refer to https://documentation.suse.com/sles/15-SP1/html/SLES-all/part-apparmor.html.

DeepSea provides three states for AppArmor profiles: 'enforce', 'complain', and 'disable'. To activate a particular AppArmor state, run:

salt -I "deepsea_minions:*" state.apply ceph.apparmor.default-STATE

To put the AppArmor profiles in an 'enforce' state:

root@master # salt -I "deepsea_minions:*" state.apply ceph.apparmor.default-enforce

To put the AppArmor profiles in a 'complain' state:

root@master # salt -I "deepsea_minions:*" state.apply ceph.apparmor.default-complain

To disable the AppArmor profiles:

root@master # salt -I "deepsea_minions:*" state.apply ceph.apparmor.default-disable

Tip: Enabling the AppArmor Service

Each of these three calls verifies if AppArmor is installed and installs it if not, and starts and enables the related systemd service. DeepSea will warn you if AppArmor was installed and started/enabled in another way and therefore runs without DeepSea profiles.

By default, DeepSea deploys Ceph clusters with tuned profiles active on Ceph Monitor, Ceph Manager, and Ceph OSD nodes. In some cases, you may need to permanently deactivate tuned profiles. To do so, put the following lines in /srv/pillar/ceph/stack/global.yml and re-run stage 3:

alternative_defaults:
 tuned_mgr_init: default-off
 tuned_mon_init: default-off
 tuned_osd_init: default-off

root@master # salt-run state.orch ceph.stage.3

The ceph.purge runner removes the entire Ceph cluster. This way you can clean the cluster environment when testing different setups. After the ceph.purge completes, the Salt cluster is reverted back to the state at the end of DeepSea stage 1. You can then either change the policy.cfg (see Book “Deployment Guide”, Chapter 5 “Deploying with DeepSea/Salt”, Section 5.5.1 “The policy.cfg File”), or proceed to DeepSea stage 2 with the same setup.

To prevent accidental deletion, the orchestration checks if the safety is disengaged. You can disengage the safety measures and remove the Ceph cluster by running:

root@master # salt-run disengage.safety
root@master # salt-run state.orch ceph.purge

Tip: Disabling Ceph Cluster Removal

If you want to prevent anyone from running the ceph.purge runner, create a file named disabled.sls in the /srv/salt/ceph/purge directory and insert the following line in the /srv/pillar/ceph/stack/global.yml file:

purge_init: disabled

Important: Rescind Custom Roles

If you previously created custom roles for Ceph Dashboard (refer to Section 6.6, “Adding Custom Roles” and Section 14.2, “User Roles and Permissions” for detailed information), you need to take manual steps to purge them before running the ceph.purge runner. For example, if the custom role for Object Gateway is named 'us-east-1', then follow these steps:

root@master # cd /srv/salt/ceph/rescind
root@master # rsync -a rgw/ us-east-1
root@master # sed -i 's!rgw!us-east-1!' us-east-1/*.sls

Back up the /etc/ceph directory. It contains crucial cluster configuration. You will need the backup of /etc/ceph for example when you need to replace the Admin Node.

You need to back up the /etc/salt/ directory. It contains the Salt configuration files, for example the Salt master key and accepted client keys.

The Salt files are not strictly required for backing up the Admin Node, but make redeploying the Salt cluster easier. If there is no backup of these files, the Salt minions need to be registered again at the new Admin Node.

Note: Security of the Salt Master Private Key

Make sure that the backup of the Salt master private key is stored in a safe location. The Salt master key can be used to manipulate all cluster nodes.

After restoring the /etc/salt directory from a backup, restart the Salt services:

root@master # systemctl restart salt-master
root@master # systemctl restart salt-minion

All files required by DeepSea are stored in /srv/pillar/, /srv/salt/ and /etc/salt/master.d.

If you need to redeploy the Admin Node, install the DeepSea package on the new node and move the backed up data back into the directories. DeepSea can then be used again without any further changes being required. Before using DeepSea again, make sure that all Salt minions are correctly registered on the Admin Node.

To log in to the dashboard Web application, point your browser to its URL including the port number. You can find its address by running

cephadm@adm > ceph mgr services | grep dashboard
"dashboard": "https://ses-dash-node.example.com:8443/",

Figure 5.1: Ceph Dashboard Login Screen #

You need a user account in order to log in to the dashboard Web application. DeepSea creates a default user 'admin' with administrator privileges for you. If you decide to log in with the default 'admin' user, retrieve the corresponding password by running

root@master # salt-call grains.get dashboard_creds

Tip: Custom User Account

If you do not want to use the default 'admin' account to access the Ceph Dashboard, create a custom user account with administrator privileges. Refer to Chapter 14, Managing Users and Roles on the Command Line for more details.

The dashboard user interface is graphically divided into several blocks: the utility menu, the main menu, and the main content pane.

Figure 5.2: Ceph Dashboard Home Page #

The top right part of the screen contains a utility menu. It includes general tasks related more to the dashboard than to the Ceph cluster. By clicking its items, you can access the following topics:

The dashboard's main menu occupies the top left part of the screen. It covers the following topics:

The content pane occupies the main part of the dashboard's screen. The dashboard home page shows plenty of helpful widgets to inform you briefly about the current status of the cluster, capacity, and performance information.

In Ceph Dashboard, you often work with lists—for example, lists of pools, OSD nodes, or RBD devices. All lists will automatically refresh themselves by default every 5 seconds. The following common widgets help you manage or adjust these list:

Click to trigger a manual refresh of the list.

Click to display or hide individual table columns.

Click and select how many rows to display on a single page.

Click inside and filter the rows by typing the string to search for.

Use to change the currently displayed page if the list spans across multiple pages.

Each dashboard widget shows specific status information related to a specific aspect of a running Ceph cluster. Some widgets are active links and after clicking them, they will redirect you to a related detailed page of the topic they represent.

Tip: More Details on Mouse Over

Some graphical widgets show you more detail when you move the mouse over them.

5.6.1 Status Widgets #

Status widgets give you a brief overview about the cluster's current status.

Figure 5.3: Status Widgets #

Cluster Status

Presents basic information about the cluster's health.

Monitors

Shows the number of running monitors and their quorum.

OSDs

Shows the total number of OSDs, as well as the number of 'up' and 'in' OSDs.

Manager Daemons

Shows the number of active and standby Ceph Manager daemons.

Hosts

Shows the total number of cluster nodes.

Object Gateways

Shows the number of running Object Gateways.

Metadata Servers

Shows the number of Metadata Servers.

iSCSI Gateway

Shows the number of configured iSCSI gateways.

5.6.2 Performance Widgets #

Performance widgets refer to basic performance data of Ceph clients.

Figure 5.4: performance Widgets #

Client IOPS

The amount of clients' read and write operations per second.

Client Throughput

The sum of clients' read and write operations per second.

Client Read/Write

Visualizes clients' read/write ratio.

Recovery Throughput

The throughput of data recovered per second.

Scrub

Shows the scrub (see Section 20.4.9, “Scrubbing a Placement Group”) status. It is either disabled, enabled, or active.

5.6.3 Capacity Widgets #

Capacity widgets show brief information about the storage capacity.

Figure 5.5: Capacity Widgets #

Pools

Shows the number of pools in the cluster.

Raw Capacity

Shows the ratio of used and available raw storage capacity.

Objects

Shows the number of data objects stored in the cluster.

PGs per OSD

Shows the number of placement groups per OSD.

PG Status

Displays a chart of the placement groups according to their status.

Click in the utility menu and select User Management.

The list contains each user's user name, full name, e-mail, and list of assigned roles.

Figure 6.1: User Management #

Click Add in the top left of the table heading to add a new user. Enter their user name, password, and optionally a full name and an e-mail.

Figure 6.2: Adding a User #

Click the little pen icon to assign predefined roles to the user. Confirm with Create User.

Click a user's table row and then select Edit to edit details about the user. Confirm with Update User.

Click a user's table row and then select Delete to delete the user account. Activate the Yes, I am sure check box and confirm with Delete User.

Click in the utility menu and select User Management. Then click the Roles tab.

The list contains each role's name, description, and whether it is a system role.

Figure 6.3: User Roles #

Click Add in the top left of the table heading to add a new custom role. Enter its name, description, and set required permissions for individual topics.

Tip: Purging Custom Roles

If you create custom user roles and intend to remove the Ceph cluster with the ceph.purge runner later on, you need to purge the custom roles first. Find more details in Section 2.17, “Removing an Entire Ceph Cluster”.

Figure 6.4: Adding a Role #

Tip: Multiple Activation

By activating the check box that precedes the topic name, you activate all permissions for that topic. By activating the All check box, you activate all permissions for all the topics.

Confirm with Create Role.

Click a custom role's table row and then select Edit in the top left of the table heading to edit a description and permissions of the custom role. Confirm with Update Role.

Click a role's table row and then select Delete to delete the role. Activate the Yes, I am sure check box and confirm with Delete Role.

Click Cluster / Hosts to view a list of cluster nodes.

Figure 7.1: Hosts #

Click a node name in the Hostname column to view performance details of the node.

The Services column lists all daemons that are running on each related node. Click a daemon name to view its detailed configuration.

Click Cluster / Monitors to view a list of cluster nodes with running Ceph monitors. The list includes each monitor's rank number, public IP address, and number of open sessions.

The list is divided into two tables: one for Ceph Monitors that are in quorum, and the second one for Ceph Monitors that are not in quorum.

Click a node name in the Name column to view the related Ceph Monitor configuration.

The Status table shows general statistics about the running Ceph Monitors.

Figure 7.2: Ceph Monitors #

Click Cluster / OSDs to view a list of nodes with running OSD daemons/disks. The list includes each node's name, status, number of placement groups, size, usage, reads/writes chart in time, and the rate of read/write operations per second.

Figure 7.3: Ceph OSDs #

Click Set Cluster-wide Flags in the table heading to pop up a window with a list of flags that apply to the whole cluster. You can activate or deactivate individual flags, and confirm with Submit.

Figure 7.4: OSD Flags #

Click Set Cluster-wide Recovery Priority in the table heading to open a pop-up window with a list of OSD recovery priorities that apply to the whole cluster. You can activate the preferred priority profile, and fine tune the individual values below. Confirm with Submit.

Figure 7.5: OSD Recovery Priority #

Click a node name in the Host column to view an extended table with details about the OSD setting and performance. Browsing through several tabs, you can see lists of Attributes, Metadata, Performance counter, and a graphical Histogram of reads and writes.

Figure 7.6: OSD Details #

Tip: Perform Specific Tasks on OSDs

After you click an OSD node name, its table row changes color slightly, meaning that you can now perform a task on the node. Click the down arrow in the top left of the table heading and select a task to perform, such as Deep Scrub. Optionally enter a required value for the task, and confirm to run it on the node.

Click Cluster / Configuration to view a complete list of Ceph cluster configuration options. The list contains the name of the option, its short description, and its current and default values.

Figure 7.7: Cluster Configuration #

Click a specific option row to highlight and see detailed information about the option, such as its type of value, minimum and maximum permitted values, whether it can be updated at runtime, and many more.

After highlighting a specific option, you can edit its value(s) by clicking the Edit button in the top left of the table heading. Confirm changes with Save.

Click Cluster / CRUSH map to view a CRUSH Map of the cluster. For more general information on CRUSH Maps, refer to Section 20.5, “CRUSH Map Manipulation”.

Click the root, nodes, or individual OSDs to view more detailed information, such as crush weight, depth in the map tree, device class of the OSD, and many more.

Figure 7.8: CRUSH Map #

Click Cluster / Manager modules to view a list of available Ceph Manager modules. Each line consists of a module name and information on whether it is currently enabled or not.

Figure 7.9: Manager Modules #

After highlighting a specific module, you can see its detailed settings in the Details table below. Edit them by clicking the Edit button in the top left of the table heading. Confirm changes with Update.

Click Cluster / Logs to view a list of cluster's recent log entries. Each line consists of a time stamp, the type of the log entry, and the logged message itself.

Click the Audit Logs tab to view log entries of the auditing subsystem. Refer to Section 14.4, “Auditing” for commands to enable or disable auditing.

Figure 7.10: Logs #

To add a new pool, click Add in the top left of the pools table. In the pool form you can enter the pool's name, type, number of placement groups, and additional information, such as pool's applications, and compression mode and algorithm. The pool form itself pre-calculates the number of placement groups that best suited to this specific pool. The calculation is based on the amount of OSDs in the cluster and the selected pool type with its specific settings. As soon as a placement groups number is set manually, it will be replaced by a calculated number. Confirm with Create pool.

Figure 8.2: Adding a New Pool #

To delete a pool, click its table row and click Delete in the top left of the pools table.

To edit a pool's options, click its table row and select Edit in the top left of the pools table.

You can change the name of the pool, increase the number of placement groups, change the list of the pool's applications and compression settings. Confirm with Edit pool.

To view more detailed information about a device, click its row in the table:

Figure 9.2: RBD Details #

To view detailed configuration of a device, click its row in the table and then the Configuration tab in the lower table:

Figure 9.3: RBD Configuration #

To add a new device, click Add in the top left of the table heading and do the following on the Create RBD screen:

Figure 9.4: Adding a New RBD #

To delete a device, click its row in the table and select Delete in the top left of the table heading. Confirm the deletion with Delete RBD.

Tip: Moving RBDs to Trash

Deleting an RBD is an irreversible action. If you Move to Trash instead, you can restore the device later on by selecting it on the Trash tab of the main table and clicking Restore in the top left of the table heading.

To create a RADOS Block Device snapshot, click the device's table row and in the Snapshots tab below the main table, click Create in the top left of the table heading. Enter the snapshot's name and confirm with Create Snapshot.

After selecting a snapshot, you can perform additional actions on the device, such as rename, protect, clone, copy, or delete. Rollback restores the device's state from the current snapshot.

Figure 9.5: RBD Snapshots #

Tip: More Information on iSCSI Gateways

For more general information about iSCSI Gateways, refer to Book “Deployment Guide”, Chapter 10 “Installation of iSCSI Gateway” and Chapter 27, Ceph iSCSI Gateway.

To list all available gateways and mapped images, click Block / iSCSI from the main menu. An Overview tab opens, listing currently configured iSCSI Gateways and mapped RBD images.

The Gateways table lists each gateway's state, number of iSCSI targets, and number of sessions. The Images table lists each mapped image's name, related pool name backstore type, and other statistical details.

The Targets tab lists currently configured iSCSI targets.

Figure 9.6: List of iSCSI Targets #

To view more detailed information about a target, click its table row. A tree-structured schema opens, listing disks, portals, initiators, and groups. Click an item to expand it and view its detailed contents, optionally with a related configuration in the table on the right.

Figure 9.7: iSCSI Target Details #

9.6.1 Adding iSCSI Targets #

To add a new iSCSI target, click Add in the top left of the Targets table and enter the required information.

Figure 9.8: Adding a New Target #

1. Enter the target address of the new gateway.

2. Click Add portal and select one or multiple iSCSI portals from the list.

3. Click Add image and select one or multiple RBD images for the gateway.

4. If you need to use authentication to access the gateway, activate the Authentication check box and enter the credentials. You can find more advanced authentication options after activating Mutual authentication and Discovery authentication.

5. Confirm with Create Target.

Tip: For More Information

For more general information and a description of RBD QoS configuration options, refer to Section 23.6, “QoS Settings”.

The QoS options can be configured at different levels.

The global configuration is at the top of the list and will be used for all newly created RBD images and for those images that do not override these values on the pool or RBD image layer. An option value specified globally can be overridden on a per-pool or per-image basis. Options specified on a pool will be applied to all RBD images of that pool unless overridden by a configuration option set on an image. Options specified on an image will override options specified on a pool and will override options specified globally.

This way it is possible to define defaults globally, adapt them for all RBD images of a specific pool, and override the pool configuration for individual RBD images.

9.7.2 Configuring Options on a New Pool #

To create a new pool and configure RBD configuration options on it, click Pools / Create. Select replicated as pool type. You will then need to add the rbd application tag to the pool to be able to configure the RBD QoS options.

Note

It is not possible to configure RBD QoS configuration options on an erasure coded pool. To configure the RBD QoS options for erasure coded pools, you need to edit the replicated metadata pool of an RBD image. The configuration will then be applied to the erasure coded data pool of that image.

9.7.3 Configuring Options on an Existing Pool #

To configure RBD QoS options on an existing pool, click Pools, then click the pool's table row and select Edit at the top left of the table.

You should see the RBD Configuration section in the dialog, followed by a Quality of Service section.

Note

If you see neither the RBD Configuration nor the Quality of Service section, you are likely either editing an erasure coded pool, which cannot be used to set RBD configuration options, or the pool is not configured to be used by RBD images. In the latter case, assign the rbd application tag to the pool and the corresponding configuration sections will show up.

Tip: General Information

For general information and the command line approach to RADOS Block Device mirroring, refer to Section 23.4, “Mirroring”.

You can use the Ceph Dashboard to configure replication of RBD images between two or more clusters.

9.8.1 Primary Cluster and Secondary Cluster(s) #

Primary cluster is where the original pool with images is created. Secondary cluster(s) is where the pool/images are replicated from the primary cluster.

Note: Relative Naming

The primary and secondary terms can be relative in the context of replication because they relate more to individual pools than to clusters. For example, in two-way replication, one pool can be mirrored from the primary cluster to the secondary one, while another pool can be mirrored from the secondary cluster to the primary one.

To add a new NFS export, click Add in the top left of the exports table and enter the required information.

Note

The following example uses admin for the Object Gateway User. For more information on the permissions for other users, see Section 19.2.1.2, “Authorization and Capabilities”.

Figure 10.3: Adding a New NFS Export #

To delete an export, click its table row and select Delete in the top left of the exports table.

To edit an existing export, click its table row and click Edit in the top left of the exports table.

You can then adjust all the details of the NFS export.

Figure 10.4: Editing an NFS Export #

Click Filesystems from the main menu to view the overview of configured file systems. The main table shows each file system's name, date of creation, and whether it is enabled or not.

By clicking on a file system's table row, you reveal details about its rank and pools added to the file system.

Figure 11.1: CephFS Details #

At the bottom of the screen, you can see statistics counting the number of related MDS inodes and client requests, collected in real time.

To view a list of configured Object Gateways, click Object Gateway / Daemons. The list includes the ID of the gateway, host name of the cluster node where the gateway daemon is running, and the gateway's version number.

Click a gateway's table row to view detailed information about the gateway. The Performance Counters tab shows details about read/write operations and cache statistics.

Figure 12.1: Gateway's Details #

Click Object Gateway / Users to view a list of existing Object Gateway users.

Click a user's table row to view details about the user account, such as status information or the user and bucket quota details.

Figure 12.2: Gateway Users #

12.2.1 Adding a New Gateway User #

To add a new gateway user, click Add in the top left of the table heading. Fill in their credentials, details about the S3 key and user/bucket quota, then confirm with Add.

Figure 12.3: Adding a New Gateway User #

Object Gateway (OGW) buckets implement the functionality of OpenStack Swift containers. Object Gateway buckets serve as containers for storing data objects.

Click Object Gateway / Buckets to view a list of OGW buckets.

12.3.2 Viewing Bucket Details #

To view detailed information about an OGW bucket, click its table row.

Figure 12.4: Gateway Bucket Details #

Tip: Bucket Quota

Below the Details table, you can find details about the bucket quota settings.

All HTTP connections to the dashboard are secured with SSL/TLS by default. A secure connection requires an SSL certificate. You can either use a self-signed certificate, or generate a certificate and have a well known certificate authority (CA) sign it.

Tip: Disabling SSL

You may want to disable the SSL support for a specific reason. For example, if the dashboard is running behind a proxy that does not support SSL.

Use caution when disabling SSL as user names and passwords will be sent to the dashboard unencrypted.

To disable SSL, run:

cephadm@adm > ceph config set mgr mgr/dashboard/ssl false

Tip: Restart Ceph Manager Processes

You need to restart the Ceph Manager processes manually after changing the SSL certificate and key. You can do so by either running

cephadm@adm > ceph mgr failACTIVE-MANAGER-NAME

or by disabling and re-enabling the dashboard module, which also triggers the manager to respawn itself:

cephadm@adm > ceph mgr module disable dashboard
cephadm@adm > ceph mgr module enable dashboard

13.1.1 Self-signed Certificates #

Creating a self-signed certificate for secure communication is simple. This way you can get the dashboard running quickly.

Note: Web Browsers Complain

Most Web browsers will complain about a self-signed certificate and require explicit confirmation before establishing a secure connection to the dashboard.

To generate and install a self-signed certificate, use the following built-in command:

cephadm@adm > ceph dashboard create-self-signed-cert

13.1.3 Certificates Signed by CA #

To properly secure the connection to the dashboard and to eliminate Web browser complaints about a self-signed certificate, we recommend using a certificate that is signed by a CA.

You can generate a certificate key pair with a command similar to the following:

root # openssl req -new -nodes -x509 \
  -subj "/O=IT/CN=ceph-mgr-dashboard" -days 3650 \
  -keyout dashboard.key -out dashboard.crt -extensions v3_ca

The above command outputs dashboard.key and dashboard.crt files. After you get the dashboard.crt file signed by a CA, enable it for all Ceph Manager instances by running the following commands:

cephadm@adm > ceph dashboard set-ssl-certificate -i dashboard.crt
cephadm@adm > ceph dashboard set-ssl-certificate-key -i dashboard.key

Tip: Different Certificates for Each Manager Instance

If you require different certificates for each Ceph Manager instance, modify the commands and include the name of the instance as follows. Replace NAME with the name of the Ceph Manager instance (usually the related host name):

cephadm@adm > ceph dashboard set-ssl-certificate NAME -i dashboard.crt
cephadm@adm > ceph dashboard set-ssl-certificate-key NAME -i dashboard.key

13.1.4 Certificates Signed with a Custom CA #

The following procedure needs to be followed once to create the root CA.

Note

This is the key used to sign the certificate requests. Anyone holding this can sign certificates on your behalf.

1. Create the Root Key:

   cephadm@adm > openssl genrsa -des3 -out rootCA.key 4096

   

   Note

   If you want a non-password protected key, remove the -des3 option.

2. Create and self-sign the root certificate:

   cephadm@adm > openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.crt

The following procedure needs to be followed for each server that needs a trusted certificate from our CA.

1. Create the certificate key:

   cephadm@adm > openssl genrsa -out mydomain.com.key 2048

   The certificate signing request is where you specify the details for the certificate you want to generate. This request is processed by the owner of the Root Key to generate the certificate.

2. These are two ways to create the CSR:

   

   Important

   When creating the certificate signing request, it is important to specify the Common Name providing the IP address or domain name for the service, otherwise the certificate cannot be verified.

   • Interactive method. For example:

     cephadm@adm > openssl req -new -key mydomain.com.key -out mydomain.com.csr

     You will then be prmopted for information. For example, the Country Name, Organization Name, and Email Address.

   • One-liner method. This is where instead of being interactively prompted, you include the information up front. For example:

     cephadm@adm > openssl req -new -sha256 -key mydomain.com.key -subj "/C=US/ST=CA/O=MyOrg, Inc./CN=mydomain.com" -out mydomain.com.csr

     If you need to pass additional configuration in the one-liner method, you can use the -config parameter. For example:

     cephadm@adm > openssl req -new -sha256 \
           -key mydomain.com.key \
           -subj "/C=US/ST=CA/O=MyOrg, Inc./CN=mydomain.com" \
           -reqexts SAN \
           -config <(cat /etc/ssl/openssl.cnf \
               <(printf "\n[SAN]\nsubjectAltName=DNS:mydomain.com,DNS:www.mydomain.com")) \
           -out mydomain.com.csr

3. Verify the CSR content:

   cephadm@adm > openssl req -in mydomain.com.csr -noout -text

4. Generate the certificate using the mydomain CSR and key along with the CA Root Key:

   cephadm@adm > openssl x509 -req -in mydomain.com.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out mydomain.com.crt -days 500 -sha256

5. Verify the certificate's content:

   cephadm@adm > openssl x509 -in mydomain.com.crt -text -noout

The Ceph Dashboard Web application binds to a specific TCP/IP address and TCP port. By default, the currently active Ceph Manager that hosts the dashboard binds to TCP port 8443 (or 8080 when SSL is disabled).

The dashboard Web application binds to "::" by default, which corresponds to all available IPv4 and IPv6 addresses. You can change the IP address and port number of the Web application so that they apply to all Ceph Manager instances by using the following commands:

cephadm@adm > ceph config set mgr mgr/dashboard/server_addr IP_ADDRESS
cephadm@adm > ceph config set mgr mgr/dashboard/server_port PORT_NUMBER

Tip: Configure Ceph Manager Instances Separately

Since each ceph-mgr daemon hosts its own instance of the dashboard, you may need to configure them separately. Change the IP address and port number for a specific manager instance by using the following commands (replace NAME with the ID of the ceph-mgr instance):

cephadm@adm > ceph config set mgr mgr/dashboard/NAME/server_addr IP_ADDRESS
cephadm@adm > ceph config set mgr mgr/dashboard/NAME/server_port PORT_NUMBER

Tip: List Configured Endpoints

The ceph mgr services command displays all endpoints that are currently configured. Look for the 'dashboard' key to obtain the URL for accessing the dashboard.

If you do not want to use the default administrator account, create a different user account and associate it with at least one role. We provide a set of predefined system roles that you can use. For more details refer to Chapter 14, Managing Users and Roles on the Command Line.

To create a user with administrator privileges, use the following command:

cephadm@adm > ceph dashboard ac-user-create USER_NAME PASSWORD administrator

To use the Object Gateway management functionality of the dashboard, you need to provide the login credentials of a user with the 'system' flag enabled:

There are several points to consider:

Single Sign-On (SSO) is an access control method that enables users to log in with a single ID and password to multiple applications simultaneously.

The Ceph Dashboard supports external authentication of users via the SAML 2.0 protocol. Because authorization is still performed by the dashboard, you first need to create user accounts and associate them with the desired roles. However, the authentication process can be performed by an existing Identity Provider (IdP).

To configure Single Sign-On, use the following command:

cephadm@adm > ceph dashboard sso setup saml2 CEPH_DASHBOARD_BASE_URL \
 IDP_METADATA IDP_USERNAME_ATTRIBUTE \
 IDP_ENTITY_ID SP_X_509_CERT \
 SP_PRIVATE_KEY

Parameters:

Note: SAML Requests

The issuer value of SAML requests will follow this pattern:

CEPH_DASHBOARD_BASE_URL/auth/saml2/metadata

To display the current SAML 2.0 configuration, run:

cephadm@adm > ceph dashboard sso show saml2

To disable Single Sign-On, run:

cephadm@adm > ceph dashboard sso disable

To check if SSO is enabled, run:

cephadm@adm > ceph dashboard sso status

To enable SSO, run:

cephadm@adm > ceph dashboard sso enable saml2

The Ceph Dashboard supports managing multiple user accounts. Each user account consists of a user name, a password (stored in encrypted form using bcrypt), an optional name, and an optional e-mail address.

User accounts are stored in Ceph Monitor’s configuration database and are globally shared across all Ceph Manager instances.

Use the following commands to manage user accounts:

This section describes what security scopes you can assign to a user role, how to manage user roles and assign them to user accounts.

14.2.2 User Roles #

A role specifies a set of mappings between a security scope and a set of permissions. There are four types of permissions: 'read', 'create', 'update', and 'delete'.

The following example specifies a role where a user has 'read' and 'create' permissions for features related to pool management, and has full permissions for features related to RBD image management:

{
  'role': 'my_new_role',
  'description': 'My new role',
  'scopes_permissions': {
    'pool': ['read', 'create'],
    'rbd-image': ['read', 'create', 'update', 'delete']
  }
}

The dashboard already provides a set of predefined roles that we call system roles. You can instantly use them after a fresh Ceph Dashboard installation:

administrator

Provides full permissions for all security scopes.

read-only

Provides read permission for all security scopes except the dashboard settings.

block-manager

Provides full permissions for 'rbd-image', 'rbd-mirroring', and 'iscsi' scopes.

rgw-manager

Provides full permissions for the 'rgw' scope.

cluster-manager

Provides full permissions for the 'hosts', 'osd', 'monitor', 'manager', and 'config-opt' scopes.

pool-manager

Provides full permissions for the 'pool' scope.

cephfs-manager

Provides full permissions for the 'cephfs' scope.

14.2.2.1 Managing Custom Roles #

You can create new user roles by using the following commands:

Create a new role:

cephadm@adm > ceph dashboard ac-role-create ROLENAME [DESCRIPTION]

Delete a role:

cephadm@adm > ceph dashboard ac-role-delete ROLENAME

Add scope permissions to a role:

cephadm@adm > ceph dashboard ac-role-add-scope-perms ROLENAME SCOPENAME PERMISSION [PERMISSION...]

Delete scope permissions from a role:

cephadm@adm > ceph dashboard ac-role-del-perms ROLENAME SCOPENAME

14.2.2.2 Assigning Roles to User Accounts #

Use the following commands to assign roles to users:

Set user roles:

cephadm@adm > ceph dashboard ac-user-set-roles USERNAME ROLENAME [ROLENAME ...]

Add additional roles to a user:

cephadm@adm > ceph dashboard ac-user-add-roles USERNAME ROLENAME [ROLENAME ...]

Delete roles from a user:

cephadm@adm > ceph dashboard ac-user-del-roles USERNAME ROLENAME [ROLENAME ...]

Tip: Purging Custom Roles

If you create custom user roles and intend to remove the Ceph cluster with the ceph.purge runner later on, you need to purge the custom roles first. Find more details in Section 2.17, “Removing an Entire Ceph Cluster”.

14.2.2.3 Example: Creating a User and a Custom Role #

This section illustrates a procedure for creating a user account capable of managing RBD images, viewing and creating Ceph pools, and having read-only access to any other scopes.

1. Create a new user named 'tux':

    cephadm@adm > ceph dashboard ac-user-create tux PASSWORD

2. Create a role and specify scope permissions:

   cephadm@adm > ceph dashboard ac-role-create rbd/pool-manager
   cephadm@adm > ceph dashboard ac-role-add-scope-perms rbd/pool-manager \
    rbd-image read create update delete
   cephadm@adm > ceph dashboard ac-role-add-scope-perms rbd/pool-manager pool read create

3. Associate the roles with the 'tux' user:

   cephadm@adm > ceph dashboard ac-user-set-roles tux rbd/pool-manager read-only

If you are accessing the dashboard via a reverse proxy configuration, you may need to service it under a URL prefix. To get the dashboard to use hyperlinks that include your prefix, you can set the url_prefix setting:

cephadm@adm > ceph config set mgr mgr/dashboard/url_prefix URL_PREFIX

Then you can access the dashboard at http://HOST_NAME:PORT_NUMBER/URL_PREFIX/.

The Ceph Dashboard's REST API can log PUT, POST, and DELETE requests to the Ceph audit log. Logging is disabled by default, but you can enable it with the following command:

cephadm@adm > ceph dashboard set-audit-api-enabled true

If enabled, the following parameters are logged per each request:

An example log entry looks like this:

2019-02-06 10:33:01.302514 mgr.x [INF] [DASHBOARD] \
 from='https://[::ffff:127.0.0.1]:37022' path='/api/rgw/user/exu' method='PUT' \
 user='admin' params='{"max_buckets": "1000", "display_name": "Example User", "uid": "exu", "suspended": "0", "email": "user@example.com"}'

Tip: Disable Logging of Request Payload

The logging of the request payload (the list of arguments and their values) is enabled by default. You can disable it as follows:

cephadm@adm > ceph dashboard set-audit-api-log-payload false

Use the systemctl command to operate all Ceph related services. The operation takes place on the node you are currently logged in to. You need to have root privileges to be able to operate on Ceph services.

cephadm@adm > ls /usr/lib/systemd/system/ceph*.target
ceph.target
ceph-osd.target
ceph-mon.target
ceph-mgr.target
ceph-mds.target
ceph-radosgw.target
ceph-rbd-mirror.target

root # systemctl start ceph.target
root # systemctl stop ceph.target
root # systemctl restart ceph.target

root # systemctl start ceph-osd.target
root # systemctl stop ceph-osd.target
root # systemctl restart ceph-osd.target

ceph-osd@.service
ceph-mon@.service
ceph-mds@.service
ceph-mgr@.service
ceph-radosgw@.service
ceph-rbd-mirror@.service

root # systemctl start ceph-osd@1.service
root # systemctl stop ceph-osd@1.service
root # systemctl restart ceph-osd@1.service

root # systemctl list-units --all --type=service ceph*

root # systemctl list-units --all --state=inactive --type=service ceph*

root@master # salt TARGET cmd.shell \
 "systemctl list-units --all --type=service ceph* | sed -e '/^$/,$ d'"

root@master # salt -I 'roles:storage' cmd.shell \
 'systemctl list-units --all --type=service ceph*'

root # systemctl status ceph-osd@1.service
root # systemctl status ceph-mon@HOSTNAME.service

After applying updates to the cluster nodes, the affected Ceph related services need to be restarted. Normally, restarts are performed automatically by DeepSea. This section describes how to restart the services manually.

Tip: Watching the Restart

The process of restarting the cluster may take some time. You can watch the events by using the Salt event bus by running:

root@master # salt-run state.event pretty=True

Another command to monitor active jobs is

root@master # salt-run jobs.active

16.2.1 Restarting All Services #

Warning: Interruption of Services

If Ceph related services—specifically iSCSI or NFS Ganesha—are configured as single points of access with no High Availability setup, restarting them will result in their temporary outage as viewed from the client side.

Tip: Samba Not Managed by DeepSea

Because DeepSea and the Ceph Dashboard do not currently support Samba deployments, you need to manage Samba related services manually. For more details, see Chapter 29, Exporting Ceph Data via Samba.

To restart all services on the cluster, run the following command:

root@master # salt-run state.orch ceph.restart

• For DeepSea prior to version 0.8.4, the Metadata Server, iSCSI Gateway, Object Gateway, and NFS Ganesha services restart in parallel.

• For DeepSea 0.8.4 and newer, all roles you have configured restart in the following order: Ceph Monitor, Ceph Manager, Ceph OSD, Metadata Server, Object Gateway, iSCSI Gateway, NFS Ganesha. To keep the downtime low and to find potential issues as early as possible, nodes are restarted sequentially. For example, only one monitoring node is restarted at a time.

The command waits for the cluster to recover if the cluster is in a degraded, unhealthy state.

root@master # salt-run state.orch ceph.restart.service_name

root@master # salt-run state.orch ceph.restart.rgw

root@master # salt-run state.orch ceph.restart.mon

root@master # salt-run state.orch ceph.restart.mgr

root@master # salt-run state.orch ceph.restart.osd

root@master # salt-run state.orch ceph.restart.mds

root@master # salt-run state.orch ceph.restart.rgw

root@master # salt-run state.orch ceph.restart.igw

root@master # salt-run state.orch ceph.restart.ganesha

There are occasions when you need to stop all Ceph related services in the cluster in the recommended order, and then be able to simply start them again. For example, in case of a planned power outage.

To check a cluster's status, execute the following:

cephadm@adm > ceph status

or

cephadm@adm > ceph -s

In interactive mode, type status and press Enter.

ceph> status

Ceph will print the cluster status. For example, a tiny Ceph cluster consisting of one monitor and two OSDs may print the following:

cluster b370a29d-9287-4ca3-ab57-3d824f65e339
 health HEALTH_OK
 monmap e1: 1 mons at {ceph1=10.0.0.8:6789/0}, election epoch 2, quorum 0 ceph1
 osdmap e63: 2 osds: 2 up, 2 in
  pgmap v41332: 952 pgs, 20 pools, 17130 MB data, 2199 objects
        115 GB used, 167 GB / 297 GB avail
               1 active+clean+scrubbing+deep
             951 active+clean

After you start your cluster and before you start reading and/or writing data, check your cluster's health:

cephadm@adm > ceph health
HEALTH_WARN 10 pgs degraded; 100 pgs stuck unclean; 1 mons down, quorum 0,2 \
node-1,node-2,node-3

Tip

If you specified non-default locations for your configuration or keyring, you may specify their locations:

cephadm@adm > ceph -c /path/to/conf -k /path/to/keyring health

The Ceph cluster returns one of the following health codes:

Tip

If you specified non-default locations for your configuration or keyring, you may specify their locations:

root # ceph -c /path/to/conf -k /path/to/keyring health

You can find the immediate state of the cluster using ceph -s. For example, a tiny Ceph cluster consisting of one monitor, and two OSDs may print the following when a workload is running:

cephadm@adm > ceph -s
cluster:
  id:     ea4cf6ce-80c6-3583-bb5e-95fa303c893f
  health: HEALTH_WARN
          too many PGs per OSD (408 > max 300)

services:
  mon: 3 daemons, quorum ses5min1,ses5min3,ses5min2
  mgr: ses5min1(active), standbys: ses5min3, ses5min2
  mds: cephfs-1/1/1 up  {0=ses5min3=up:active}
  osd: 4 osds: 4 up, 4 in
  rgw: 1 daemon active

data:
  pools:   8 pools, 544 pgs
  objects: 253 objects, 3821 bytes
  usage:   6252 MB used, 13823 MB / 20075 MB avail
  pgs:     544 active+clean

The output provides the following information:

Tip: How Ceph Calculates Data Usage

The used value reflects the actual amount of raw storage used. The xxx GB / xxx GB value means the amount available (the lesser number) of the overall storage capacity of the cluster. The notional number reflects the size of the stored data before it is replicated, cloned or snapshot. Therefore, the amount of data actually stored typically exceeds the notional amount stored, because Ceph creates replicas of the data and may also use storage capacity for cloning and snapshotting.

Other commands that display immediate status information are:

To get the information updated in real time, put any of these commands (including ceph -s) as an argument of the watch command:

root # watch -n 10 'ceph -s'

Press Ctrl–C when you are tired of watching.

To check a cluster’s data usage and distribution among pools, use the ceph df command. To get more details, use ceph df detail.

cephadm@adm > ceph df
RAW STORAGE:
    CLASS     SIZE       AVAIL      USED        RAW USED     %RAW USED
    hdd       40 GiB     32 GiB     137 MiB      8.1 GiB         20.33
    TOTAL     40 GiB     32 GiB     137 MiB      8.1 GiB         20.33
POOLS:
    POOL             ID     STORED     OBJECTS    USED       %USED    MAX AVAIL
    iscsi-images      1     3.9 KiB          8    769 KiB        0       10 GiB
    cephfs_data       2     1.6 KiB          5    960 KiB        0       10 GiB
    cephfs_metadata   3      54 KiB         22    1.5 MiB        0       10 GiB
[...]

The RAW STORAGE section of the output provides an overview of the amount of storage your cluster uses for your data.

The POOLS section of the output provides a list of pools and the notional usage of each pool. The output from this section does not reflect replicas, clones or snapshots. For example, if you store an object with 1MB of data, the notional usage will be 1MB, but the actual usage may be 2MB or more depending on the number of replicas, clones and snapshots.

Note

The numbers in the POOLS section are notional. They are not inclusive of the number of replicas, snapshots or clones. As a result, the sum of the USEDand %USED amounts will not add up to the RAW USED and %RAW USED amounts in the RAW STORAGE section of the output.

You can check OSDs to ensure they are up and on by executing:

cephadm@adm > ceph osd stat

or

cephadm@adm > ceph osd dump

You can also view OSDs according to their position in the CRUSH map.

cephadm@adm > ceph osd tree

Ceph will print a CRUSH tree with a host, its OSDs, whether they are up and their weight.

# id    weight  type name       up/down reweight
-1      3       pool default
-3      3               rack mainrack
-2      3                       host osd-host
0       1                               osd.0   up      1
1       1                               osd.1   up      1
2       1                               osd.2   up      1

Ceph prevents you from writing to a full OSD so that you do not lose data. In an operational cluster, you should receive a warning when your cluster is getting near its full ratio. The mon osd full ratio defaults to 0.95, or 95% of capacity before it stops clients from writing data. The mon osd nearfull ratio defaults to 0.85, or 85% of capacity, when it generates a health warning.

Full OSD nodes will be reported by ceph health:

cephadm@adm > ceph health
  HEALTH_WARN 1 nearfull osds
  osd.2 is near full at 85%

or

cephadm@adm > ceph health
  HEALTH_ERR 1 nearfull osds, 1 full osds
  osd.2 is near full at 85%
  osd.3 is full at 97%

The best way to deal with a full cluster is to add new OSD hosts/disks allowing the cluster to redistribute data to the newly available storage.

Tip: Preventing Full OSDs

After an OSD becomes full—it uses 100% of its disk space—it will normally crash quickly without warning. Following are a few tips to remember when administering OSD nodes.

• Each OSD's disk space (usually mounted under /var/lib/ceph/osd/osd-{1,2..}) needs to be placed on a dedicated underlying disk or partition.

• Check the Ceph configuration files and make sure that Ceph does not store its log file to the disks/partitions dedicated for use by OSDs.

• Make sure that no other process writes to the disks/partitions dedicated for use by OSDs.

After you start the cluster and before first reading and/or writing data, check the Ceph Monitors' quorum status. When the cluster is already serving requests, check the Ceph Monitors' status periodically to ensure that they are running.

To display the monitor map, execute the following:

cephadm@adm > ceph mon stat

or

cephadm@adm > ceph mon dump

To check the quorum status for the monitor cluster, execute the following:

cephadm@adm > ceph quorum_status

Ceph will return the quorum status. For example, a Ceph cluster consisting of three monitors may return the following:

{ "election_epoch": 10,
  "quorum": [
        0,
        1,
        2],
  "monmap": { "epoch": 1,
      "fsid": "444b489c-4f16-4b75-83f0-cb8097468898",
      "modified": "2011-12-12 13:28:27.505520",
      "created": "2011-12-12 13:28:27.505520",
      "mons": [
            { "rank": 0,
              "name": "a",
              "addr": "192.168.1.10:6789\/0"},
            { "rank": 1,
              "name": "b",
              "addr": "192.168.1.11:6789\/0"},
            { "rank": 2,
              "name": "c",
              "addr": "192.168.1.12:6789\/0"}
           ]
    }
}

Placement groups map objects to OSDs. When you monitor your placement groups, you will want them to be active and clean. For a detailed discussion, refer to Section 17.11, “Monitoring OSDs and Placement Groups”.

The Ceph admin socket allows you to query a daemon via a socket interface. By default, Ceph sockets reside under /var/run/ceph. To access a daemon via the admin socket, log in to the host running the daemon and use the following command:

cephadm@adm > ceph --admin-daemon /var/run/ceph/socket-name

To view the available admin socket commands, execute the following command:

cephadm@adm > ceph --admin-daemon /var/run/ceph/socket-name help

The admin socket command enables you to show and set your configuration at runtime. Refer to Section 25.1, “Runtime Configuration” for details.

Additionally, you can set configuration values at runtime directly (the admin socket bypasses the monitor, unlike ceph tell daemon-type.id injectargs, which relies on the monitor but does not require you to log in directly to the host in question).

When a Ceph storage cluster gets close to its maximum capacity, Ceph prevents you from writing to or reading from Ceph OSDs as a safety measure to prevent data loss. Therefore, letting a production cluster approach its full ratio is not a good practice, because it sacrifices high availability. The default full ratio is set to .95, meaning 95% of capacity. This a very aggressive setting for a test cluster with a small number of OSDs.

Tip: Increase Storage Capacity

When monitoring your cluster, be alert to warnings related to the nearfull ratio. It means that a failure of some OSDs could result in a temporary service disruption if one or more OSDs fails. Consider adding more OSDs to increase storage capacity.

A common scenario for test clusters involves a system administrator removing a Ceph OSD from the Ceph storage cluster to watch the cluster rebalance. Then removing another Ceph OSD, and so on until the cluster eventually reaches the full ratio and locks up. We recommend a bit of capacity planning even with a test cluster. Planning enables you to estimate how much spare capacity you will need in order to maintain high availability. Ideally, you want to plan for a series of Ceph OSD failures where the cluster can recover to an active + clean state without replacing those Ceph OSDs immediately. You can run a cluster in an active + degraded state, but this is not ideal for normal operating conditions.

The following diagram depicts a simplistic Ceph storage cluster containing 33 Ceph nodes with one Ceph OSD per host, each of them reading from and writing to a 3 TB drive. This exemplary cluster has a maximum actual capacity of 99 TB. The mon osd full ratio option is set to 0.95. If the cluster falls to 5 TB of the remaining capacity, it will not allow the clients to read and write data. Therefore the storage cluster’s operating capacity is 95 TB, not 99 TB.

Figure 17.1: Ceph Cluster #

It is normal in such a cluster for one or two OSDs to fail. A less frequent but reasonable scenario involves a rack’s router or power supply failing, which brings down multiple OSDs simultaneously (for example, OSDs 7-12). In such a scenario, you should still strive for a cluster that can remain operational and achieve an active + clean state—even if that means adding a few hosts with additional OSDs in short order. If your capacity usage is too high, you may not lose data. But you could still sacrifice data availability while resolving an outage within a failure domain if capacity usage of the cluster exceeds the full ratio. For this reason, we recommend at least some rough capacity planning.

Identify two numbers for your cluster:

If you divide the total capacity of your cluster by the number of OSDs in your cluster, you will find the mean average capacity of an OSD within your cluster. Consider multiplying that number by the number of OSDs you expect will fail simultaneously during normal operations (a relatively small number). Finally, multiply the capacity of the cluster by the full ratio to arrive at a maximum operating capacity. Then, subtract the number of the amount of data from the OSDs you expect to fail to arrive at a reasonable full ratio. Repeat the foregoing process with a higher number of OSD failures (a rack of OSDs) to arrive at a reasonable number for a near full ratio.

The following settings only apply on cluster creation and are then stored in the OSD map:

[global]
 mon osd full ratio = .80
 mon osd backfillfull ratio = .75
 mon osd nearfull ratio = .70

Tip

These settings only apply during cluster creation. Afterward they need to be changed in the OSD Map using the ceph osd set-nearfull-ratio and ceph osd set-full-ratio commands.

Tip: Check OSD Weight

If some OSDs are nearfull, but others have plenty of capacity, you may have a problem with the CRUSH weight for the nearfull OSDs.

High availability and high reliability require a fault-tolerant approach to managing hardware and software issues. Ceph has no single point-of-failure, and can service requests for data in a 'degraded' mode. Ceph’s data placement introduces a layer of indirection to ensure that data does not bind directly to particular OSD addresses. This means that tracking down system faults requires finding the placement group and the underlying OSDs at root of the problem.

Tip: Access in Case of Failure

A fault in one part of the cluster may prevent you from accessing a particular object. That does not mean that you cannot access other objects. When you run into a fault, follow the steps for monitoring your OSDs and placement groups. Then begin troubleshooting.

Ceph is generally self-repairing. However, when problems persist, monitoring OSDs and placement groups will help you identify the problem.

17.11.1 Monitoring OSDs #

An OSD’s status is either in the cluster ('in') or out of the cluster ('out'). At the same time, it is either up and running ('up') or it is down and not running ('down'). If an OSD is 'up', it may be either in the cluster (you can read and write data) or out of the cluster. If it was in the cluster and recently moved out of the cluster, Ceph will migrate placement groups to other OSDs. If an OSD is out of the cluster, CRUSH will not assign placement groups to it. If an OSD is 'down', it should also be 'out'.

Note: Unhealthy State

If an OSD is 'down' and 'in', there is a problem and the cluster will not be in a healthy state.

If you execute a command such as ceph health, ceph -s or ceph -w, you may notice that the cluster does not always echo back HEALTH OK. With regard to OSDs, you should expect that the cluster will not echo HEALTH OK under the following circumstances:

• You have not started the cluster yet (it will not respond).

• You have just started or restarted the cluster and it is not ready yet, because the placement groups are being created and the OSDs are in the process of peering.

• You have just added or removed an OSD.

• You have just modified your cluster map.

An important aspect of monitoring OSDs is to ensure that when the cluster is up and running, all the OSDs in the cluster are up and running, too. To see if all the OSDs are running, execute:

root # ceph osd stat
x osds: y up, z in; epoch: eNNNN

The result should tell you the total number of OSDs (x), how many are 'up' (y), how many are 'in' (z), and the map epoch (eNNNN). If the number of OSDs that are 'in' the cluster is more than the number of OSDs that are 'up', execute the following command to identify the ceph-osd daemons that are not running:

root # ceph osd tree
#ID CLASS WEIGHT  TYPE NAME             STATUS REWEIGHT PRI-AFF
-1       2.00000 pool openstack
-3       2.00000 rack dell-2950-rack-A
-2       2.00000 host dell-2950-A1
0   ssd 1.00000      osd.0                up  1.00000 1.00000
1   ssd 1.00000      osd.1              down  1.00000 1.00000

If an OSD with, for example, ID 1 is down, start it:

cephadm@osd > sudo systemctl start ceph-osd@1.service

See Section 17.12, “OSD Is Not Running” for problems associated with OSDs that have stopped or that will not restart.

17.11.2 Placement Group Sets #

When CRUSH assigns placement groups to OSDs, it looks at the number of replicas for the pool and assigns the placement group to OSDs such that each replica of the placement group gets assigned to a different OSD. For example, if the pool requires three replicas of a placement group, CRUSH may assign them to osd.1, osd.2 and osd.3 respectively. CRUSH actually seeks a pseudo-random placement that will take into account failure domains you set in your CRUSH Map, so you will rarely see placement groups assigned to nearest neighbor OSDs in a large cluster. We refer to the set of OSDs that should contain the replicas of a particular placement group as the acting set. In some cases, an OSD in the acting set is down or otherwise not able to service requests for objects in the placement group. When these situations arise, it may match one of the following scenarios:

• You added or removed an OSD. Then, CRUSH reassigned the placement group to other OSDs and therefore changed the composition of the acting set, causing the migration of data with a 'backfill' process.

• An OSD was 'down', was restarted, and is now recovering.

• An OSD in the acting set is 'down' or unable to service requests, and another OSD has temporarily assumed its duties.

  Ceph processes a client request using the up set, which is the set of OSDs that will actually handle the requests. In most cases, the up set and the acting set are virtually identical. When they are not, it may indicate that Ceph is migrating data, an OSD is recovering, or that there is a problem (for example, Ceph usually echoes a HEALTH WARN state with a 'stuck stale' message in such scenarios).

To retrieve a list of placement groups, run:

cephadm@adm > ;ceph pg dump

To view which OSDs are within the acting set or the up set for a given placement group, run:

cephadm@adm > ceph pg mapPG_NUM
osdmap eNNN pg RAW_PG_NUM (PG_NUM) -> up [0,1,2] acting [0,1,2]

The result should tell you the osdmap epoch (eNNN), the placement group number (PG_NUM), the OSDs in the up set ('up'), and the OSDs in the acting set ('acting'):

Tip: Cluster Problem Indicator

If the up set and acting set do not match, this may be an indicator either of the cluster rebalancing itself, or of a potential problem with the cluster.

17.11.3 Peering #

Before you can write data to a placement group, it must be in an 'active' state, and it should be in a 'clean' state. For Ceph to determine the current state of a placement group, the primary OSD of the placement group (the first OSD in the acting set), peers with the secondary and tertiary OSDs to establish agreement on the current state of the placement group (assuming a pool with three replicas of the PG).

Figure 17.2: Peering Schema #

17.11.4 Monitoring Placement Group States #

If you execute a command such as ceph health, ceph -s or ceph -w, you may notice that the cluster does not always echo back the HEALTH OK message. After you check to see if the OSDs are running, you should also check placement group states.

Expect that the cluster will not echo HEALTH OK in a number of placement group peering-related circumstances:

• You have just created a pool and placement groups have not peered yet.

• The placement groups are recovering.

• You have just added an OSD to or removed an OSD from the cluster.

• You have just modified your CRUSH Map and your placement groups are migrating.

• There is inconsistent data in different replicas of a placement group.

• Ceph is scrubbing a placement group’s replicas.

• Ceph does not have enough storage capacity to complete backfilling operations.

If one of the above mentioned circumstances causes Ceph to echo HEALTH WARN, do not panic. In many cases, the cluster will recover on its own. In some cases, you may need to take action. An important aspect of monitoring placement groups is to ensure that when the cluster is up and running, all placement groups are 'active' and preferably in the 'clean state'. To see the status of all placement groups, run:

cephadm@adm > ceph pg stat
x pgs: y active+clean; z bytes data, aa MB used, bb GB / cc GB avail

The result should tell you the total number of placement groups (x), how many placement groups are in a particular state such as 'active+clean' (y) and the amount of data stored (z).

In addition to the placement group states, Ceph will also echo back the amount of storage capacity used (aa), the amount of storage capacity remaining (bb), and the total storage capacity for the placement group. These numbers can be important in a few cases:

• You are reaching your near full ratio or full ratio.

• Your data is not getting distributed across the cluster because of an error in your CRUSH configuration.

Tip: Placement Group IDs

Placement group IDs consist of the pool number (not pool name) followed by a period (.) and the placement group ID—a hexadecimal number. You can view pool numbers and their names from the output of ceph osd lspools. For example, the default pool rbd corresponds to pool number 0. A fully qualified placement group ID has the following form:

POOL_NUM.PG_ID

And it typically looks like this:

0.1f

To retrieve a list of placement groups, run the following:

cephadm@adm > ceph pg dump

You can also format the output in JSON format and save it to a file:

cephadm@adm > ceph pg dump -o FILE_NAME --format=json

To query a particular placement group, run the following:

cephadm@adm > ceph pg POOL_NUM.PG_ID query

The following list describes the common placement group states in detail.

CREATING

When you create a pool, it will create the number of placement groups you specified. Ceph will echo 'creating' when it is creating one or more placement groups. When they are created, the OSDs that are part of the placement group’s acting set will peer. When peering is complete, the placement group status should be 'active+clean', which means that a Ceph client can begin writing to the placement group.

Figure 17.3: Placement Groups Status #

PEERING

When Ceph is peering a placement group, it is bringing the OSDs that store the replicas of the placement group into agreement about the state of the objects and metadata in the placement group. When Ceph completes peering, this means that the OSDs that store the placement group agree about the current state of the placement group. However, completion of the peering process does not mean that each replica has the latest contents.

Note: Authoritative History

Ceph will not acknowledge a write operation to a client until all OSDs of the acting set persist the write operation. This practice ensures that at least one member of the acting set will have a record of every acknowledged write operation since the last successful peering operation.

With an accurate record of each acknowledged write operation, Ceph can construct and enlarge a new authoritative history of the placement group—a complete and fully ordered set of operations that, if performed, would bring an OSD’s copy of a placement group up to date.

ACTIVE

When Ceph completes the peering process, a placement group may become 'active'. The 'active' state means that the data in the placement group is generally available in the primary placement group and the replicas for read and write operations.

CLEAN

When a placement group is in the 'clean' state, the primary OSD and the replica OSDs have successfully peered and there are no stray replicas for the placement group. Ceph replicated all objects in the placement group the correct number of times.

DEGRADED

When a client writes an object to the primary OSD, the primary OSD is responsible for writing the replicas to the replica OSDs. After the primary OSD writes the object to storage, the placement group will remain in a 'degraded' state until the primary OSD has received an acknowledgement from the replica OSDs that Ceph created the replica objects successfully.

The reason a placement group can be 'active+degraded' is that an OSD may be 'active' even though it does not hold all of the objects yet. If an OSD goes down, Ceph marks each placement group assigned to the OSD as 'degraded'. The OSDs must peer again when the OSD comes back up. However, a client can still write a new object to a degraded placement group if it is 'active'.

If an OSD is 'down' and the 'degraded' condition persists, Ceph may mark the down OSD as 'out' of the cluster and remap the data from the 'down' OSD to another OSD. The time between being marked 'down' and being marked 'out' is controlled by the mon osd down out interval option, which is set to 600 seconds by default.

A placement group can also be 'degraded' because Ceph cannot find one or more objects that should be in the placement group. While you cannot read or write to unfound objects, you can still access all of the other objects in the 'degraded' placement group.

RECOVERING

Ceph was designed for fault-tolerance at a scale where hardware and software problems are ongoing. When an OSD goes 'down', its contents may fall behind the current state of other replicas in the placement groups. When the OSD is back 'up', the contents of the placement groups must be updated to reflect the current state. During that time period, the OSD may reflect a 'recovering' state.

Recovery is not always trivial, because a hardware failure may cause a cascading failure of multiple OSDs. For example, a network switch for a rack or cabinet may fail, which can cause the OSDs of a number of host machines to fall behind the current state of the cluster. Each of the OSDs must recover when the fault is resolved.

Ceph provides a number of settings to balance the resource contention between new service requests and the need to recover data objects and restore the placement groups to the current state. The osd recovery delay start setting allows an OSD to restart, re-peer and even process some replay requests before starting the recovery process. The osd recovery thread timeout sets a thread timeout, because multiple OSDs may fail, restart and re-peer at staggered rates. The osd recovery max active setting limits the number of recovery requests an OSD will process simultaneously to prevent the OSD from failing to serve. The osd recovery max chunk setting limits the size of the recovered data chunks to prevent network congestion.

BACK FILLING

When a new OSD joins the cluster, CRUSH will reassign placement groups from OSDs in the cluster to the newly added OSD. Forcing the new OSD to accept the reassigned placement groups immediately can put excessive load on the new OSD. Backfilling the OSD with the placement groups allows this process to begin in the background. When backfilling is complete, the new OSD will begin serving requests when it is ready.

During the backfill operations, you may see one of several states: 'backfill_wait' indicates that a backfill operation is pending, but is not yet in progress; 'backfill' indicates that a backfill operation is in progress; 'backfill_too_full' indicates that a backfill operation was requested, but could not be completed because of insufficient storage capacity. When a placement group cannot be backfilled, it may be considered 'incomplete'.

Ceph provides a number of settings to manage the load associated with reassigning placement groups to an OSD (especially a new OSD). By default, osd max backfills sets the maximum number of concurrent backfills to or from an OSD to 10. The backfill full ratio enables an OSD to refuse a backfill request if the OSD is approaching its full ratio (90%, by default) and change with ceph osd set-backfillfull-ratio command. If an OSD refuses a backfill request, the osd backfill retry interval enables an OSD to retry the request (after 10 seconds, by default). OSDs can also set osd backfill scan min and osd backfill scan max to manage scan intervals (64 and 512, by default).

REMAPPED

When the acting set that services a placement group changes, the data migrates from the old acting set to the new acting set. It may take some time for a new primary OSD to service requests. So it may ask the old primary to continue to service requests until the placement group migration is complete. When data migration completes, the mapping uses the primary OSD of the new acting set.

STALE

While Ceph uses heartbeats to ensure that hosts and daemons are running, the ceph-osd daemons may also get into a 'stuck' state where they are not reporting statistics in a timely manner (for example, a temporary network fault). By default, OSD daemons report their placement group, boot and failure statistics every half second (0.5), which is more frequent than the heartbeat thresholds. If the primary OSD of a placement group’s acting set fails to report to the monitor or if other OSDs have reported the primary OSD as 'down', the monitors will mark the placement group as 'stale'.

When you start your cluster, it is common to see the 'stale' state until the peering process completes. After your cluster has been running for a while, seeing placement groups in the 'stale' state indicates that the primary OSD for those placement groups is down or not reporting placement group statistics to the monitor.

cephadm@adm > ceph pg dump_stuck [unclean|inactive|stale|undersized|degraded]

cephadm@adm > ceph osd map POOL_NAME OBJECT_NAME [NAMESPACE]

cephadm@adm > rados put test-object-1 testfile.txt --pool=data

cephadm@adm > rados -p data ls

cephadm@adm > ceph osd map data test-object-1
osdmap e537 pool 'data' (0) object 'test-object-1' -> pg 0.d1743484 \
(0.4) -> up ([1,0], p0) acting ([1,0], p0)

cephadm@adm > rados rm test-object-1 --pool=data

Under normal circumstances, simply restarting the ceph-osd daemon will allow it to rejoin the cluster and recover.

cephadm@adm > ceph health
HEALTH_WARN 1/3 in osds are down

cephadm@adm > ceph health detail
HEALTH_WARN 1/3 in osds are down
osd.0 is down since epoch 23, last address 192.168.106.220:6800/11080

cephadm@adm > ceph osd set-nearfull-ratio 0.0 to 1.0

cephadm@adm > ceph osd set-nearfull-ratio 0.0 to 1.0
cephadm@adm > ceph osd set-full-ratio 0.0 to 1.0
cephadm@adm > ceph osd set-backfillfull-ratio 0.0 to 1.0

cephadm@adm > ceph health
HEALTH_WARN 1 nearfull osd(s)

cephadm@adm > ceph health detail
HEALTH_ERR 1 full osd(s); 1 backfillfull osd(s); 1 nearfull osd(s)
osd.3 is full at 97%
osd.4 is backfill full at 91%
osd.2 is near full at 87%