The performance of storage is measured on two different, but related axes: latency and throughput. In some cases, one is more important than the other. An example of throughput being more important is that of backup use cases, where throughput is the most critical measurement and maximum performance is achieved with larger transfer sizes. Conversely, for a high-performance database, latency is the most important measurement.

Latency is the time from when the request was made to when it was completed. This is usually measured in terms of milliseconds. This performance can be directly tied to CPU clock speed, the system bus, and device performance. Throughput is the measure of an amount of data that can be written or retrieved within a particular time period. The measure is usually in MB/s and GB/s, or perhaps MiB/s and GiB/s.

A third measurement that is often referred to is IOPS. This stands for Input/output Operations Per Second. This measure is somewhat ambiguous as the result is reliant on the size of the I/O operation, what type (read/write), and details about the I/O pattern: fresh write, overwrite, random write, mix of reads and writes. While ambiguous, it is still a valid tool to use when measuring the changes in the performance of your storage environment. For example, it is possible to make adjustments that may not affect the latency of a single 4K sequential write operation, but would allow many more of the operations to happen in parallel, resulting in a change in throughput and IOPS.

Understanding the requirements of the workload in relation to whether it needs scale-up or scale-out performance is often a key to proper tuning as well, particularly when it comes to tuning the hardware architecture and to creating test scenarios that provide valid information regarding tuning for the application(s) in question.

Here are some general principals to follow when planning a Ceph cluster network:

There are a lot of options when it comes to processor selection in the market today. There are multiple varieties available from the major vendors in the x86 space and a wide array of options in the 64-bit Arm space as well. Without regard to the core architecture, there is one rule that is always true, and that is that higher clockspeed allows more work to be done in the same amount of time. This consideration is most important when working with higher speed storage and network devices.

CPU selection is also an important consideration for specific services. Some services, such as metadata servers, NFS, Samba, and ISCSI gateways benefit from a smaller number of much faster cores, while the OSD nodes need a more core dense solution.

A second consideration is whether to use a single socket or multiple sockets. The answer to this will depend on the device density, type of network hardware being utilized, etc. In many nodes, a single socket will provide better performance as the processor interlink is a bottleneck, though this would most likely be noticed in an all NVMe based node type. The general recommendation is to use a single socket whenever possible.

When considering which processor to choose, there are several considerations aside from clock-speed that should be taken into consideration. They include:

Storage device selection can dramatically affect the performance and reliability of a Ceph cluster. When building for performance, it is important to understand the nature of the device in regard to reads/writes and the workloads that will be applied. This is particularly true with flash media.

It is also important to understand the impact of the storage bus and hardware pieces along the way. Clearly, 6 Gb/s is slower than 12 Gb/s, and 12 Gb/s is slower than PCIe Gen3 (8 Gb/s per lane), but what about mixing SATA 3 Gb/s and SATA 6 Gb/s, or mixing 6 Gb/s and 12 Gb/s SAS?

The general rule is not to mix. When a 6 Gb/s device is introduced to a 12 Gb/s bus, the entire bus slows down to 6 Gb/s, greatly reducing the overall throughput capability. Where this really would hurt is in a dense SAS SSD system. If there are 24 SAS SSDs on a two-channel, 12 Gb/s bus and one of the devices is only 6 Gb/s, then the 12 Gb/s SAS drives that can push 850 MB/s now oversubscribe the bus due to the reduced data rate.

Another consideration is the presence of bus expanders. Bus expanders allow a system to multiplex multiple devices on a single channel. The result is higher density at lower performance maximum. In some cases, the expanders may work acceptably, such as with HDDs, but with SSDs, they are likely to quickly become a bottleneck.

Below are some generic tuning options applicable to performance tuning for server platforms:

Ceph makes use of both a Write-Ahead-Log (WAL) and RocksDB. The WAL is the internal journal where small writes are queued before commiting to the backend storage. RocksDB is where Ceph stores metadata associated with the objects written to BlueStore. When using spinning media, or perhaps even SSDs, it generally makes sense to locate the RocksDB and WAL on a faster device, such as NVMe. When doing so, proper sizing of these is critical to ensuring a stable performance profile of the cluster over time.

From a performance perspective, the rule of thumb is to divide the write performance of the WAL/RocksDB device by the write performance of the data device. This yields what should be considered to be the maximum ratio of data devices per WAL/RocksDB device.

4.3.1.2 RocksDB #

RocksDB operates with a series of tiered levels, each being an order of magnitude larger than the last. Levels one through four are 256 MB, 2.56 GB, 25.6 GB, 256 GB respectively. Allocating appropriate space for these is an act of aggregation. Given that few installations utilize enough metadata to require the fourth tier, allocating for the first three and associated maintenance is sufficient. 25.6+2.56+.256 GB = 28.416 GB. Rounding up to 30 GB and providing 100% overhead to allow for maintenance takes the space allocation suggested for the first three tiers to 60 GB.

Note

Keep in mind that we recommend reserving 4 GB for the WAL device. The recommended size for DB is a total of 64 GB for most workloads. See Book “Deployment Guide”, Chapter 2 “Hardware Requirements and Recommendations”, Section 2.4.3 “Recommended Size for the BlueStore's WAL and DB Device” for more information.

Making the decision to provision fast space for the fourth tier of RocksDB is entirely related to the expected metadata load. Protocols like RBD use little metadata, while CephFS is somewhere from a mild to moderate amount. S3 and native RADOS can utilize the highest amounts of metadata and are generally the cases where it starts making sense to evaluate whether it makes sense to move the fourth tier makes to faster media.

To perform iperf3 tests for network performance, consider increasing the window size (with the -w option) and running multiple streams to fully test the bandwidth capability. If using the standard MTU, your NICs should be capable of running at approximately 70-80% of the advertised bandwidth. If you move up to jumbo frames, the NIC should be able to saturate the link.

This is not necessarily true for faster topologies such as 100 Gb. In those topologies, saturating the NIC can require substantial OS and driver tuning, in combination with ensuring the hardware has the appropriate CPU clock speeds and settings.

This is a sample of the iperf3 commands used on a 100 Gb network. In the command line, the -N disables Nagle's buffering algorithm and the -l sets the buffer length to higher than the default 128 kB, resulting in slightly higher throughput.

  server# iperf3 -s

  client# iperf3   -c server -N -l 256k
  Connecting to host sr650-1, port 5201
  [  4] local 172.16.227.22 port 36628 connected to 172.16.227.21 port 5201
  [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
  [  4]   0.00-1.00   sec  4.76 GBytes  40.9 Gbits/sec    0   1.48 MBytes
  [  4]   1.00-2.00   sec  4.79 GBytes  41.1 Gbits/sec    0   2.52 MBytes
  [  4]   2.00-3.00   sec  4.73 GBytes  40.6 Gbits/sec    0   2.52 MBytes
  [  4]   3.00-4.00   sec  4.73 GBytes  40.6 Gbits/sec    0   2.52 MBytes
  [  4]   4.00-5.00   sec  4.74 GBytes  40.7 Gbits/sec    0   2.52 MBytes
  [  4]   5.00-6.00   sec  4.73 GBytes  40.6 Gbits/sec    0   2.52 MBytes
  [  4]   6.00-7.00   sec  4.72 GBytes  40.6 Gbits/sec    0   2.52 MBytes
  [  4]   7.00-8.00   sec  4.72 GBytes  40.6 Gbits/sec    0   2.52 MBytes
  [  4]   8.00-9.00   sec  4.73 GBytes  40.7 Gbits/sec    0   2.52 MBytes
  [  4]   9.00-10.00  sec  4.73 GBytes  40.6 Gbits/sec    0   2.52 MBytes
  - - - - - - - - - - - - - - - - - - - - - - - - -
  [ ID] Interval           Transfer     Bandwidth       Retr
  [  4]   0.00-10.00  sec  47.4 GBytes  40.7 Gbits/sec    0             sender
  [  4]   0.00-10.00  sec  47.4 GBytes  40.7 Gbits/sec                  receiver

Use fio to test individual storage devices to understand the per-device performance maxima. Do this for all devices to ensure that none are out of specification, or are connected to expanders which lower bandwidth. Doing an exhaustive study of different I/O sizes and patterns would provide the most information about performance expectations, but is beyond the scope of this document.

We recommend testing at least random 4 kB, random 64 kB, and sequential 64 kB and 1 MB buffers. This should give a reasonable overall view of the device’s performance characteristics. When testing, it is important to use the raw device (/dev/sdX) and to use the direct=1 option with multiple jobs to maximize device performance under stress.

Make sure that the test size (dataset) is large enough to over-run any caches that may apply. We recommend using the ramp-time parameter to allow for enough time for the cache overrun to occur before performance measurements are made. This helps to ensure that performance numbers are not tainted by cache-only performance.

Run fio against all devices in an OSD node simultaneously to identify bottlenecks. It should scale very near linearly; if not, check controller firmware, slot placement, and if necessary, split devices across multiple controllers. This simulates the node under heavy load.

We recommend using the same I/O patterns and block sizes that the individual devices were tested with. The job count should be a multiple of the total number of devices in the system, to allow for even distribution across all devices and buses.

  latency_target=10ms
  latency_window=5s
  latency_percentile=99

One key area of kernel performance tuning is to disable its side-channel attack mitigations. The bulk of the benefits from this occur with smaller I/Os, ranging in size from 4 kB to 64 kB. In particular, 64 kB random read and sequential writes doubled in performance in a limited test environment using only two client nodes.

Changing these options requires a clear understanding of the security implications, as they involve disabling mitigations for side-channel attacks on some CPUs. The need to disable these mitigations may be minimized with newer processors. You must carefully evaluate whether this is something needed for the particular hardware being utilized. In the test configuration, a Salt state was utilized to apply these changes. The Salt state should be in a subdirectory of /srv/salt on the Salt master and is applied by using a salt state.apply command similar to below:

salt '*' state.apply my_kerntune

The Salt state and steps used in this testing can be found in Appendix A, Salt State for Kernel Tuning. This needs to be adjusted to work in each customer environment. An example of adjusting the grub2 configuration file can also be found in the appendix.

The first aspect to tune is to ensure that I/O is flowing in the most optimal pattern. For the test cluster used in this test, that means enabling multi-queue block I/O. This is done through adding a boot-time kernel parameter, as found in Section 12.4 https://documentation.suse.com/sles/15-SP1/html/SLES-all/cha-tuning-io.html#cha-tuning-io-barrier. This is not an action that should be taken unilaterally on clusters which contain spinning media devices, due to potential performance degradation for those devices. The general result is that there multiple I/O queues are assigned to each device, allowing more jobs to be handled simultaneously by those device that can service large numbers of requests, such as SSD and NVMe drives.

To get the best read performance, it may be necessary to adjust the read_ahead and write cache settings for the SSD devices. In our particular testing environment, disabling write cache and forcing read_ahead to 2 MB resulted in the best overall performance.

Before tuning, it is important to check and record the default values, and measure any differences in performance compared to that baseline.

By placing the following file in /etc/udev/rules.d, devices will be detected according to the model name shown in /sys/block/{devname}/device/model, and instructed to disable write caching and set the read_ahead_kb option to 2 MB.

  /etc/udev/rules.d/99-ssd.rules

  # Setting specific kernel parameters for a subset of block devices (Intel SSDs)
  SUBSYSTEM=="block", ATTRS{model}=="INTEL SSDS*", ACTION=="add|change", ATTR{queue/read_ahead_kb}="2048"
  SUBSYSTEM=="block", ATTRS{model}=="INTEL SSDS*", ACTION=="add|change", RUN+="/sbin/hdparm -W 0 /dev/%k"

Proper tuning of the network stack can substantially assist improving the latency and throughput of the cluster. A full script for the testing we performed can be found in Appendix C, Network Tuning.

The first change, and one with the highest impact, is to utilize jumbo frame packets. For this to be done, all interfaces utilizing the cluster must be set to use an MTU of 9000 bytes. Network switches are often set to use 9100 or higher. This is suitable, as they are only passing packets, not creating them.

5.2.4.1 Network Device Tuning #

5.2.4.1.1 Jumbo Frames #

The following Salt command ensures the bonded interfaces on all nodes under control (including the test load generation nodes) were utilizing an MTU of 9000:

salt '*' cmd.run 'ip link set bond0 mtu 9000'

To set this persistently, utilize YaST to set the MTU for the bonded interface.

5.2.4.1.2 PCIe Bus Adjustment #

Adjusting the PCIe maximum read request size can provide a slight boost to performance. Be aware that this tuning is card- and slot-specific and must only be done in conjunction with the conditions and instructions supplied by the manufacturer. The maximum PCIe read request size was set with the following Salt commands:

Warning

This should only be done with guidance from the NIC manufacturer and is specific to bus location, driver version and hardware.

  salt '*' cmd.run 'setpci -s 5b:00.0 68.w=5936'
  salt '*' cmd.run 'setpci -s 5b:00.1 68.w=5936'

5.2.4.1.3 TCP RSS #

The next item on the tuning list is helpful in ensuring that a single CPU core is not responsible for all packet processing. A small script is used to spread the I/O across multiple local (from a NUMA perspective) cores.

Note

This is not necessary if the number of queues returned by ls /sys/class/net/{ifname}/queues/rx-*|wc -l is equal to the number of physical cores in a single CPU socket.

salt '*' cmd.run 'for j in `cat /sys/class/net/bond0/bonding/slaves`;do \
LOCAL_CPUS=`cat /sys/class/net/$j/device/local_cpus`;echo $LOCAL_CPUS > \
/sys/class/net/$j/queues/rx-0/rps_cpus;done'

5.2.4.1.4 Ring Buffers #

Many NIC drivers start with a default value for the receive (RX) and transmit (TX) buffers that is not optimal for high-throughput scenarios, and does not allow enough time for the kernel to drain the buffer before it fills up.

The current and maximum settings can be revealed by issuing the following command to the proper NICs:

ethtool -g eth4

The output from this command should look similar to this:

  Ring parameters for eth4:
  Pre-set maximums:
  RX:		8192
  RX Mini:	0
  RX Jumbo:	0
  TX:		8192
  Current hardware settings:
  RX:		1024
  RX Mini:	0
  RX Jumbo:	0
  TX:		1024

Here we can see that the NIC can allocate buffers of up to 8 kB, but it iscurrently using ones of only 1 kB. To adjust this for the cluster, issue the following command:

  salt '*' cmd.run 'ethtool -G eth4 rx 8192 tx 8192'
  salt '*' cmd.run 'ethtool -G eth5 rx 8192 tx 8192'

Setting this value persistently can be achieved via the YaST configuration module:

Figure 5.1: YaST Configuration Module #

Additionally, the settings can be made persistent by editing the configuration files for the physical interfaces found in /etc/sysconfig/network. A script can be found in Appendix B that will change all interfaces to the maximum ring buffer value.

  salt '*' cmd.run 'sysctl -w net.ipv4.tcp_low_latency=1'

  salt '*' cmd.run 'sysctl -w net.ipv4.tcp_fastopen=1'

salt '*' cmd.run 'sysctl -w net.ipv4.tcp_rmem="10240 87380 2147483647"'
salt '*' cmd.run 'sysctl -w net.ipv4.tcp_wmem="10240 87380 2147483647"'

salt '*' cmd.run 'sysctl -w net.ipv4.tcp_timestamps=1'

salt '*' cmd.run 'sysctl -w net.ipv4.tcp_sack=1'

salt '*' cmd.run 'sysctl -w net.core.netdev_max_backlog=250000'

salt '*' cmd.run 'sysctl -w net.core.somaxconn=2048'

salt '*' cmd.run 'sysctl -w net.core.rmem_max=2147483647'
salt '*' cmd.run 'sysctl -w net.core.wmem_max=2147483647'

It is possible to disable all logging to reduce latency in the various codepaths.

Warning

This tuning should be used with caution and understanding that logging will need to be re-enabled should support be required. This implies that an incident would need to be reproduced after logging is re-enabled.

  debug ms=0
  debug mds=0
  debug osd=0
  debug optracker=0
  debug auth=0
  debug asok=0
  debug bluestore=0
  debug bluefs=0
  debug bdev=0
  debug kstore=0
  debug rocksdb=0
  debug eventtrace=0
  debug default=0
  debug rados=0
  debug client=0
  debug perfcounter=0
  debug finisher=0

Under certain conditions where the cluster is physically secure and isolated inside a secured network with no external exposure, it is possible to disable cephx. There are two levels at which cephx can be disabled. The first is to disable signing of authentication traffic. This can be accomplished with the following settings:

cephx_require_signatures = false
cephx_cluster_require_signatures = false
cephx_sign_messages = false

The second level of tuning completely disables cephx authentication. This should only be done on networks that are isolated from public network infrastructure. This change is achieved by adding the following three lines in the global section:

auth cluster required = none
auth service required = none
auth client required = none

The backend processes for performing RADOS operations show up in throttle-*objector_ops when dumping various daemons. If there is too much time being spent in wait, there may be some performance to gain by increasing the memory for in-flight ops or by increasing the total number of inflight operations overall.

objecter inflight op bytes = 1073741824 # default 100_M
objecter inflight ops = 24576

Increasing the number of op threads may be helpful with SSD and NVMe devices, as it provides more work queues for operations.

osd_op_num_threads_per_shard = 4

In checking the performance of BlueStore, it is important to understand if your metadata is spilling over from the high-speed device, if defined, to the bulk data storage device. The parameters useful in this case are found under bluefs slow_used_bytes. If slow_used_bytes is greater than zero, the cluster is using the storage device instead of the RocksDB/WAL device. This is an indicator that more space needs to be allocated to RocksDB/WAL.

Starting with the Ceph Nautilus release, spillover is shown in the output of the ceph health command.

The process of allocating more space depends on how the OSD was deployed. If it was deployed by a version prior to SUSE Enterprise Storage 6, the OSD will need to be re-deployed. If it was deployed with version 6 or after, it may be possible to expand the LVM that the RocksDB and WAL reside on, subject to available space.

Ceph is thin provisioned, including the Write-Ahead Log (WAL) files. By pre-extending the files for the WAL, time is saved by not having to engage the allocator. It also potentially reduces the likelihood of fragmentation of the WAL files. This likely only provides benefit during the early life of the cluster.

bluefs_preextend_wal_files=1

BlueStore has the ability to perform buffered writes. Buffered writes enable populating the read cache during the write process. This setting, in effect, changes the BlueStore cache into a write-through cache.

bluestore_default_buffered_write = true

To prevent writes to the WAL when using a fast device, such as SSD and NVMe, set:

prefer_deferred_size_ssd=0 (pre-deployment)

Important

The following settings are not necessary for fresh deployments. Apply only to upgraded deployments or early SUSE Enterprise Storage 6 deployments as they may still benefit.

The following settings have been shown to slightly improve small object write performance under mixed workload conditions. Reducing the alloc_size to 4 kB helps reduce write amplification for small objects and with erasure coded pools of smaller objects. This change needs to be done before OSD deployment. If done after the fact, the OSDs will need to be re-deployed for it to take effect.

It is advised that spinning media continue to use 64 kB while SSD/NVMe are likely to benefit from setting to 4 kB.

min_alloc_size_ssd=4096
min_alloc_size_hdd=65536

Warning

Setting the alloc_size_ssd to 64 kB may reduce maximum throughput capability of the OSD.

Increasing the BlueStore cache size can improve performance with many workloads. While FileStore OSDs cache data in the kernel's page cache, BlueStore OSDs cache data within the memory allocated by the OSD daemon itself. The OSD daemon will allocate memory up to its memory target (as controlled by the osd_memory_target parameter), and this determines the potential size of the BlueStore cache. The BlueStore cache is a read cache that by default is populated when objects are read. By setting the cache's minimum size higher than the default, it is guaranteed that the value specified will be the minimum cache available for each OSD. The idea is that more low-probability cache hits may occur.

The default osd_memory_target value is 4 GB: For example, each OSD daemon running on a node can be expected to consume that much memory. If a node's total RAM is significantly higher than number of OSDs × 4 GB and there are no other daemons running on the node, performance can be increased by increasing the value of osd_memory_target. This should be done with care to ensure that the operating system will still have enough memory for its needs, while leaving a safety margin.

If you want to ensure that the BlueStore cache will not fall below a certain minimum, use the osd_memory_cache_min parameter. Here is an example (the values are expressed in bytes):

osd_memory_target = 6442450944
osd_memory_cache_min = 4294967296

Tip

As a best practice, start with the full memory of the node. Deduct 16 GB or 32 GB for the OS and then deduct appropriate amounts for any other workloads running on the node. For example, MDS cache if the MDS is colocated. Divide the remainder by the number of OSDs on that host. Ensure you leave room for improvement. For example:

(256 GB - 32 GB ) / 20 OSDs = 11,2 GB/OSD (max)

Using this example, configure approximately 8 or 10 GB per OSD.

By default, BlueStore automatically tunes cache ratios between data and key-value data. In some cases it may be helpful to manually tune the ratios or even increase the cache size. There are several relevant counters for the cache:

If the misses are high, it is worth experimenting with increasing the cache settings or adjusting the ratios.

Adjusting the BlueStore cache size above default has the potential to improve performance of small-block workloads. This can be done globally by adjusting the _cache_size value. By default, the cluster utilizes different values for HDD and SSD/NVMe devices. The best practice would be to increase the specific media cache you are interested in tuning:

Note

If the cache size parameters are adjusted and auto mode is utilized, osd_memory_target should be adjusted to accomodate the OSD base RAM and cache allocation.

In some cases, manually tuning the cache allocation percentage may improve performance. This is achieved by modifying the disabling autotuning of the cache with this configuration line:

bluestore_cache_autotune=0

Changing this value invalidates tuning of the osd_memory_cache_min value.

The cache allocations are modified by adjusting the following:

Any unspecified portion is used for caching the objects themselves.

As RBD is a native protocol, the tuning is directly related to OSD or general Ceph core options that are covered in previous sections.

Read ahead cache defaults to 512 kB; test by tuning up and down on the client nodes.

echo {bytes} > /sys/block/rbd0/queue/read_ahead_kb

If your workload performs large sequential reads such as backup and restore, then this can make a significant difference in restore performance.

In filesystems with millions of files, there is some advantage to utilizing very low-latency media, such as NVMe, for the CephFS metadata pool.

Utlizing the ceph-daemon perf dump command, there is a significant amount of data that can be examined for the Ceph Metadata Servers. It should be noted that the MDS perf counters only apply to metadata operations. The actual IO path is from clients straight to OSDs.

CephFS supports multiple metadata servers. These servers can operate in a multiple-active mode to provide load balancing of the metadata operation requests. To identify whether the MDS infrastructure is under-performing, one would examine the MDS data for request count and reply latencies. This should be done during an idle period on the cluster to form a baseline and then compared when under load. If the average time for reply latencies climbs too high, the MDS server needs to be examined further to identify whether the number of active metadata servers should be augmented, or whether simply increasing the metadata server cache may be sufficient. A sample of the output from the general MDS data for count and reply latency are below:

  "mds": {
    # request count, interesting to get a sense of MDS load
           "request": 0,
           "reply": 0,
    # reply and the latencies of replies can point to load issues
           "reply_latency": {
               "avgcount": 0,
               "sum": 0.000000000,
               "avgtime": 0.000000000
           }
          }

Examining the mds_mem section of the output can help with understanding how the cache is utilized. High inode counters can indicate that a large number of files are open concurrently. This generally indicates that more memory may need to be provided to the MDS. If MDS memory cannot be increased, additional active MDS daemons should be deployed.

  "mds_mem": {

           "ino": 13,
           "ino+": 13,
           "ino-": 0,
           "dir": 12,
           "dir+": 12,
           "dir-": 0,
           "dn": 10,
           "dn+": 10,
           "dn-": 0,

A high cap count can indicate misbehaving clients. For example, clients that do not hand back caps. This may indicate that some clients need to be upgraded to a more recent version, or that the client needs to be investigated for possible issues.

  "cap": 0,
  "cap+": 0,
  "cap-": 0,

This final section shows memory utilization. The RSS value is the current memory size used. If this is roughly equal to the mds_cache_memory_limit, the MDS could probably use more memory.

  "rss": 41524,
  "heap": 314072
},

Another important aspect of tuning a distributed file system is recognizing problematic workloads. The output values below provide some insight to what the MDS daemon is spending its time on. Each heading has the same three attributes as the req_create_latency. With this information, it may be possible to better tune the workloads.

  "mds_server": {
           "dispatch_client_request": 0,
           "dispatch_server_request": 0,
           "handle_client_request": 0,
           "handle_client_session": 0,
           "handle_slave_request": 0,
           "req_create_latency": {
               "avgcount": 0,
               "sum": 0.000000000,
               "avgtime": 0.000000000
           },
           "req_getattr_latency": {},
           "req_getfilelock_latency": {},
           "req_link_latency": {},
           "req_lookup_latency": {},
           "req_lookuphash_latency": {},
           "req_lookupino_latency": {},
           "req_lookupname_latency": {},
           "req_lookupparent_latency": {},
           "req_lookupsnap_latency": {},
           "req_lssnap_latency": {},
           "req_mkdir_latency": {},
           "req_mknod_latency": {},
           "req_mksnap_latency": {},
           "req_open_latency": {},
           "req_readdir_latency": {},
           "req_rename_latency": {},
           "req_renamesnap_latency": {},
           "req_rmdir_latency": {},
           "req_rmsnap_latency": {},
           "req_rmxattr_latency": {},
           "req_setattr_latency": {},
           "req_setdirlayout_latency": {},
           "req_setfilelock_latency": {},
           "req_setlayout_latency": {},
           "req_setxattr_latency": {},
           "req_symlink_latency": {},
           "req_unlink_latency": {},
       }

Tuning the metadata server cache allows for more metadata operations to come from RAM, resulting in improved performance. The example below sets the cache to 16 GB.

mds_cache_memory_limit=17179869184

From the client side, there are a number of performance affecting mount options that can be employed. It is important to understand the potential impact on the applications being utilized before employing these options.

The following mount options may be adjusted to improve performance, but we recommend that their impact is clearly understood prior to implementation in a production environment.

The ideal situation is to understand how many total objects a bucket will host as this allows a bucket to be created with an appropriate number of shards at the outset. To gather information on bucket sharding, issue:

radosgw-admin bucket limit check

The output of this command appears like the following format:

  "user_id": "myusername",
          "buckets": [
              {
                  "bucket": "mybucketname",
                  "tenant": "",
                  "num_objects": 611493,
                  "num_shards": 50,
                  "objects_per_shard": 12229,
                  "fill_status": "OK"
              }
          ]

By default, Ceph reshards buckets to try and maintain reasonable performance. If it is known ahead of time how many shards a bucket may need, based on a ratio of 1 shard per 100 000 objects, it may be pre-sharded. This reduces contention and potential latency issues when resharding will occur. To pre-shard the bucket, it should be created and then submitted for sharding with the rgw-admin command. For example:

radosgw-admin bucket reshard --bucket={bucket name} --num-shards={prime number}

Where the num-shards is a prime number. Each shard should represent about 100 000 objects.

If a process relies on listing the buckets on a frequent basis to iterate through results, yet only uses a small number of results for each iteration it is useful to set the rgw_max_listing_results parameter.

By default, the Object Gateway process is limited to eight simultaneous I/O operations for the index. This can be adjusted with the rgw_bucket_index_max_aio parameter.

When working with larger objects, increasing the size of the Object Gateway windows for put and get can help with performance. Modify the following values in the Object Gateway section of the configuration:

rgw put obj min window size = [size in bytes, 16MiB default]
rgw get obj min window size = [size in bytes, 16MiB default]

Nagle's algorithm was introduced to maximize the use of buffers and attempt to reduce the number of small packets transmitted over the network. While this is helpful in lower bandwdith environments, it can represent a performance degredation in high-bandwidth environments. Disabling it from RGW nodes can improve performance. Including the following in the Ceph configuation RGW section:

tcp_nodelay=1

The default replication setting keeps three total copies of every object written. The provides a high level of data protection by allowing up to two devices or nodes to fail while still protecting the data.

There are use cases where protecting the data is not important, but where performance is. In these cases, such as HPC scratch storage, it may be worthwhile to lower the replication count. This can be achieved by issuing a command such as:

ceph osd pool set rbd size 2

When using erasure coding, it is best to utilize optimized coding pool sizes. Experimental data suggests that the optimial pool sizes have either four or eight data chunks. It is also important to map this in relation to your failure domain model. If your cluster failure domain is at the node level, you will need at least k+m number of nodes. Similarly, if your failure domain it at the rack level, then your cluster needs to be spread over k+m racks. The key consideration is that distribution of the data in relation to the failure domain should be taken into consideration.

When using erasure coding schemes with failure domains larger than a single node, the use of Local Reconstruction Codes (LRC) may be beneficial due to lowered utilization of the network backbone, especially during failure and recovery scenatios.

There are particular use cases where erasure coding may even increase performance. These are mostly limited to large block (1 MB+) sequential read/write workloads. This is due to the parallelization of I/O requests that occurs when splitting objects into chunks to write to multiple OSDs.

Hit set parameters allow for tuning of cache pools. Hit sets in Ceph are usually bloom filters and provide a memory-efficient way of tracking objects that are already in the cache pool.

The hit set is a bit array that is used to store the result of a set of hashing functions applied on object names. Initially, all bits are set to 0. When an object is added to the hit set, its name is hashed and the result is mapped on different positions in the hit set, where the value of the bit is then set to 1.

To find out whether an object exists in the cache, the object name is hashed again. If any bit is 0, the object is definitely not in the cache and needs to be retrieved from cold storage.

It is possible that the results of different objects are stored in the same location of the hit set. By chance, all bits can be 1 without the object being in the cache. Therefore, hit sets working with a bloom filter can only tell whether an object is definitely not in the cache and needs to be retrieved from cold storage.

A cache pool can have more than one hit set tracking file access over time. The setting hit_set_count defines how many hit sets are being used, and hit_set_period defines for how long each hit set has been used. After the period has expired, the next hit set is used. If the number of hit sets is exhausted, the memory from the oldest hit set is freed and a new hit set is created. The values of hit_set_count and hit_set_period multiplied by each other define the overall time frame in which access to objects has been tracked.

Figure 7.1: Bloom Filter with 3 Stored Objects #

Compared to the number of hashed objects, a hit set based on a bloom filter is very memory-efficient. Less than 10 bits are required to reduce the false positive probability below 1%. The false positive probability can be defined with hit_set_fpp. Based on the number of objects in a placement group and the false positive probability Ceph automatically calculates the size of the hit set.

The required storage on the cache pool can be limited with min_write_recency_for_promote and min_read_recency_for_promote. If the value is set to 0, all objects are promoted to the cache pool as soon as they are read or written and this persists until they are evicted. Any value greater than 0 defines the number of hit sets ordered by age that are searched for the object. If the object is found in a hit set, it will be promoted to the cache pool. Keep in mind that backing up objects may also cause them to be promoted to the cache. A full backup with the value of '0' can cause all data to be promoted to the cache tier while active data gets removed from the cache tier. Therefore, changing this setting based on the backup strategy may be useful.

Note

The longer the period and the higher the min_read_recency_for_promote and min_write_recency_for_promote values, the more RAM the ceph-osd daemon consumes. In particular, when the agent is active to flush or evict cache objects, all hit_set_count hit sets are loaded into RAM.

7.7.1.1 Use GMT for Hit Set #

Cache tier setups have a bloom filter called hit set. The filter tests whether an object belongs to a set of either hot or cold objects. The objects are added to the hit set using time stamps appended to their names.

If cluster machines are placed in different time zones and the time stamps are derived from the local time, objects in a hit set can have misleading names consisting of future or past time stamps. In the worst case, objects may not exist in the hit set at all.

To prevent this, the use_gmt_hitset defaults to '1' on a newly created cache tier setups. This way, you force OSDs to use GMT (Greenwich Mean Time) time stamps when creating the object names for the hit set.

Warning: Leave the Default Value

Do not touch the default value '1' of use_gmt_hitset. If errors related to this option are not caused by your cluster setup, never change it manually. Otherwise, the cluster behavior may become unpredictable.

The cache tiering agent performs two main functions:

7.7.2.1 Absolute Sizing #

The cache tiering agent can flush or evict objects based on the total number of bytes or the total number of objects. To specify a maximum number of bytes, execute the following:

cephadm@adm > ceph osd pool set cachepool target_max_bytes num_of_bytes

To specify the maximum number of objects, execute the following:

cephadm@adm > ceph osd pool set cachepool target_max_objects num_of_objects

Note

Ceph is not able to determine the size of a cache pool automatically, therefore configuration of the absolute size is required here. Otherwise, flush and evict will not work. If you specify both limits, the cache tiering agent will begin flushing or evicting when either threshold is triggered.

Note

All client requests will be blocked only when target_max_bytes or target_max_objects is reached.

cephadm@adm > ceph osd pool set cachepool cache_target_dirty_ratio 0.0...1.0

cephadm@adm > ceph osd pool set hot-storage cache_target_dirty_ratio 0.4

cephadm@adm > ceph osd pool set cachepool cache_target_dirty_high_ratio 0.0..1.0

cephadm@adm > ceph osd pool set cachepool cache_target_full_ratio 0.0..1.0

You can specify the minimum age of a recently modified (dirty) object before the cache tiering agent flushes it to the backing storage pool. Note that this will only apply if the cache actually needs to flush/evict objects:

cephadm@adm > ceph osd pool set cachepool cache_min_flush_age num_of_seconds

You can specify the minimum age of an object before it will be evicted from the cache tier:

cephadm@adm > ceph osd pool set cachepool cache_min_evict_age num_of_seconds

To add LVM cache to an existing OSD, use the following command:

cephadm@osd > ceph-volume lvmcache add
 --cachemetadata METADATA-PARTITION
 --cachedata DATA-PARTITION
 --osd-id OSD-ID

The optional --data, --db or --wal specifies which partition to cache. Default is --data.

Tip: Specify Logical Volume (LV)

Alternatively, you can use the --origin instead of the --osd-id option to specify which LV to cache:

[...]
--origin VOLUME-GROUP/LOGICAL-VOLUME

To remove existing LVM cache from an OSD, use the following command:

cephadm@osd > ceph-volume lvmcache rm --osd-id OSD-ID

To specify caching mode, use the following command:

cephadm@osd > ceph-volume lvmcache mode --set CACHING-MODE --osd-id OSD-ID

CACHING-MODE is either 'writeback' (default) or 'writethrough'