This is a draft document that was built and uploaded automatically. It may document beta software and be incomplete or even incorrect. Use this document at your own risk.

Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Administration Guide / Operating a Cluster / Ceph Cluster Configuration
Applies to SUSE Enterprise Storage 6

25 Ceph Cluster Configuration

This chapter provides a list of important Ceph cluster settings and their description. The settings are sorted by topic.

25.1 Runtime Configuration

Section 2.14, “Adjusting ceph.conf with Custom Settings” describes how to make changes to the Ceph configuration file ceph.conf. However, the actual cluster behavior is determined not by the current state of the ceph.conf file but by the configuration of the running Ceph daemons, which is stored in memory.

You can query an individual Ceph daemon for a particular configuration setting using the admin socket on the node where the daemon is running. For example, the following command gets the value of the osd_max_write_size configuration parameter from the daemon named osd.0: 

cephadm@adm > ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok \
config get osd_max_write_size
{
  "osd_max_write_size": "90"
}

You can also change the daemons' settings at runtime. Remember that this change is temporary and will be lost after the next daemon restart. For example, the following command changes the osd_max_write_size parameter to '50' for all OSDs in the cluster: 

cephadm@adm > ceph tell osd.* injectargs --osd_max_write_size 50
Warning
Warning: injectargs Is Not Reliable

Unfortunately, changing the cluster settings with the injectargs command is not 100% reliable. If you need to be sure that the changed parameter is active, change it in the configuration files on all cluster nodes and restart all daemons in the cluster.

25.2 Ceph OSD and BlueStore

25.2.1 Automatic Cache Sizing

BlueStore can be configured to automatically resize its caches when tc_malloc is configured as the memory allocator and the bluestore_cache_autotune setting is enabled. This option is currently enabled by default. BlueStore will attempt to keep OSD heap memory usage under a designated target size via the osd_memory_target configuration option. This is a best effort algorithm and caches will not shrink smaller than the amount specified by osd_memory_cache_min. Cache ratios will be chosen based on a hierarchy of priorities. If priority information is not available, the bluestore_cache_meta_ratio and bluestore_cache_kv_ratio options are used as fallbacks.

bluestore_cache_autotune

Automatically tunes the ratios assigned to different BlueStore caches while respecting minimum values. Default is True.

osd_memory_target

When tc_malloc and bluestore_cache_autotune are enabled, try to keep this many bytes mapped in memory.

Note
Note

This may not exactly match the RSS memory usage of the process. While the total amount of heap memory mapped by the process should generally stay close to this target, there is no guarantee that the kernel will actually reclaim memory that has been unmapped.

osd_memory_cache_min

When tc_malloc and bluestore_cache_autotune are enabled, set the minimum amount of memory used for caches.

Note
Note

Setting this value too low can result in significant cache thrashing.

25.3 Ceph Object Gateway

You can influence the Object Gateway behavior by a number of options in the /etc/ceph/ceph.conf file under sections named 

[client.radosgw.INSTANCE_NAME]

If an option is not specified, its default value is used. A complete list of the Object Gateway options follows:

25.3.1 General Settings

rgw_frontends

Configures the HTTP front-end(s). Specify multiple front-ends in a comma-delimited list. Each front-end configuration may include a list of options separated by spaces, where each option is in the form “key=value” or “key”. Default is 

rgw_frontends = beast port=7480
rgw_data

Sets the location of the data files for the Object Gateway. Default is /var/lib/ceph/radosgw/CLUSTER_ID.

rgw_enable_apis

Enables the specified APIs. Default is 's3, swift, swift_auth, admin All APIs'.

rgw_cache_enabled

Enables or disables the Object Gateway cache. Default is 'true'.

rgw_cache_lru_size

The number of entries in the Object Gateway cache. Default is 10000.

rgw_socket_path

The socket path for the domain socket. FastCgiExternalServer uses this socket. If you do not specify a socket path, the Object Gateway will not run as an external server. The path you specify here needs to be the same as the path specified in the rgw.conf file.

rgw_fcgi_socket_backlog

The socket backlog for fcgi. Default is 1024.

rgw_host

The host for the Object Gateway instance. It can be an IP address or a host name. Default is 0.0.0.0

rgw_port

The port number where the instance listens for requests. If not specified, the Object Gateway runs external FastCGI.

rgw_dns_name

The DNS name of the served domain.

rgw_script_uri

The alternative value for the SCRIPT_URI if not set in the request.

rgw_request_uri

The alternative value for the REQUEST_URI if not set in the request.

rgw_print_continue

Enable 100-continue if it is operational. Default is 'true'.

rgw_remote_addr_param

The remote address parameter. For example, the HTTP field containing the remote address, or the X-Forwarded-For address if a reverse proxy is operational. Default is REMOTE_ADDR.

rgw_op_thread_timeout

The timeout in seconds for open threads. Default is 600.

rgw_op_thread_suicide_timeout

The time timeout in seconds before the Object Gateway process dies. Disabled if set to 0 (default).

rgw_thread_pool_size

Number of threads for the Beast server. Increase to a higher value if you need to serve more requests. Defaults to 100 threads.

rgw_num_rados_handles

The number of RADOS cluster handles for Object Gateway. Each Object Gateway worker thread now gets to pick a RADOS handle for its lifetime. This option may be deprecated and removed in future releases. Default is 1.

rgw_num_control_oids

The number of notification objects used for cache synchronization between different rgw instances. Default is 8.

rgw_init_timeout

The number of seconds before the Object Gateway gives up on initialization. Default is 30.

rgw_mime_types_file

The path and location of the MIME types. Used for Swift auto-detection of object types. Default is /etc/mime.types.

rgw_gc_max_objs

The maximum number of objects that may be handled by garbage collection in one garbage collection processing cycle. Default is 32.

rgw_gc_obj_min_wait

The minimum wait time before the object may be removed and handled by garbage collection processing. Default is 2 * 3600.

rgw_gc_processor_max_time

The maximum time between the beginning of two consecutive garbage collection processing cycles. Default is 3600.

rgw_gc_processor_period

The cycle time for garbage collection processing. Default is 3600.

rgw_s3_success_create_obj_status

The alternate success status response for create-obj. Default is 0.

rgw_resolve_cname

Whether the Object Gateway should use DNS CNAME record of the request host name field (if host name is not equal to the Object Gateway DNS name). Default is 'false'.

rgw_obj_stripe_size

The size of an object stripe for Object Gateway objects. Default is 4 << 20.

rgw_extended_http_attrs

Add a new set of attributes that can be set on an entity (for example, a user, a bucket, or an object). These extra attributes can be set through HTTP header fields when putting the entity or modifying it using the POST method. If set, these attributes will return as HTTP fields when requesting GET/HEAD on the entity. Default is 'content_foo, content_bar, x-foo-bar'.

rgw_exit_timeout_secs

Number of seconds to wait for a process before exiting unconditionally. Default is 120.

rgw_get_obj_window_size

The window size in bytes for a single object request. Default is '16 << 20'.

rgw_get_obj_max_req_size

The maximum request size of a single GET operation sent to the Ceph Storage Cluster. Default is 4 << 20.

rgw_relaxed_s3_bucket_names

Enables relaxed S3 bucket name rules for US region buckets. Default is 'false'.

rgw_list_buckets_max_chunk

The maximum number of buckets to retrieve in a single operation when listing user buckets. Default is 1000.

rgw_override_bucket_index_max_shards

Represents the number of shards for the bucket index object. Setting 0 (default) indicates there is no sharding. It is not recommended to set a value too large (for example 1000) as it increases the cost for bucket listing. This variable should be set in the client or global sections so that it is automatically applied to radosgw-admin commands.

rgw_curl_wait_timeout_ms

The timeout in milliseconds for certain curl calls. Default is 1000.

rgw_copy_obj_progress

Enables output of object progress during long copy operations. Default is 'true'.

rgw_copy_obj_progress_every_bytes

The minimum bytes between copy progress output. Default is 1024 * 1024.

rgw_admin_entry

The entry point for an admin request URL. Default is 'admin'.

rgw_content_length_compat

Enable compatibility handling of FCGI requests with both CONTENT_LENGTH AND HTTP_CONTENT_LENGTH set. Default is 'false'.

rgw_bucket_quota_ttl

The amount of time in seconds for which cached quota information is trusted. After this timeout, the quota information will be re-fetched from the cluster. Default is 600.

rgw_user_quota_bucket_sync_interval

The amount of time in seconds for which the bucket quota information is accumulated before synchronizing to the cluster. During this time, other Object Gateway instances will not see the changes in the bucket quota stats related to operations on this instance. Default is 180.

rgw_user_quota_sync_interval

The amount of time in seconds for which user quota information is accumulated before synchronizing to the cluster. During this time, other Object Gateway instances will not see the changes in the user quota stats related to operations on this instance. Default is 180.

rgw_bucket_default_quota_max_objects

Default maximum number of objects per bucket. It is set on new users if no other quota is specified, and has no effect on existing users. This variable should be set in the client or global sections so that it is automatically applied to radosgw-admin commands. Default is -1.

rgw_bucket_default_quota_max_size

Default maximum capacity per bucket in bytes. It is set on new users if no other quota is specified, and has no effect on existing users. Default is -1.

rgw_user_default_quota_max_objects

Default maximum number of objects for a user. This includes all objects in all buckets owned by the user. It is set on new users if no other quota is specified, and has no effect on existing users. Default is -1.

rgw_user_default_quota_max_size

The value for user maximum size quota in bytes set on new users if no other quota is specified. It has no effect on existing users. Default is -1.

rgw_verify_ssl

Verify SSL certificates while making requests. Default is 'true'.

rgw_max_chunk_size

Maximum size of a chunk of data that will be read in a single operation. Increasing the value to 4 MB (4194304) will provide better performance when processing large objects. Default is 128 kB (131072).

Multisite Settings
rgw_zone

The name of the zone for the gateway instance. If no zone is set, a cluster-wide default can be configured with the radosgw-admin zone default command.

rgw_zonegroup

The name of the zonegroup for the gateway instance. If no zonegroup is set, a cluster-wide default can be configured with the radosgw-admin zonegroup default command.

rgw_realm

The name of the realm for the gateway instance. If no realm is set, a cluster-wide default can be configured with theradosgw-admin realm default command.

rgw_run_sync_thread

If there are other zones in the realm to synchronize from, spawn threads to handle the synchronization of data and metadata. Default is 'true'.

rgw_data_log_window

The data log entries window in seconds. Default is 30.

rgw_data_log_changes_size

The number of in-memory entries to hold for the data changes log. Default is 1000.

rgw_data_log_obj_prefix

The object name prefix for the data log. Default is 'data_log'.

rgw_data_log_num_shards

The number of shards (objects) on which to keep the data changes log. Default is 128.

rgw_md_log_max_shards

The maximum number of shards for the metadata log. Default is 64.

Swift Settings
rgw_enforce_swift_acls

Enforces the Swift Access Control List (ACL) settings. Default is 'true'.

rgw_swift_token_expiration

The time in seconds for expiring a Swift token. Default is 24 * 3600.

rgw_swift_url

The URL for the Ceph Object Gateway Swift API.

rgw_swift_url_prefix

The URL prefix for the Swift StorageURL that goes in front of the “/v1” part. This allows to run several Gateway instances on the same host. For compatibility, setting this configuration variable to empty causes the default “/swift” to be used. Use explicit prefix “/” to start StorageURL at the root.

Warning
Warning

Setting this option to “/” will not work if S3 API is enabled. Keep in mind that disabling S3 will make it impossible to deploy the Object Gateway in the multisite configuration!

rgw_swift_auth_url

Default URL for verifying v1 authentication tokens when the internal Swift authentication is not used.

rgw_swift_auth_entry

The entry point for a Swift authentication URL. Default is 'auth'.

rgw_swift_versioning_enabled

Enables the Object Versioning of OpenStack Object Storage API. This allows clients to put the X-Versions-Location attribute on containers that should be versioned. The attribute specifies the name of container storing archived versions. It must be owned by the same user as the versioned container for reasons of access control verification—ACLs are not taken into consideration. Those containers cannot be versioned by the S3 object versioning mechanism. Default is 'false'.

Logging Settings
rgw_log_nonexistent_bucket

Enables the Object Gateway to log a request for a non-existent bucket. Default is 'false'.

rgw_log_object_name

The logging format for an object name. See the manual page man 1 date for details about format specifiers. Default is '%Y-%m-%d-%H-%i-%n'.

rgw_log_object_name_utc

Whether a logged object name includes a UTC time. If set to 'false' (default), it uses the local time.

rgw_usage_max_shards

The maximum number of shards for usage logging. Default is 32.

rgw_usage_max_user_shards

The maximum number of shards used for a single user’s usage logging. Default is 1.

rgw_enable_ops_log

Enable logging for each successful Object Gateway operation. Default is 'false'.

rgw_enable_usage_log

Enable the usage log. Default is 'false'.

rgw_ops_log_rados

Whether the operations log should be written to the Ceph Storage Cluster back-end. Default is 'true'.

rgw_ops_log_socket_path

The Unix domain socket for writing operations logs.

rgw_ops_log_data_backlog

The maximum data backlog data size for operations logs written to a Unix domain socket. Default is 5 << 20.

rgw_usage_log_flush_threshold

The number of dirty merged entries in the usage log before flushing synchronously. Default is 1024.

rgw_usage_log_tick_interval

Flush pending usage log data every 'n' seconds. Default is 30.

rgw_log_http_headers

Comma-delimited list of HTTP headers to include in log entries. Header names are case-insensitive, and use the full header name with words separated by underscores. For example, 'http_x_forwarded_for', 'http_x_special_k'.

rgw_intent_log_object_name

The logging format for the intent log object name. See the manual page man 1 date for details about format specifiers. Default is '%Y-%m-%d-%i-%n'.

rgw_intent_log_object_name_utc

Whether the intent log object name includes a UTC time. If set to 'false' (default), it uses the local time.

Keystone Settings
rgw_keystone_url

The URL for the Keystone server.

rgw_keystone_api_version

The version (2 or 3) of OpenStack Identity API that should be used for communication with the Keystone server. Default is 2.

rgw_keystone_admin_domain

The name of the OpenStack domain with the administrator privilege when using OpenStack Identity API v3.

rgw_keystone_admin_project

The name of the OpenStack project with the administrator privilege when using OpenStack Identity API v3. If not set, the value of the rgw keystone admin tenant will be used instead.

rgw_keystone_admin_token

The Keystone administrator token (shared secret). In the Object Gateway, authentication with the administrator token has priority over authentication with the administrator credentials (options rgw keystone admin user, rgw keystone admin password, rgw keystone admin tenant, rgw keystone admin project, and rgw keystone admin domain). The administrator token feature is considered as deprecated.

rgw_keystone_admin_tenant

The name of the OpenStack tenant with the administrator privilege (Service Tenant) when using OpenStack Identity API v2.

rgw_keystone_admin_user

The name of the OpenStack user with the administrator privilege for Keystone authentication (Service User) when using OpenStack Identity API v2.

rgw_keystone_admin_password

The password for the OpenStack administrator user when using OpenStack Identity API v2.

rgw_keystone_accepted_roles

The roles required to serve requests. Default is 'Member, admin'.

rgw_keystone_token_cache_size

The maximum number of entries in each Keystone token cache. Default is 10000.

rgw_keystone_revocation_interval

The number of seconds between token revocation checks. Default is 15 * 60.

rgw_keystone_verify_ssl

Verify SSL certificates while making token requests to Keystone. Default is 'true'.

25.3.1.1 Additional Notes

rgw_dns_name

If the parameter rgw dns name is added to the ceph.conf, make sure that the S3 client is configured to direct requests at the endpoint specified by rgw dns name.

25.3.2 HTTP Front-ends

25.3.2.1 Beast

port, ssl_port

IPv4 & IPv6 listening port numbers. You can specify multiple port numbers: 

port=80 port=8000 ssl_port=8080

Default is 80.

endpoint, ssl_endpoint

The listening addresses in the form 'address[:port]', where the address is an IPv4 address string in dotted decimal form, or an IPv6 address in hexadecimal notation surrounded by square brackets. Specifying an IPv6 endpoint would listen to IPv6 only. The optional port number defaults to 80 for endpoint and 443 for ssl_endpoint. You can specify multiple addresses: 

endpoint=[::1] endpoint=192.168.0.100:8000 ssl_endpoint=192.168.0.100:8080
ssl_private_key

Optional path to the private key file used for SSL-enabled endpoints. If not specified, the ssl_certificate file is used as a private key.

tcp_nodelay

If specified, the socket option will disable Nagle's algorithm on the connection. It means that packets will be sent as soon as possible instead of waiting for a full buffer or timeout to occur.

'1' disables Nagle's algorithm for all sockets.

'0' keeps Nagle's algorithm enabled (default).

Example 25.1: Example Beast Configuration in /etc/ceph/ceph.conf 
rgw_frontends = beast port=8000 ssl_port=443 ssl_certificate=/etc/ssl/ssl.crt error_log_file=/var/log/radosgw/civetweb.error.log

25.3.2.2 CivetWeb

port

The listening port number. For SSL-enabled ports, add an 's' suffix (for example, '443s'). To bind a specific IPv4 or IPv6 address, use the form 'address:port'. You can specify multiple endpoints either by joining them with '+' or by providing multiple options: 

port=127.0.0.1:8000+443s
port=8000 port=443s

Default is 7480.

num_threads

The number of threads spawned by Civetweb to handle incoming HTTP connections. This effectively limits the number of concurrent connections that the front-end can service.

Default is the value specified by the rgw_thread_pool_size option.

request_timeout_ms

The amount of time in milliseconds that Civetweb will wait for more incoming data before giving up.

Default is 30000 milliseconds.

access_log_file

Path to the access log file. You can specify either a full path, or a path relative to the current working directory. If not specified (default), then accesses are not logged.

error_log_file

Path to the error log file. You can specify either a full path, or a path relative to the current working directory. If not specified (default), then errors are not logged.

Example 25.2: Example Civetweb Configuration in /etc/ceph/ceph.conf 
rgw_frontends = civetweb port=8000+443s request_timeout_ms=30000 error_log_file=/var/log/radosgw/civetweb.error.log

25.3.2.3 Common Options

ssl_certificate

Path to the SSL certificate file used for SSL-enabled endpoints.

prefix

A prefix string that is inserted into the URI of all requests. For example, a Swift-only front-end could supply a URI prefix of '/swift'.