Best practices for Swift deployed by TripleO

Recent director deployments are using Swift for more and more services. As of now, the overcloud Swift deployment will be used by Glance and Gnocchi, and the undercloud is used by Heat to distribute configuration settings. Additionally users can use the overcloud deployment to store their data, and this might result in even more storage usage.

A default director deployment uses the controller disk to store Swift data, in addition to other services like MariaDB and RabbitMQ. All of them will create a large number of IOPS, and the disks on the controller nodes should be able to handle this. Depending on your usage and load you might need to tweak your settings or architecture to avoid bottlenecks, especially when using only a single spinning disk per controller node and no separate storage nodes.

Additionally, it is recommended to run Swift with more than three disks and nodes to increase availability in case some disks or nodes are failing. With the default setup using three replicas, it is quite possible that a request fails if there is a hardware issue affecting one of the disks or nodes and using only three disks. The Swift proxy will always try to write in parallel to multiple nodes/disks, and requires a quorum to confirm a write request to the user. With one failed node/disk it is likely that anoder node/disk is not able to respond within the default timeout, and the request will wail because there is no other node to route the request to if there are not enough nodes.

Note: some of the settings below are only available on OpenStack Platform 10.

Use separate disks for Swift

A default director installation will use a special directory /srv/node/d1 on the system disk for Swift storage. As mentioned earlier, this disk is used by other services as well, and therefore it might become a performance bottle neck if the controller uses a spinning disk. Using one or more separate disk for Swift on the controller nodes is the recommended way for any deployment not using separate Swift storage nodes. This can be done easily by creating an additional Heat template to be used by director. The following template is an example how to use two separate disks on each controller node with Swift. A XFS filesystem will be created on these disks (using the whole disk), and the rings will use them as storage devices.

parameter_defaults:
  SwiftMountCheck: true
  SwiftRawDisks: {"sdb": {}, "sdc": {}}
  SwiftUseLocalDir: false

The SwiftMountCheck parameter will ensure that Swift writes data only if the storage disk is mounted. This will prevent overfilling the root disk. Furthermore, this also enables more checks within Swift to properly detect disk failures and being able to recover from them. It is highly recommended to set the SwiftMountCheck to true if real disks will be used. The SwiftUseLocalDir parameter will disable using the local d1 directory, which is by default stored on the operating system disk.

Make sure Ironic uses the intended root disk if your nodes are using multiple disks. Please refer to Content from docs.openstack.org is not included.Setting the Root Device for Deployment for further information.

Using SSDs on the controller nodes

If only a single disk is used on the controller nodes a SSD is recommended to keep up with the amount of IOPS.

Use separate storage nodes

If enough nodes are available these can be deployed easily as additional Swift storage nodes. If multiple disks are available on these nodes, they should be defined as SwiftRawDisks in a Heat template as described above. A template might look like this:

parameter_defaults:
  SwiftRawDisks: {"sdb": {}, "sdc": {}}
  SwiftMountCheck: true
  SwiftUseLocalDir: false
  ObjectStorageCount: 3

However, this will also require disks sdb and sdc on the controller nodes, because the controller role includes the Swift storage service as well. To disable the Swift storage service on the controller nodes one has to modify the controller role. You can do this by re-defining the ControllerService in your template file like this:

parameter_defaults:
  ControllerServices:
    # include all ControllerServices from /usr/share/openstack-tripleo-heat-templates/roles_data.yaml
    # except  "- OS::TripleO::Services::SwiftStorage"
    - OS::TripleO::Services::CACerts
    # ... many more ...
    - OS::TripleO::Services::VipHosts

Use a different storage backend for Gnocchi

Gnocchi will write and read metric data constantly. Depending on the scale of the deployment this will create a continuous additional load, and if only a single spinning disk is used the disk might be unable to keep up with the load, resulting in timeouts and failed deployments.

If the deployment uses a Ceph storage cluster, this can be used directly by Gnocchi as a backend to store metrics. A possible Heat template might look like this:

parameter_defaults:
  GnocchiBackend: rbd

Finally, if none of the other options are applicable, one can switch back to a file-based backend for Gnocchi. For example::

parameter_defaults:
  GnocchiBackend: file

Increase default partition for large deployments

Swift distributes data across disks and nodes using modified hash rings (more details available in the upstream documentation: Content from docs.openstack.org is not included.The Rings). There are three rings by default - one for accounts, one for containers, and one for objects. Each ring uses a fixed parameter called Partition Power. This value is important and can only be changed for new containers and their objects. It is therefore important to choose the value before initial deployment.
The default value for director-deployed environments is 10. This is a reasonable value for smaller deployments, especially if one only plans to use disks on the controller nodes for Swift. Larger deployments - for example when using separate Swift storage nodes - should use a higher value. The following table will help you to select an appropriate partition power if you're using three replicas. Do not choose a much higher value; this might impact replication times severely.

The partition power can be set before initial deployment using the following Heat template:

parameter_defaults:
  SwiftPartPower: 11

If you started with a small cluster and only a few disks (and used a low partition power therefore) and want to upscale your Swift deployment to many more disks, you can use a Storage Policy and an additional object server ring for new containers.

For Red Hat OpenStack Platform 10, use Ceph RADOSGW as a replacement for Swift

Beginning with Red Hat OpenStack Platform 10, there's available heat templates that can be used with Director to deploy the Ceph RADOSGW instead of Swift. This isn't the same as using Ceph as a back-end for Swift. Instead it is a drop-in replacement of Swift. There are options to consider when selecting this configuration, such as the dependence on Swift services other than the Swift client. Essentially the Swift client is used to query the RADOSGW object service, with swift-proxy and other swift services disabled. See this solution article for additional details.