Defining disks to be used with OpenStack Swift in Red Hat OpenStack Platform 17

Defining disks to be used with SwiftRawDisk

As documented in Disk recommendation for the Object Storage service it is recommended to use additional local disks for Swift object storage. A simple template might look like this:

parameter_defaults:
  SwiftRawDisks: {"sdb": {}, "sdc": {}}

Sometimes disks are not assigned in order, for example the root disk might already use sdb and the deployment might fail. While persistence is guaranteed later on by using disk uuids in /etc/fstab, this is not true during the initial deployment.

Therefore it is required to define disks to be used by their unique ids. This can be enforced by using udev rules to create symlinks to specific disks using their WWN. The following steps are required:

1. Get the introspection data for the disks
2. Select disks and define an udev ruleset
3. Create a pre-config template to store the ruleset on all nodes
4. Deploy the overcloud using the additional pre-config template

1. Get the introspection data for the disks

Disks can be identified using introspection data:

$ openstack baremetal introspection data save <UUID> | jq .inventory.disks
[
  {
    "name": "/dev/vda",
    "model": "",
    "size": 21474836480,
    "rotational": true,
    "wwn": null,
    "serial": null,
    "vendor": "0x1af4",
    "wwn_with_extension": null,
    "wwn_vendor_extension": null,
    "hctl": null,
    "by_path": "/dev/disk/by-path/pci-0000:00:08.0"
  },
  {
    "name": "/dev/vdb",
    "model": "",
    "size": 5368709120,
    "rotational": true,
    "wwn": null,
    "serial": "94ed5ac9-2a5b-45e7-a",
    "vendor": "0x1af4",
    "wwn_with_extension": null,
    "wwn_vendor_extension": null,
    "hctl": null,
    "by_path": "/dev/disk/by-path/pci-0000:00:10.0"
  },
  {
    "name": "/dev/vdc",
    "model": "",
    "size": 5368709120,
    "rotational": true,
    "wwn": null,
    "serial": "3865b489-c805-4bf8-b",
    "vendor": "0x1af4",
    "wwn_with_extension": null,
    "wwn_vendor_extension": null,
    "hctl": null,
    "by_path": "/dev/disk/by-path/virtio-pci-0000:00:11.0"
  }
]

We choose vdb and vdc in this case, and use their serial or wwn identifier.¹

2. Select disks and define an udev ruleset

Based on this data we could create a udev ruleset like this:

# /etc/udev/rules.d/90-swift.rules
# Server 1
ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="*9f43223e-14c3-4631-a*", SYMLINK+="swift0"
ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="*600f5d04-b14f-4922-8*", SYMLINK+="swift1"

# Server 2
ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="*94ed5ac9-2a5b-45e7-a*", SYMLINK+="swift0"
ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="*3865b489-c805-4bf8-b*", SYMLINK+="swift1"

Here we define two unique disks per server using their WWN or serial. Each server will now have symlinks from /dev/swift[0|1] to the corresponding devices, if these exist. This also ensures that there are no duplicates and symlinks are only created on nodes with the correct disks.

3. Create a pre-config template to store the ruleset on all nodes

To create such a udev ruleset we will use the procedure as described in Pre-configuration: customizing all overcloud roles.

# ~/templates/swiftdisks.yaml
heat_template_version: 2014-10-16

description: >
  Extra hostname configuration

parameters:
  server:
    type: string
  DeployIdentifier:
    type: string

resources:
  CustomExtraConfigPre:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config: |
        #!/bin/sh
        cat <<EOF > /etc/udev/rules.d/99-swift.rules
        # Server 1
        ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="*9f43223e-14c3-4631-a*", SYMLINK+="swift0"
        ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="*600f5d04-b14f-4922-8*", SYMLINK+="swift1"
        # Server 2
        ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="*94ed5ac9-2a5b-45e7-a*", SYMLINK+="swift0"
        ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="*3865b489-c805-4bf8-b*", SYMLINK+="swift1"
        EOF
        udevadm trigger

  CustomExtraDeploymentPre:
    type: OS::Heat::SoftwareDeployment
    properties:
      server: {get_param: server}
      config: {get_resource: CustomExtraConfigPre}
      actions: ['CREATE']
      input_values:
        deploy_identifier: {get_param: DeployIdentifier}

outputs:
  deploy_stdout:
    description: Deployment reference, used to trigger pre-deploy on changes
    value: {get_attr: [CustomExtraDeploymentPre, deploy_stdout]}

4. Deploy the overcloud using the additional pre-config template

Now we can simply use swift0 and swift1 as parameters for SwiftRawDisk and create a pre-config template ~/templates/pre_config.yaml:

resource_registry:
  OS::TripleO::ControllerExtraConfigPre: /home/stack/swiftdisks.yaml
  OS::TripleO::ObjectStorageExtraConfigPre: /home/stack/swiftdisks.yaml


parameter_defaults:
  SwiftRawDisks: {"swift0": {}, "swift1": {}}

During deployment one has to use this template in addition to any other template for the overcloud:

$ openstack overcloud deploy --templates \
    ...
    -e /home/stack/templates/pre_config.yaml \
    ...

```
<disk type='file' device='disk'>
  <driver name='qemu' type='qcow2' cache='unsafe'/>
  <source file='/var/lib/libvirt/images/swift-0-disk4.qcow2'/>
  <target dev='vdd' bus='virtio'/>
  <serial>3865b481-c805-4bf8-bfa6-b321f6923901</serial>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x12' function='0x0'/>
</disk>
```