The CRUSH map got over-written to the default after a node reboot, why?

Solution Unverified - Updated

Environment

  • Red Hat Ceph Enterprise 1.2.3

  • Red Hat Ceph Enterprise 1.3

Issue

  • The CRUSH map got over-written to the default after a node reboot, why?

  • All the changes related to OSDs made in the CRUSH map were removed and reset to default when the node got rebooted.

  • The CRUSH map was set with custom hostnames instead of the actual ones, and it was reset back when the OSD processes were restarted, why?

Resolution

  • This is an expected and default behavior for OSD processes.

  • The OSD process, at the time of startup, checks the value of 'hostname -s' and updates the setting "osd_crush_location" in /etc/ceph/ceph.conf.

  • In case the hostname is set to a custom value, the following configuration should be set in /etc/ceph/ceph.conf so that the set value is not changed.

osd crush update on start = false

  • This prevents the OSD startup script to set its own hostname at initialization.

  • This is standard when using a customized CRUSH map.

  • Another way is to define the location of the CRUSH map in ceph.conf. This will override any locations the OSD startup script will define.

[osd.0]
osd crush location = "host=ssd-ceph1"

[osd.1]
osd crush location = "host=sata-ceph1"

Root Cause

  • Custom hostnames were set in the CRUSH map instead of the actual hostnames, under the [osd.{X}] section. This hostname meant that the particular OSD is present on the mentioned host.

  • When the OSD process starts up, it will read the output of 'hostname -s' on the node it is present, and then update that in the CRUSH map. This made the custom hostnames in the CRUSH map to be removed, and get written with the actual short hostname (hostname -s).

  • In this case, the hostnames under the section [osd.{X}] were changed to a custom name. Since the OSD process tries to update it's actual location (the host on where it resides) in the CRUSH map, the custom hostname set manually was overwritten with the actual hostname.

  • In case this needs to be prevented, add "osd crush update on start = false" in /etc/ceph/ceph.conf, so that the OSD process won't change the custom settings in the CRUSH map.

SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.