How to handle an IP address change of a monitor in a Ceph cluster?

Solution Verified - Updated

Environment

  • Red Hat Ceph Storage

Issue

  • In case a monitor node has undergone an IP address change, the Ceph cluster has to be reconfigured to reflect the monitor IP address change. What steps need to be followed?

Resolution

Step 1 - On an existing and still running MON

  • Grab a copy of the monitor map

    # ceph mon getmap -o /tmp/monmap

NOTE: Before proceeding, it may be best to view the monmaps contents with the following command:

# monmaptool --print /tmp/monmap

  • Copy /tmp/monmap to the failed MON server in /tmp/monmap 

    # scp {running_mon_host}:/tmp/monmap {changed_ip_mon_host}:/tmp/monmap

Step 2 - On an existing and still running MON

  • Update the configuration file /etc/ceph/ceph.conf file to reflect the new IP address of the MON node (mon_host parameter in the configuration file).

  • Distribute/Copy the updated /etc/ceph/ceph.conf file to all Ceph cluster nodes (MONs, OSDs and MDSs) and Ceph client nodes.

Step 3 - On the MON server with the updated IP address, execute the following commands:

   # monhostname=usrgw
   # newip=192.168.122.114
   # monmaptool --rm ${monhostname} /tmp/monmap
   # monmaptool --add ${monhostname} ${newip}:6789 /tmp/monmap

Step 4 - The next step is to propagate the modified monmap into all the monitors, and inject the modified monmap into each monitor.

  • Distribute/Copy the updated monmap /tmp/monmap to all the Monitor nodes.

  • Make sure to stop all your monitors. Injection must be done while the daemon is not running.

  • Inject the monmap into all the Monitors

    # ceph-mon -i {mon-id} --inject-monmap /tmp/monmap

  • Restart all the Monitors.

Step 5 - On the MON server with the updated IP address

  • Verify the MON has rejoined the quorum and the cluster is operational

    # ceph daemon /var/run/ceph/ceph-mon.`hostname -s`.asok quorum_status

  • If succesfull, the monitor's JSON entries should list all MON with their IP addresses. 

    "mons": [
            {
                "rank": 0,
                "name": "ceph-node3",
                "addr": "10.74.249.68:6789/0",
                "public_addr": "10.74.249.68:6789/0"
            },
            {
                "rank": 1,
                "name": "ceph-node2",
                "addr": "10.74.249.83:6789/0",
                "public_addr": "10.74.249.83:6789/0"
            },
            {
                "rank": 2,
                "name": "ceph-node1",
                "addr": "10.74.251.200:6789/0",
                "public_addr": "10.74.251.200:6789/0"
            }
        ]

Root Cause

  • A network update has forced the network administrators to assign a new IP address to one of the MON nodes in your Ceph cluster
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.