Red Hat Ceph Storage 5 rootless deployment utilizing ansible ad-hoc commands

Updated 14 Jan 2023

Issue

Restricted and automated infrastructures limit the use of the superuser account to enhance security
Delegate permissions to the minimum necessary set of privileges required

Environment

Red Hat Ceph Storage 5

Prerequisites for the Environment

ansible for managing and configuring the target nodes
distributed user for ssh key-based access
sudo password-less permissions for distributing user
revoking root ssh access in the ssh daemon configuration

Resolution

Changing Red Hat Ceph Storage 5 to a none privileged user for deployment and orchestration of its components can be achieved by deploying ssh based key authentication and password-less sudo access.
Should this issue occurs in the environment, please open a new support case in the Red Hat Customer Portal referring to this article.

Note ! Red Hat does not support and recommend limiting the sudo permissions to a specific set of commands, at this point in time.
Note ! Red Hat strongly recommends to verify the described deployment in test or development environments.

Described scenarios

Creating a none privileged distributed user for cephadm boostrapping a Red Hat Ceph Cluster 5
Update an existing Red Hat Ceph Storage 5

Creating a none privileged distributed user for cephadm

Ensure to choose a unique, not shared user name to avoid conflicts

Limit the necessary command set to a minimum

# ansible -b -i hosts \
     -m user \
     -a 'name=cephorch \
         comment="Cephadm orchestrator" \
         generate_ssh_key=true \
         ssh_key_comment="Cephadm orchestrator" \
         password_lock=true' \
         nodes
         
node1.example.com | CHANGED => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "append": false,
    "changed": false,
    "comment": "Cephadm orchestrator",
    "group": 1001,
    "home": "/home/cephorch",
    "move_home": false,
    "name": "cephorch",
    "shell": "/bin/bash",
    "ssh_fingerprint": "3072 SHA256:T0...[output ommited]... Cephadm orchestrator (RSA)",
    "ssh_key_file": "/home/cephorch/.ssh/id_rsa",
    "ssh_public_key": "ssh-rsa AAA...[output ommited]... Cephadm orchestrator",
    "state": "present",
    "uid": 1001
}
...[output omitted]...
node2.example.com | CHANGED => {
...[output omitted]...
node3.example.com | CHANGED => {
...[output omitted]...

Note that in the output the ssh public key generated by the selected bootstrap node is shown
The value of the ssh public key is required to be distributed to all nodes for ssh-key-based access.

Deploying ssh-key-based access for the none privileged distributed user

If necessary, retrieve the value from the bootstrap node as follows and store it into a file on the management node

# ansible -b -i hosts \
     -a 'cat /home/cephorch/.ssh/id_rsa.pub' \
     bootstrap | tail -1 > \
  cephorch.pub
   
# ansible -b -i hosts \
    -m authorized_key \
    -a 'user=cephorch \
        state=present \
        key={{ lookup("file", "cephorch.pub") }}' \
    nodes

node1.example.com | CHANGED => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": true,
    "comment": null,
    "exclusive": false,
    "follow": false,
    "key": "ssh-rsa AAAAB3...[output ommited]... Cephadm orchestrator",
    "key_options": null,
    "keyfile": "/home/cephorch/.ssh/authorized_keys",
    "manage_dir": true,
    "path": null,
    "state": "present",
    "user": "cephorch",
    "validate_certs": true
}
node2.example.com | CHANGED => {
...[output ommited]...
node3.example.com | CHANGED => {
...[output ommited]...

Verify connectivity to all nodes from the bootstrap node.

Ensure to deploy an ansible.cfg and inventory file on the newly created user.

# ssh ansible@bootstrap
Activate the web console with: systemctl enable --now cockpit.socket
Last login: Sun Dec 19 08:28:41 2021 from 127.0.0.1
$ sudo runuser -u cephorch /bin/bash
[cephorch@bootstrap ~]$ cat <<EOF > ansible.cfg
[defaults]
inventory = ./hosts
remote_user = cephorch
EOF 
[cephorch@bootstrap ~]$ cat <<EOF > inventory
bootstrap:
  hosts:
      node1.example.com:
nodes:
  hosts:
    node1.example.com:
    node2.example.com:
    node3.example.com:
EOF
[cephorch@bootstrap ~]$ ansible -m ping nodes,bootstrap
node1.example.com | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
node2.example.com | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
node3.example.com | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3.6"
    },
    "changed": false,
    "ping": "pong"
}

Bootstrap your Red Hat Ceph Storage 5

Disable, root login on the ssh daemon by setting the PermitRootLogin option to no.

[cephorch@bootstrap ~]$ ansible -b -m lineinfile \
    -a 'path=/etc/ssh/sshd_config \
        regexp="^PermitRootLogin yes" \
        line="PermitRootLogin no" \
        state=present' \
    nodes

Restart the sshd daemon for the changes to take effect.

[cephorch@bootstrap ~]$ ansible -b -m service \
                                -a 'name=sshd state=restarted' nodes

Ensure to update the cephadm-ansible ansible.cfg file with the remote_user option

Alternatively, specify the appropriate command line switch.

[cephorch@bootstrap ~]$ ansible -b -m lineinfile \
    -a 'path=/usr/share/cephadm-ansible/ansible.cfg \
         line="remote_user = cephorch" \
         insertafter="\[defaults\]" \
         state=present' \
    localhost
   
localhost | CHANGED => {
    "backup": "",
    "changed": true,
    "msg": "line added"
}

Run the cephadm-preflight.yml playbook

Prepare the Red Hat Ceph Storage nodes by running the cephadm-preflight.yml playbook.

[cephorch@bootstrap ~]$ cd /usr/share/cephadm-ansible
[cephorch@bootstrap cephadm-ansible]$ ansible-playbook -b \
  -i /home/cephorch/hosts \
  --extra-vars='ceph_origin=' \
  cephadm-preflight.yml

PLAY [all] ******************************************************************************

TASK [enable red hat storage tools repository] ******************************************
Sunday 19 December 2021  09:07:24 +0000 (0:00:00.024)       0:00:00.024 

...[output ommited]...

PLAY RECAP ******************************************************************************
node1.example.com            : ok=2    changed=1    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0   
node2.example.com            : ok=2    changed=1    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0   
node3.example.com            : ok=2    changed=1    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0   

Sunday 19 December 2021  09:09:20 +0000 (0:00:00.891)       0:01:56.104 
=============================================================================== 
install prerequisites packages ----------------------------------------------- 114.04s
ensure chronyd is running ---------------------------------------------------- 0.89s
configure red hat ceph stable community repository --------------------------- 0.20s
remove ceph_stable repositories ---------------------------------------------- 0.15s
configure red hat ceph stable noarch community repository -------------------- 0.15s
configure red hat ceph community repository stable key ----------------------- 0.13s
enable red hat storage tools repository -------------------------------------- 0.13s
install epel-release --------------------------------------------------------- 0.13s
fetch ceph red hat development repository ------------------------------------ 0.13s
configure ceph red hat development repository -------------------------------- 0.12s

Bootstraping Red Hat Ceph Storage 5 cluster with cephadm

Extend bootstraping the Red Hat Ceph Storage cluster command with the three mandatory options to reflect the changes on ssh transport level.
- ssh-private-key
- ssh-public-key
- ssh-user

Content from docs.ceph.com is not included.https://docs.ceph.com/docs/pacific/mgr/telemetry/

Verify that the bootstrapped Red Hat Ceph Storage 5 is up and is ready for orchestration.

[cephorch@bootstrap ~]$ sudo cephadm shell 

[ceph: root@node1 /]# ceph health
HEALTH_WARN OSD count 0 < osd_pool_default_size 3

[ceph: root@vm07 /]# ceph -s
  cluster:
    id:     ec428a82-60ac-11ec-8c29-52540059a157
    health: HEALTH_WARN
            OSD count 0 < osd_pool_default_size 3
 
  services:
    mon: 1 daemons, quorum node1.example.com (age 40m)
    mgr: node1.example.com.vjhvdm(active, since 39m)
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     

[ceph: root@node1 /]# ceph config-key get mgr/cephadm/ssh_user
cephorch

Continue the steps at Extending Red Hat Ceph Storage 5

Update an existing Red Hat Ceph Storage 5

Changing an existing Red Hat Ceph Storage 5 Cluster to an unprivileged user by re-using the ssh key generated by the cephadm bootstrap command.

Retrieve the cluster ssh-key for the none privileged distributed user

retrieve the value from the cluster configuration as follows and store it into a file on the management node

# cephadm shell ceph config-key get \
     mgr/cephadm/ssh_identity_pub | \
     tail -1 > \
  cephorch.pub

distribute the ssh public key to all nodes for the unprivileged user

# ansible -b -i hosts \
    -m authorized_key \
    -a 'user=cephorch \
        state=present \
        key={{ lookup("file", "cephorch.pub") }}' \
    nodes
    
node1.example.com | CHANGED => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": true,
    "comment": null,
    "exclusive": false,
    "follow": false,
    "key": "ssh-rsa AAAAB3...[output ommited]... Cephadm orchestrator",
    "key_options": null,
    "keyfile": "/home/cephorch/.ssh/authorized_keys",
    "manage_dir": true,
    "path": null,
    "state": "present",
    "user": "cephorch",
    "validate_certs": true
}
node2.example.com | CHANGED => {
...[output ommited]...
node3.example.com | CHANGED => {
...[output ommited]...

verify that the nodes are access able prior changing the Red Hat Ceph Storage configuration

   [ceph: root@bootstrap /]# ceph config-key get \
       mgr/cephadm/ssh_identity_key > \
       /root/.ssh/id_rsa
   
   [ceph: root@bootstrap /]# chmod 0600 /root/.ssh/id_rsa
   
   [ceph: root@bootstrap /]# ssh cephorch@node1.example.com "date"
   Mon Dec 20 11:53:08 UTC 2021
   
   [ceph: root@bootstrap /]# ssh cephorch@node2.example.com "date"
   Mon Dec 20 11:53:08 UTC 2021
   
   [ceph: root@bootstrap /]# ssh cephorch@node2.example.com "date"
   Mon Dec 20 11:53:25 UTC 2021

update the ssh user in the Red Hat Ceph Storage 5 configuration accordingly

[ceph: root@bootstrap /]# ceph config-key \
                               set mgr/cephadm/ssh_user cephorch

Disable, root login on the ssh daemon by setting the PermitRootLogin option to no.

[cephorch@bootstrap ~]$ ansible -b -m lineinfile \
    -a 'path=/etc/ssh/sshd_config \
        regexp="^PermitRootLogin yes" \
        line="PermitRootLogin no" \
        state=present' \
    nodes

Restart the sshd daemon for the changes to take effect.

[cephorch@bootstrap ~]$ ansible -b -m service \
                                -a 'name=sshd state=restarted' nodes

Extending Red Hat Ceph Storage 5

After finishing updating or bootstrapping the Red hat Ceph Storage 5 with an unprivileged user, proceed with extending or maintenance tasks of the configuration.

Use your favorite editor to write your Cluster definition (a basic example is shown below)

# Host definitions
service_type: host
addr: 127.0.0.1
hostname: node1.example.com
labels:
- mon
- mgr
- osd
- _admin
---
service_type: host
addr: 127.0.0.2
hostname: node2.example.com
labels:
- mon
- mgr
- osd
---
service_type: host
addr: 127.0.0.3
hostname: node3.example.com
labels:
- mon
- mgr
- osd
---
# Monitor deployment definition
service_type: mon
placement:
  hosts:
    - node1.example.com
    - node2.example.com
    - node3.example.com
---
# Manager deployment definition
service_type: mgr
placement:
  hosts:
    - node1.example.com
    - node2.example.com
    - node3.example.com
---
# OSD deployment definition
service_type: osd
service_id: default_osd_group
placement:
  host_pattern: "*"
data_devices:
  paths:
    - /dev/vdb
    - /dev/vdc
    - /dev/vdd

Apply to definition to the Red Hat Ceph Cluster 5

Verify the Cluster health and the configuration is in place

[ceph: root@node1 /]# ceph orch apply -i rollout.yml 
Added host 'node1.example.com' with addr '127.0.0.1'
Added host 'node1.example.com' with addr '127.0.0.2'
Added host 'node1.example.com' with addr '127.0.0.3'
Scheduled mon update...
Scheduled mgr update...
Scheduled osd.default_osd_group update...

[ceph: root@node1 /]# ceph -s
  cluster:
    id:     ec428a82-60ac-11ec-8c29-52540059a157
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum node1.example.com,node2,node3 (age 5s)
    mgr: node1.example.com.vjhvdm(active, since 61m), standbys: node2.xebxcp, node3.slhagg
    osd: 7 osds: 6 up (since 78s), 7 in (since 1.50128s); 1 remapped pgs
 
  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   30 MiB used, 60 GiB / 60 GiB avail
    pgs:     1 active+clean+remapped

SBR

Ceph

Product(s)

Red Hat Ceph Storage

Category

Customize or extend

Tags

Article Type

General