How to add Include statement in the /etc/ssh/sshd_config file in RHOCP4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.12 (preparing to upgrade to 4.13+)
    • 4.13
    • 4.14

Issue

  • How to add Include statement in the /etc/ssh/sshd_config file in RHOCP4?

  • After upgrading from RHOCP 4.12 to 4.13 it was not possible to ssh into the nodes so to fix this issue impose adding the Include /etc/ssh/sshd_config.d/*.conf statement in /etc/ssh/sshd_config file.

  • Accessing a node using SSH failing with the follwoing message after openshift upgrade:

    $ ssh -i sshkey core@10.0.0.1
core@10.0.0.1: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

Resolution

  • Starting with RHOCP 4.13, due to the changes in the OpenSSH configuration directory, it is recommended to apply any customization in a separate file within /etc/ssh/sshd_config.d/ directory, instead of directly customizing the /etc/ssh/sshd_config file, as shown in the default file:

    # To modify the system-wide sshd configuration, create a  *.conf  file under
#  /etc/ssh/sshd_config.d/  which will be automatically included below
Include /etc/ssh/sshd_config.d/*.conf

  • Check login using ssh not working after RHOCP 4.13 upgrade for more information.

Workaround


If still wanted to customize `OpenSSH` with the `/etc/ssh/sshd_config` file, find the below steps to create a `machineconfig` to append `Include /etc/ssh/sshd_config.d/*.conf` statement to `/etc/ssh/sshd_config`:

  1. Take backup of /etc/ssh/sshd_config file from one of the master node and from worker node.

    $ oc debug node/<worker-node-name> -- cat /host/etc/ssh/sshd_config  > sshd_config_worker.bkp
$ oc debug node/<master-node-name> -- cat /host/etc/ssh/sshd_config  > sshd_config_master.bkp

  2. Add the line Include /etc/ssh/sshd_config.d/*.conf in the start of the both the file and encode it with base64.

        $ cat sshd_config_worker.bkp | base64 -w 0

    W3N5c3RlbV9dfkZWZhdWx0X3NlY3RdCk1pblByb3RvY29sID0gVExTdjEuMgpDaXBoZXJTdHJpbmcgPSBERUZBVUxUQFNFQ0xFVkVMPTEK    
    #sample output

    $ cat sshd_config_master.bkp | base64 -w 0

    T3N5c3RlbV9dfkZWZhdWx0X3NlY3RdCk1pblasdfkjalHSHFLAKHDFdjEuMgpDaXBoZXJTdHJpbmcgPSBERUZBVUxUQFNFQ0xFVkVMPTEH    
    #sample output

  3. Create the MachineConfig object yaml file, replacing the base64 string with the above base64 encoding. Create MachineConfig files for each type of machine present in the cluster:

    This example is for worker node, create a machine-config-sshd_config-worker.yaml file.

    $ cat << EOF > ./machine-config-sshd_config-worker.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: <pool> #replace pool with worker or master
  name: 100-worker-sshd-config      #replace the name of pool
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,T3N5c3RlbV9dfkZWZhdWx0X3NlY3RdCk1pblasdfkjalHSHFLAKHDFdjEuMgpDaXBoZXJTdHJpbmcgPSBERUZBVUxUQFNFQ0xFVkVMPTEH  #replace the base64 encoding with respective master and worker configuration encoding.
        filesystem: root
        mode: 0600
        overwrite: true
        path: /etc/ssh/sshd_config
EOF

  4. Create same for master nodes.

  5. Use oc apply command to apply changes.

    $ oc apply -f ./machine-config-sshd_config-worker.yaml
$ oc apply -f ./machine-config-sshd_config-master.yaml

Root Cause

In RHOCP 4.13 the location for ssh keys changed, as reported by the This page is not included, but the link has been rewritten to point to the nearest parent document.release notes. By default the following sshd configuration is present for retrieving users keys starting with that version:

AuthorizedKeysCommand /usr/libexec/ssh-key-dir %u

Diagnostic Steps

Access the node using oc debug and confirm the line Include /etc/ssh/sshd_config.d/*.conf is not present in the sshd configuration:

$ oc debug node/worker-0 -- chroot /host grep Include /etc/ssh/sshd_config 2> /dev/null
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.