Failing ssh access to nodes using RSA key after RHOCP 4 upgrade.
Environment
- Red Hat OpenShift Container Platform (RHOCP)
- 4.13
- 4.14
- Red Hat Enterprise Linux (RHEL)
- 8
Issue
-
SSH to RHCOS nodes is not possible after the upgrade to RHOCP 4.13.
-
oc debug nodeis working but not SSH for accessing a node. -
Accessing a node using the correct SSH key is reporting:
$ ssh -i sshkey core@10.0.0.1 core@10.0.0.1: Permission denied (publickey,gssapi-keyex,gssapi-with-mic). -
On the SSH daemon logs the following error is recorded while the the SSH attempt fails:
userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Resolution
-
Replace the RSA SSH key used for accessing the nodes with a more secure ED25519 key, following the steps in the How to update SSH keys after installation in Openshift 4 ? solution.
-
Be sure to use one of the supported RHEL releases as the client machine.
-
On the client machine set the
DEFAULTcrypto policy:$ sudo update-crypto-policies --set DEFAULT -
Note: there is another issue of a different nature that may cause SSH access to not work after upgrading to 4.13 or 4.14, for reference see Login using ssh not working after RHOCP 4.13 upgrade solution.
-
For freshly installed 4.14, worker/master node show
refusing RSA key: Invalid key lengtherror for SSH connection from bastion node due to 1024 bits RSA key. Verify private key bit size usingssh-keygen -l -f id_rsa. Generate 2048 bit key usingssh-keygen -t rsa -b 2048 -N '' -f <filename>command and update it as per How to update SSH keys after installation in Openshift 4 ? solution.[root@bastion ~]# ssh -i id_rsa core@10.254.30.184 core@10.254.30.184: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).[root@worker-0 ~]# journalctl -xe --no-pager | grep -i rsa worker-0-l94vh sshd[765037]: main: sshd: ssh-rsa algorithm is disabled worker-0-l94vh sshd[765037]: refusing RSA key: Invalid key length [preauth] worker-0-l94vh sshd[765037]: Connection closed by authenticating user core 10.0.0.63 port 43734 [preauth]
Root Cause
- RHOCP 4.13 and 4.14 are based on RHEL 9.2 where the
ssh-rsais removed by default from thePubkeyAcceptedAlgorithms. - If the client machine offers the
ssh-rsaalgorithm the SSH daemon refuses the connection. - From the
crypto-policies-20200713-1.git51d1222.el8package thessh-rsais removed by the default crypto policy.
Diagnostic Steps
-
On the client machine confirm the
LEGACYcrypto policy is in use:$ sudo update-crypto-policies --show LEGACY -
On the client machine confirm the
ssh-rsaalgorithm is the first in the list for thePubkeyAcceptedKeyTypesin the/etc/crypto-policies/back-ends/openssh.configfile:$ cat /etc/crypto-policies/back-ends/openssh.config |grep PubkeyAcceptedKeyTypes PubkeyAcceptedKeyTypes ssh-rsa,[..omitted output..] -
On the RHOCP node confirm the message is reported when an SSH access is attempted:
$ oc adm node-logs worker0.example.com -u sshd.service [..omitted output..] Jan 11 21:43:56.780800 worker0.example.com sshd[4154771]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth] [..omitted output..]
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.