FIPS mode can't decrypt existing passphrase-protected ssh keys

Environment

• Red Hat Enterprise Linux 5
• Red Hat Enterprise Linux 6
• Red Hat Enterprise Linux 7

Issue

• It seems encrypted ssh private keys can't be decrypted on FIPS systems. Why? What to do about it?

• We generated a passphrase-protected ssh keypair with ssh-keygen and used it successfully
  After we enabled FIPS mode (e.g., by following instructions for RHEL6), our ssh key no longer accepts our passphrase

  [root]# ssh server.example.com
  FIPS mode initialized
  Enter passphrase for key '/root/.ssh/id_rsa':
  Enter passphrase for key '/root/.ssh/id_rsa':
  Enter passphrase for key '/root/.ssh/id_rsa':
  root@server.example.com's password:
  

• SSH keys not working after FIPS mode enabled on server

• How to create ssh keys in a non FIPS compliant systems so that the ssh keys are complaint after the system is upgraded to FIPS compliant mode.

Resolution

Option 1

• On a non-FIPS system:
  Use an openssl pkcs8 command to decrypt the private key and convert it into a FIPS-compatible format, e.g.:

  openssl pkcs8 -topk8 -v2 aes128 -in <INPUT_KEY> -out <NEW_OUTPUT_KEY>
  

  As an example:

  $ sysctl crypto.fips_enabled
  crypto.fips_enabled = 0
  
  $ head -3 id_rsa
  -----BEGIN RSA PRIVATE KEY-----
  Proc-Type: 4,ENCRYPTED
  DEK-Info: DES-EDE3-CBC,BAA8407BB514678E
  
  $ openssl pkcs8 -topk8 -v2 aes128 -in id_rsa -out new-id_rsa
  Enter pass phrase for id_rsa:   <PASSPHRASE OF EXISTING KEY>
  Enter Encryption Password:      <PASSPHRASE FOR NEW KEY>
  Verifying - Enter Encryption Password:
  
  $ head -2 new-id_rsa
  -----BEGIN ENCRYPTED PRIVATE KEY-----
  MIIFHzBJBgkqhkiG9w0BBQ0wPDAbBgkqhkiG9w0BBQwwDgQIuxFRUu9lmhYCAggA
  

  This new private key will work on both FIPS and non-FIPS systems, so it would be best to simply replace the original private key

Option 2

• On a system in FIPS-enforcing mode:
  Use a simple ssh-keygen command to generate a new passphrase-protected key which will automatically be FIPS-compatible (and then of course propagate the public key out to all the necessary systems)

  $ sysctl crypto.fips_enabled
  crypto.fips_enabled = 1
  
  $ ssh-keygen
  Generating public/private rsa key pair.
  Enter file in which to save the key (/home/user/.ssh/id_rsa):
  Created directory '/home/user/.ssh'.
  Enter passphrase (empty for no passphrase):
  Enter same passphrase again:
  Your identification has been saved in /home/user/.ssh/id_rsa.
  Your public key has been saved in /home/user/.ssh/id_rsa.pub.
  The key SHA1 fingerprint is:
  89:00:dc:cb:00:48:2e:67:18:1e:70:04:21:80:6c:7f:f1:17:b6:58 user@example.com
  The key's SHA1 randomart image is:
  +--[ RSA 2048]----+
  |/O+.             |
  |**+...   E       |
  |+o+o..o + o      |
  |.o .oo + +       |
  |    . . S        |
  |                 |
  |                 |
  |                 |
  |                 |
  +-----------------+
  
  $ head -2 .ssh/id_rsa
  -----BEGIN ENCRYPTED PRIVATE KEY-----
  MIIFDjBABgkqhkiG9w0BBQ0wMzAbBgkqhkiG9w0BBQwwDgQI1CmM/gJkXKMCAggA
  

Root Cause

• The ssh-keygen command--as shipped in every modern version of RHEL--generates keys in a format (RFC4716) that necessitates using the MD5 digest algorithm at some point in the process (as part of a transformation performed on the input passphrase)

• On a FIPS-enforcing system:

  • MD5 is completely disabled so virtually all encrypted SSH keys (private keys protected by a passphrase) will not be usable

  • OpenSSL (called by ssh-keygen) automatically generates PKCS8-formatted keys without MD5 (when run on a system in FIPS mode)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.