openssl enc fails in FIPS mode

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7

Issue

  • Using the openssl enc command to encrypt or decrypt data fails on systems where FIPS is enabled. Example of running it on a normal RHEL machine:

    [user]$ sysctl crypto.fips_enabled
crypto.fips_enabled = 0
[user]$ openssl aes-256-cbc -k PASS </etc/redhat-release | openssl aes-256-cbc -d -k PASS
Red Hat Enterprise Linux Workstation release 6.3 (Santiago)

    Here's what happens on a box where the kernel is in FIPS-enforcing mode:

    [user]$ sysctl crypto.fips_enabled
crypto.fips_enabled = 1
[user]$ openssl aes-256-cbc -k PASS </etc/redhat-release | openssl aes-256-cbc -d -k PASS
10283:error:06080090:digital envelope routines:EVP_DigestInit_ex:disabled for fips:digest.c:292:
bad decrypt
10284:error:06080090:digital envelope routines:EVP_DigestInit_ex:disabled for fips:digest.c:292:
10284:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:325:
...

  • Similar results as above are seen using any other FIPS-approved ciphers with openssl enc (e.g., aes-128-cbc)

Resolution

  • Use the -md <DIGEST> option to specify a message-digest algorithm like sha256 or sha5121
    Examples:

    • Encrypting with openssl enc on a FIPS system

      [user]$ openssl aes-256-cbc -md sha256 -k PASS < /etc/redhat-release >/tmp/data

    • Decrypting with openssl enc on a FIPS system

      [user]$ openssl aes-256-cbc -md sha256 -k PASS -d < /tmp/data
  Red Hat Enterprise Linux Workstation release 7.0 (Maipo)

  • As the example shows, the same message-digest (specified by -md) must be used for encryption AND decryption

  • Data encrypted on a non-FIPS system by openssl enc without explicitly specifying -md will be impossible to decrypt on a FIPS system

1

Run openssl dgst -h on a system in FIPS mode to see full list of available choices

Root Cause

  • openssl enc uses a message-digest algorithm (specified by the -md option) to create the symmetric encryption/decryption key from user-supplied key material1

  • openssl enc defaults to using Content from en.wikipedia.org is not included.MD5 if the -md option is not specified (in all versions of OpenSSL as shipped in RHEL5, RHEL6, and RHEL7; however, note that RHEL8 defaults to using Content from en.wikipedia.org is not included.SHA-256)

  • Use of MD5 is disallowed in any FIPS-compliant system, thus the Linux kernel (and OpenSSL) will not make it available when a system is running in FIPS mode

1

While the -k option is used in the Issue and Resolution of this article for simplicity, the "PASS PHRASE ARGUMENTS" section of the openssl(1) man page details more common/modern ways to provide key material (e.g., -pass pass:XXX or -pass stdin or -pass file:/PATH or -pass fd:number)

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.