OCP4: CVE-2021-44228 affecting Elasticsearch (Red Hat OpenShift Logging)

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Red Hat OpenShift Logging (RHOL)
  • 5
• Red Hat Openshift Elasticsearch Operator
  • 5

Issue

• Red Hat OpenShift Logging mitigation for CVE-2021-44228 in Red Hat OpenShift Container Platform 4

Resolution

The vulnerability is already fixed in the following OpenShift Logging releases:

• Openshift Logging 5.0.10 as part of errata RHSA-2021:5137.
• Openshift Logging 5.1.5 as part of errata RHSA-2021:5128.
• Openshift Logging 5.2.4 as part of errata RHSA-2021:5127.
• Openshift Logging 5.3.1 as part of errata RHSA-2021:5129.

Workaround

There is a workaround if upgrading is not yet possible.

Important: If you follow the optional step 8, then please ensure that you revert the Openshift Logging and/or the Elasticsearch operator managementState back to managed state to upgrade to a version including the fix.

Note: This workaround takes ElasticSearch for Openshift Logging stack as an example, but the workaround can be applied the same way on Elasticsearch deployed using the official Red Hat ElasticSearch Operator.

1. Change to project where Logging stack deployed (by default openshift-logging project):

$ oc project openshift-logging

2. Find the Elasticsearch replicasets deployed for passing later to oc set env command:

$ oc get rs -l component=elasticsearch
NAME                                      DESIRED   CURRENT   READY   AGE
elasticsearch-cdm-ba9c6evk-1-796f6cfdbc   1         1         1       16d
elasticsearch-cdm-ba9c6evk-2-7959d4d857   1         1         1       16d
elasticsearch-cdm-ba9c6evk-3-5f9c5d668c   1         1         1       17d

3. Set environment variable ES_JAVA_OPTS in the Elasticsearch container for system property log4j2.formatMsgNoLookups to true:

$ oc set env -c elasticsearch replicaset/<elasticsearch_replicaset_name> ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"

3.1. You can confirm this by:

  $ oc set env -c elasticsearch rs -l component=elasticsearch --list | grep ES_JAVA_OPTS

  ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true
  ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true
  ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true

4. Scale down the Elasticsearch ReplicaSet so new ES pod is spawned afterwards:

$ oc scale replicaset/<elasticsearch_replicaset_name> --replicas=0

5. Check new ES pod has been spawned automatically after scaling down ReplicaSet to 0 (there is no need to scale up as that would be done automatically):

  $ oc get pods -l component=elasticsearch

  NAME                                            READY   STATUS    RESTARTS   AGE
  elasticsearch-cdm-ba9c6evk-1-796f6cfdbc-4dqc6   2/2     Running   0          27m
  elasticsearch-cdm-ba9c6evk-2-7959d4d857-z5km9   2/2     Running   0          2d9h
  elasticsearch-cdm-ba9c6evk-3-5f9c5d668c-cr8lj   2/2     Running   0          2d9h

  $ oc  set env -c elasticsearch pods -l component=elasticsearch --list | grep ES_JAVA_OPTS

  ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true
  ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true
  ES_JAVA_OPTS=-Dlog4j2.formatMsgNoLookups=true

6. oc exec into the newly-spawned ES pod to check Java command-line arguments passed correctly including -Dlog4j2.formatMsgNoLookups=true:

$ oc exec -c elasticsearch <elasticsearch_pod> -- grep -a log4j2.formatMsgNoLookups /proc/1/cmdline

-Dlog4j2.formatMsgNoLookups=true should be visible in above output.

7. Repeat steps 3 to 6 for each Elasticsearch ReplicaSet one at a time slowly to allow ES pods respawn while maintaining ES cluster quorum:

8. (Optional) To avoid the OpenShift Logging operator and Elasticsearch operator to rewrite changes while the fix is still being worked on, you can set it to unmanaged:

$ oc edit clusterlogging/instance
[...]
spec:
[...]
    managementState: Unmanaged
[...]
$ oc edit elasticsearch/elasticsearch
[...]
spec:
[...]
    managementState: Unmanaged
[...]

And confirm it by checking:

  $ oc get clusterlogging
  NAME       MANAGEMENT STATE
  instance   Unmanaged

  $ oc get elasticsearch
  NAME            MANAGEMENT STATE   HEALTH   NODES   DATA NODES   SHARD ALLOCATION   INDEX MANAGEMENT
  elasticsearch   Unmanaged          green    3       3            all    

• (Optional) After the changes above the procedure can be done at the deployment level instead of ReplicaSets as explained in steps 2 to 5:

    $ oc patch deployment/<elasticsearch_deployment_name> --type=merge -p '{"spec":{"paused": false}}'
  
    $ oc set env deployment/<elasticsearch_deployment_name> -c elasticsearch ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true"
  

  This will create new ReplicaSets automatically and pods with the environment defined. After that steps 5 to 7 can be executed.
  This method does not need to be done if following steps 2 to 5.

Root Cause

CVE-2021-44228
This content is not included.Bugzilla

Diagnostic Steps

Check that OpenShift Logging operator has not reverted ES_JAVA_OPTS environment variable addition after making changes:

$ oc describe rs/<elasticsearch_replicaset_name> | grep ES_JAVA_OPTS

Expected output:

      ES_JAVA_OPTS:             -Dlog4j2.formatMsgNoLookups=true

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.