How to check the size of largest secrets and configmaps on OpenShift?

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • etcd

Issue

  • How to check the size of the secrets and configmaps in the etcd database?
  • How to ensure secrets and configmaps are not reaching or over to the object size limit?

Resolution

NOTE: the following commands use oc exec with an etcd pod to get the results. When it is not possible to use oc commands but an etcd pod is running, it is possible to use crictl exec -ti $(crictl ps --label "io.kubernetes.container.name=etcdctl" -q) within a control plane node instead, adding the commands from the sh -c (included) till the end of the command.

  • Displaying the top 20 largest secrets and configmaps stored in the etcd:

    $ export ETCD_POD_NAME=$(oc get pods -n openshift-etcd -l app=etcd --field-selector="status.phase==Running" -o name)
$ oc exec -n openshift-etcd -c etcdctl ${ETCD_POD_NAME} -- sh -c 'etcdctl get / --prefix --keys-only  | grep -oE -e "^/[a-z|.]+/secrets/[-a-z|.0-9]*/[-a-z|.0-9]*" -e "^/[a-z|.]+/configmaps/[-a-z|.0-9]*/[-a-z|.0-9]*" | sort -u | while read KEY; do printf "$KEY\t" && etcdctl get ${KEY##* } --prefix --print-value-only | wc -c | numfmt --to=iec ; done | sort -k2hr' | head -20 | awk -F'/' 'BEGIN{print "Type Namespace Name Size\n---- --------- ---- ----"}{print $3" "$4" "$5}'| column -t

Please note that this command can take some time to run as it will depend of the number of objects are defined in the cluster.
The solution how to list the number of objects and size in etcd on OpenShift? provides the number and global size of the objects stored in the etcd database.

  • Displaying the top 20 largest secrets and configmaps for a specific namespace:
    In order to reduce the time it is possible to target a specific Namespace (such as openshift-config) by replacing the regex rule [-a-z|.0-9]*/[-a-z|.0-9]* by the namespace within both grep expression -e, such as:

    $ export ETCD_POD_NAME=$(oc get pods -n openshift-etcd -l app=etcd --field-selector="status.phase==Running" -o name)
$ oc exec -n openshift-etcd -c etcdctl ${ETCD_POD_NAME} -- sh -c 'etcdctl get / --prefix --keys-only  | grep -oE -e "^/[a-z|.]+/secrets/openshift-config/[-a-z|.0-9]*" -e "^/[a-z|.]+/configmaps/openshift-config/[-a-z|.0-9]*" | sort -u | while read KEY; do printf "$KEY\t" && etcdctl get ${KEY##* } --prefix --print-value-only | wc -c | numfmt --to=iec ; done | sort -k2hr' | head -20 | awk -F'/' 'BEGIN{print "Type Namespace Name Size\n---- --------- ---- ----"}{print $3" "$4" "$5}'| column -t

*Note:

  • To display ALL secrets/configmaps, simply remove the |head -20 from the command.
  • To display only the secrets or configmaps, simply remove the relevant -e expression from the grep command.*

Root Cause

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

There is a 1.5MiB object size limit described in the etcd documentation Content from etcd.io is not included.request size limit, and that is currently the default in Openshift. If an object is bigger than that it may become corrupted.

In addition to the above, there is an additional restriction in Kubernetes for the secrets and the configmaps. Size of individual secrets and configmaps is limited to 1 MiB as per the Kubernetes documentation:

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.