How to edit log level verbosity of Kubelet in OpenShift 4

Solution Verified - Updated 8 Jan 2025

Environment

Red Hat Openshift Container Platform (RHOCP)
- 4
Kubelet

Issue

In order to troubleshoot some issues with the nodes, it is recommended to establish verbosity of log level around kubelet depending on the issue to be tracked.
It is needed to understand log level verbosity of Kubelet.

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Starting with OpenShift 4.7, it is now possible to configure the log level for kubelet via the KubeletConfig resource setting the spec.logLevel value. Refer to creating a KubeletConfig CRD to edit kubelet parameters, and configure the spec.logLevel as desired. Refer also to the Content from github.com is not included.upstream documentation for additional example:

$ oc get kubeletconfig [kubeletconfig_name] -o yaml

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: [kubeletconfig_name]
[...]
spec:
  logLevel: 5
[...]

Note: increasing the logLevel uses more IO, CPU, and resources on all the machines within the pool, so it is recommended to back to the default configuration after the required logs are collected.

Kubelet Log level values

As per general use case, it is recommended to apply`0-4`as debug-level logs, and `5-8` as trace-level logs. By default, it is set as `2`.

Verbosity	Description
`--v=0`	Generally useful so it is ALWAYS visible to an operator.
`--v=1`	A reasonable default log level if you don't want verbosity.
`--v=2`	Useful steady state information about the service and important log messages that may correlate to significant changes in the system. This is the recommended default log level.
`--v=3`	Extended information about changes.
`--v=4`	Debug level verbosity.
`--v=6`	Display requested resources.
`--v=7`	Display HTTP request headers.
`--v=8`	Display HTTP request contents.

One time scenario: temporarily change the `kubelet` log level on a single node

NOTE: after collecting the logs, it is needed to change back the kubelet log level to the former value, to avoid issues with the machine-config.

In order to modify the kubelet as a one-time-scenario without rebooting the node due to the change of machine-config (spec":{"paused":false}}), allowing us to modify it without affecting the service, it is needed to connect to the node in debug mode as per below example:

$ oc debug node/<node>
$ chroot /host

Once it is ensured the access, it is needed to check the actual content:

$ systemctl cat kubelet
# /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Wants=rpc-statd.service crio.service
After=crio.service

[Service]
Type=notify
ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests
ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state
EnvironmentFile=/etc/os-release
EnvironmentFile=-/etc/kubernetes/kubelet-workaround
EnvironmentFile=-/etc/kubernetes/kubelet-env

ExecStart=/usr/bin/hyperkube \
kubelet \
  --config=/etc/kubernetes/kubelet.conf \
  --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
  --rotate-certificates \
  --kubeconfig=/var/lib/kubelet/kubeconfig \
  --container-runtime=remote \
  --container-runtime-endpoint=/var/run/crio/crio.sock \
  --allow-privileged \
  --node-labels=node-role.kubernetes.io/master,node.openshift.io/os_id=${ID} \
  --minimum-container-ttl-duration=6m0s \
  --client-ca-file=/etc/kubernetes/ca.crt \
  --cloud-provider= \
  --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec \
  \
  --anonymous-auth=false \
  --v=1 \

Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
# /etc/systemd/system/kubelet.service.d/10-default-env.conf

Define the new verbosity required in /etc/systemd/system/kubelet.service unit and restart the service. In the following example, it is changed from v=1 to v=8:

$ sed -i -e 's/--v=1/--v=8/g' /etc/systemd/system/kubelet.service
$ systemctl daemon-reload
$ systemctl restart kubelet

Gather the logs and then, change back the kubelet log level to the former value, otherwise, you might face issues at machine-config as per following:

E0514 12:47:17.998892    2281 daemon.go:1350] content mismatch for file /etc/systemd/system/kubelet.service: [Unit]

Root Cause

Starting with OpenShift 4.7, it is possible to configure the log level for kubelet via the KubeletConfig resource.

Diagnostic Steps

Information about the spec.logLevel for the KubeletConfig resource:

$ oc explain kubeletconfig.spec.logLevel
KIND:     KubeletConfig
VERSION:  machineconfiguration.openshift.io/v1

FIELD:    logLevel <integer>

DESCRIPTION:
     logLevel defines the log level of the Kubelet

Once the log level is configured properly, logs can be gathered as per below examples:

    ##For generic roles:
    $ oc adm node-logs --role master -u kubelet
    $ oc adm node-logs --role worker -u kubelet

    ##For specific nodes:
    $ oc adm node-logs <node-name> -u kubelet

    ##Or inside a node:
    $ journalctl -b -f -u kubelet.service

It is also possible to gather the logs of all nodes directly:

$ for n in $(oc get node --no-headers | awk '{print $1}'); do oc adm node-logs $n | gzip > $n.log.gz; done

SBR

Shift

Product(s)

Red Hat OpenShift Container Platform

Components

kubernetes

Category

Configure

Tags

kubernetes

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.