Modification of log rotation of CRI-O in Openshift 4

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4

Issue

• How to set up log rotation of pods under CRI-O.
• The kubelet is in charge of doing the log rotation, which is actually overriding whatever value the crio.conf file has.
• How to specify the maximum size allowed for container log files?
• How to change the default unlimited container log file size?
• How to set log_size_max for CRI-O in OpenShift 4?

Resolution

NOTE: Increasing the value of containerLogMaxSize means that more node resources are needed at the filesystem level as using more disk, and more cpu, memory and disk I/O as the logs are rotated and compressed, then, it's needed to take into consideration this before setting a value

There are two machineconfigs for the kubelet, one for masters and one for workers(rest of nodes):

$ oc get machineconfig | grep -i kubelet
01-master-kubelet                                           d5599de7a6b86ec385e0f9c849e93977fcb4eeb8   2.2.0             22h
01-worker-kubelet                                           d5599de7a6b86ec385e0f9c849e93977fcb4eeb8   2.2.0             22h

It is possible to set up a custom kubelet for the worker nodes and the same operation would be needed for master nodes.
First, we label the worker machineconfigpool with custom-kubelet custom tag of logrotation:

$ oc label machineconfigpool worker custom-kubelet=logrotation
machineconfigpool.machineconfiguration.openshift.io/worker labeled

Then create the CR. Ensure that matchLabels is the same as custom-kubelet value and under kubeletConfig, modify according to the needs related to log rotation:

$ touch logrotation.yaml
$ vi logrotation.yaml
$ cat logrotation.yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: cr-logrotation
spec:
  machineConfigPoolSelector:
    matchLabels:
      custom-kubelet: logrotation
  kubeletConfig:
    containerLogMaxFiles: 10
    containerLogMaxSize: 100Mi
$ oc create -f logrotation.yaml
kubeletconfig.machineconfiguration.openshift.io/cr-logrotation created

Complete the following verification steps:

$ oc get kubeletconfig
NAME             AGE
cr-logrotation   6s

$ oc get kubeletconfig -o yaml
apiVersion: v1
items:
- apiVersion: machineconfiguration.openshift.io/v1
  kind: KubeletConfig
  metadata:
    creationTimestamp: "2020-03-24T11:46:11Z"
    finalizers:
    - 99-worker-75c6f8de-867f-471e-8a09-09d05ee48e0d-kubelet
    generation: 1
    name: cr-logrotation
    resourceVersion: "373981"
    selfLink: /apis/machineconfiguration.openshift.io/v1/kubeletconfigs/cr-logrotation
    uid: e0dc2521-25f5-4982-8592-3ec83a6139c9
  spec:
    kubeletConfig:
      containerLogMaxSize: 100Mi              ««««««««««««««««««««««««««« MODIFIED.
      containerLogMaxFiles: 10                ««««««««««««««««««««««««««« MODIFIED.
    machineConfigPoolSelector:
      matchLabels:
        custom-kubelet: logrotation           ««««««««««««««««««««««««««« MODIFIED.
  status:
    conditions:
    - lastTransitionTime: "2020-03-24T11:46:11Z"
      message: Success
      status: "True"
      type: Success
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Now there will be 3 machineconfigs:

$ oc get machineconfig | grep -i kubelet
01-master-kubelet                                           d5599de7a6b86ec385e0f9c849e93977fcb4eeb8   2.2.0             22h
01-worker-kubelet                                           d5599de7a6b86ec385e0f9c849e93977fcb4eeb8   2.2.0             22h
99-worker-75c6f8de-867f-471e-8a09-09d05ee48e0d-kubelet      d5599de7a6b86ec385e0f9c849e93977fcb4eeb8   2.2.0             74s

The Machine Config Operator (MCO) will take into consideration this new machineconfig generating new machineConfigPools` and applying them to the nodes restarting them in the same way that when upgrading the cluster.

Once the new machineConfig is applied by the MCO, to verify it, start a debug pod into one of the workers:

$ oc debug node/xxx
Starting pod/xxx-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.179.52
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# cat /etc/kubernetes/kubelet.conf 
containerLogMaxFiles: 10
containerLogMaxSize: 100Mi

Root Cause

The default settings used in OpenShift 4 are as:

• containerLogMaxSize: the rotation is being done by default at ~ 50M
• containerLogMaxFiles: maintaining 5 files per container
• being 250M as maximum for container on the disk.

The container log file size and number are managed by the kubelet on each node which then passes the setting on to CRI-O.

The current kubelet configuration used by a node is located at /etc/kubernetes/kubelet.conf. The default parameters are overwritten from CRI-O:

  containerLogMaxSize: 50Mi
  containerLogMaxFiles: 5

Pod/container logs are already rotated from OCP 4.3 automatically by default. The kubelet.conf file is managed by the machine-config operator so additional steps are necessary.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.