How ephemeral storage works in OCP4

Solution Unverified - Updated

Environment

  • RedHat Openshift OpenShift Platform
    • 4

Issue

Found that with containerLogMaxSize equals 100Mi, and the limits.ephemeral-storage is 150Mi, the pod will be evicted because it exceed the limitation of ephemeral-storage

Resolution

  • Increase the ephemeral-storage limit to cover the block size
  • Reduce the containerLogMaxSize to trigger log rotation more earilier

Root Cause

According to What is Ephemeral storage, ephemeral storage counts the size of blocks allocated to the file, not the actual size of the log file, so the issue happen when the blocks allocated to the file already exceed the limit of ephemeral-storage, while the actual file size is still under the log rotation limit, the pod will be evicted but the log file not being rotated

Because the improvement of ext4, it uses a "multiblock allocator" (mballoc) which allocates many blocks in a single call, instead of a single block per call, avoiding a lot of overhead. This improves the performance, and it's particularly useful with delayed allocation and extents. Refer to Content from ext4.wiki.kernel.org is not included.Ext4 How-to

So when file grows to around 32Mi, the block size will be 32Mi, however when the file grows over 32Mi, the file system will allocate 64Mi blocks to the file while the file is only 33 or 34Mi which is far lower than the default 50Mi trigger of log rotation

[root@ip-10-0-221-187 /]# while true; do ls -lsh /var/log/pods/log_logworker-logger-755f5c8c95-659q8_b087dffb-4c24-405f-83f6-1eef9fd68165/logworker-logger/0.log; sleep 2; done
192K -rw-------. 1 root root 177K Nov 28 01:04 0.log
960K -rw-------. 1 root root 537K Nov 28 01:05 0.log
960K -rw-------. 1 root root 899K Nov 28 01:05 0.log
2.0M -rw-------. 1 root root 1.3M Nov 28 01:05 0.log
...
2.0M -rw-------. 1 root root 2.0M Nov 28 01:05 0.log
4.0M -rw-------. 1 root root 2.3M Nov 28 01:05 0.log
...
4.0M -rw-------. 1 root root 3.7M Nov 28 01:05 0.log
8.0M -rw-------. 1 root root 4.1M Nov 28 01:05 0.log
...
8.0M -rw-------. 1 root root 7.9M Nov 28 01:05 0.log
16M -rw-------. 1 root root 8.3M Nov 28 01:05 0.log
...
16M -rw-------. 1 root root 16M Nov 28 01:06 0.log
32M -rw-------. 1 root root 17M Nov 28 01:06 0.log
...
32M -rw-------. 1 root root 32M Nov 28 01:08 0.log
64M -rw-------. 1 root root 33M Nov 28 01:08 0.log
...
64M -rw-------. 1 root root 49M Nov 28 01:09 0.log

How to set log rotation in OCP4
How to limit ephemeral storage

SBR
Product(s)
Components

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.