How to check the storage used by images in /var/lib/containers/storage/

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4

Issue

• The disk usage for /var/lib/containers/storage/ shown by du looks wrong.
• How to check pod's disk consumption for container images

Resolution

• Because how overlayfs works, du could show a wrong size, even larger than the filesystem size.
• Use podman system df to check the image size instead. Note that podman and crio have different views of running containers, so podman system df would not show correct container disk usage, only the correct image disk usage.

$ oc debug node/[node_name]
[...]
sh-4.4# chroot /host bash
# podman system df
TYPE            TOTAL   ACTIVE   SIZE     RECLAIMABLE
Images          35      0        12.5GB   12.5GB (100%)
Containers      0       0        0B       0B (0%)
Local Volumes   0       0        0B       0B (0%)

• It's also possible to use the -v flag to view the exact size of each image.

# podman system df -v

• Note that when disk pressure is happening, the node kubelet will automatically try to free up storage resources using garbage collection.

• To investigate ephemeral storage used by containers, see When a node has DiskPressure events, how to find which Pod is using the most local disk space ?

Root Cause

For how overlayfs works, some files/directories can be counted several times by du, so it's not recommended to use du to check the size of /var/lib/containers/storage/.

Diagnostic Steps

Access to a node:

$ oc debug node/[node_name]
[...]
sh-4.4# chroot /host bash

Check the disk size with lsblk:

[root@node_name /]# lsblk
NAME                         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
[...]
`-vda4                       252:4    0 39.5G  0 part 
  `-coreos-luks-root-nocrypt 253:0    0 39.5G  0 dm   /sysroot

Check the filesystem usage with df:

[root@node_name /]# df -h /var/lib/containers/storage/overlay/
Filesystem                            Size  Used Avail Use% Mounted on
/dev/mapper/coreos-luks-root-nocrypt   40G   13G   28G  31% /var/lib/containers/storage/overlay

Check the size shown by du (could be even larger than the size of the filesystem, as in this case):

[root@node_name /]# du -sh /var/lib/containers/storage/overlay/
41G    /var/lib/containers/storage/overlay/

Check the size using podman system df:

[root@node_name /]# podman system df
TYPE            TOTAL   ACTIVE   SIZE     RECLAIMABLE
Images          35      0        12.5GB   12.5GB (100%)
Containers      0       0        0B       0B (0%)
Local Volumes   0       0        0B       0B (0%)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.