Introducing 'default pid limit' within docker to improve reliability of OpenShift Container Platform

Updated

In an effort to constantly improve stability, reliability, and security of OpenShift Container Platform (OCP), Red Hat is happy to announce the introduction of a default PID limit within docker to limit the amount of PIDs being used by a single container.

Background


Red Hat continuously works with the upstream [kubernetes](https://kubernetes.io/) community to improve support of kubernetes with respect to stability, reliability, and security. Within these efforts, we also continue to push the limits of containers as a multi-tenant cloud solution for our customers. Along the way, potential threats such as resource exhaustion Denial of service attacks have been thwarted, but there is still more to do in this regard.

Because of this, Red Hat has been focusing on container separation and has been investing in multiple technologies to improve container security, such as docker and CRI-O. The PID-limit feature was added to Moby 1.11 to help prevent fork bombs, a type of denial-of-service attack wherein a process continually replicates itself to deplete available system resources, slowing down or crashing the system due to resource starvation, but was not added to docker.

Up until RHSA-2019:0487, docker 1.13 did not have a way to implement a default PID limit at the container level, which would allow intentional or unintentional fork bombs from being created and which could impact OCP stability and reliability.

Impact


A fork bomb could cause resource starvation on the OCP Node, where the container is scheduled and runs. Due to resource starvation, the OCP Node might potentially fail to `fork()` or `clone()` system related processes, and thus report a *NotReady* state. In such a scenario, OpenShift Container Platform might re-schedule the pod on another suitable OCP Node which, depending on the fork bomb, could suffer the same issue as the previous OCP Node, and therefore, also report *NotReady*.

Resolution


To implement default PID limitation with docker, update the version of docker to the following Errata:

After RHSA-2019:0487 has been installed, it’s important to understand that each container running will be limited to the default maximum PID value of 4096. It’s therefore important to review the OpenShift Container Platform Cluster prior to installing RHSA-2019:0487 to understand whether applications are in use that are running with more than 4096 processes within a single container.

If there is a need to run applications with more than 4096 processes within one single container, the OpenShift Container Platform Cluster operator will be required to adjust the default maximum PID value to a higher number, as per the instructions below:

  • Edit /etc/sysconfig/docker

  • Append --default-pids-limit=<new value> to OPTIONS= and save the file

      $ cat /etc/sysconfig/docker
  # /etc/sysconfig/docker
  
  # Modify these options if you want to change the way the docker daemon runs
  OPTIONS='--selinux-enabled --log-driver=json-file --signature-verification=false --default-pids-limit=8192'
  [...]
    • In OpenShift Container Platform, the openshift_docker_options can be set to openshift_docker_options='--default-pids-limit=<new value>' to apply the new value on all Node(s)

  • Restart docker using systemctl restart docker

In general, Red Hat recommends to always use the latest version of OpenShift Container Platform and Red Hat Enterprise Linux in order to keep up with the steadily-improving security posture of OpenShift Container Platform as a secure, multi-tenant cloud solution. Red Hat continues to work closely with Kubernetes, docker, OpenShift Container Platform, as well as the Operating System and Kernel to improve the security and performance of the entire stack.

SBR
Product(s)
Components
Article Type