Idled services cannot wake up automatically after restart of ovnkube-master in Red Hat OpenShift Container Platform 4

Environment

Red Hat OpenShift Container Platform 4.6
Red Hat OpenShift Container Platform 4.7

Issue

Idled services cannot be woken up automatically after the leader ovnkube-master process is restarted or after the ovnkube-master leader changes.

For example, this affects services which are idled during a cluster upgrade. When such a service receives network traffic, it is not woken up automatically.

Resolution

This is a known, long-standing issue in all versions of Red Hat OpenShift Platform 4 with the OVN Kubernetes network plugin prior to version 4.8.

The issue is fixed starting with version 4.8 and beyond.

Workaround

Manually scale the backend deployment or deployment config to unidle the application, for example:

oc scale --replicas=1 dc <dc_name>

Root Cause

When an idle service is woken up, the ovnkube master process will look at a specific cache that contains active service VIPs. If an idle service is part of the cache, it can be woken up.

When the ovnkube master daemon is restarted, this cache starts out empty and must be populated anew.
Unfortunately, earlier versions of OpenShift Container Platform 4 could only add services with active endpoints to aforementioned cache.

Due to the fact that an idled service has no active endpoints, the cache is built incompletely. When an affected service is to be woken up automatically, the ovnkube master process finds no hit in the cache and thus cannot wake up the service.

For a thorough discussion of the root cause, see This content is not included.Bugzilla 2041307.

Due to significant changes in the code base, there is a very high complexity to backport the fix into Red Hat OpenShift Container Platform 4.6 and 4.7. Due to the limited scope of affected components and customers, the decision was made not to backport the changes.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.