The events time-to-live (event-ttl) in OpenShift 4

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Kubernetes events

Issue

• After how many hours or minutes of creation do the events get deleted from the projects?
• What is the ttl for the events in an RHOCP cluster?
• How to change the event-ttl in OCP 4?
• Why old events are not deleted after the event-ttl value?

Resolution

The event-ttl value is hardcoded to 3 hours and it is currently not supported to change the value in OpenShift 4.

A RFE requesting support for the event-ttl to be configurable has been approved (This content is not included.RFE-4559), and it is expected to have it as a Technology Preview feature in an upcoming OpenShift release. As per the implementation of the new feature, the event-ttl will be configurable from 5 minutes up to 180 minutes (3 hours).

Workarounds

Keep events for more than 3 hours

When it is needed to keep events for more than 3 hours, instead of increasing the TTL for the events, it is recommended to use the Event Router to forward the events like other logs to the log store. Refer to collecting and storing Kubernetes events for additional information.
>Note: for configuring the ClusterLogForwarder for the Event Router logs, they will be part of the infrastructure logs.

Delete events manually

When it is needed to delete the events in a cluster (like for example, due to an issue causing a very big amount of events, or if there are frequent etcd leader changes and the events are not being deleted), refer to how to delete all kubernetes.io/events in etcd.

Root Cause

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Whenever an event is generated in an OCP cluster, a lease is set to it with a TTL value associated. The Content from github.com is not included.event-ttl value is hardcoded to 3 hours.

Note: etcd bad performance could cause the events to not being deleted from etcd after the event-ttl value, specially if the etcd leader changes frequently, as the TTL counters are stored locally in the etcd.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.