Defining retention for logs in LokiStack on RHOCP 4

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Red Hat OpenShift Logging (RHOL)
  • 5
  • 6
• LokiStack
• Loki

Issue

• As per enabling stream-based retention with Loki, it is recommended to define retention period under spec.limits.global.retention for global retention and spec.limits.tenants.[tenant_name].retention for tenant based retention (application, infrastructure, audit). How does this prune data from the object storage bucket?
• Is there any other alternative way to set retention for logs storage in object storage bucket?
• We would like assistance deleting old logs stored in Loki

Resolution

If there is no retention set on S3 bucket or within the LokiStack CR, then the objects are never deleted from the backend object storage.

Log retention recommendation

It is recommended to set retention on the object storage provider side for deletion of the objects as it is more efficient in terms of costs: computational and money. The reason is that pruning the objects is a cost free task provided by the storage provider. When it is set in the Loki side:

• The Compactor runs at every compactor.compaction-interval and the objects are marked firstly for deletion and later deleted asynchronously. This creates a scenario of sweeper lag which means there could be logs beyond the mention retention time.
• The API calls for getting the chunks and performing deletions could incur costs in Cloud Environments.

Important note: when setting retention on the object storage provider, the retention specification is not required on the LokiStack definition.

Configuring log retention in Loki

If for any reason it is not desired to configure the log retention in the S3 bucket as recommended, it is possible to configure log retention in Loki.

Set retention by time

The spec.limits.global.retention and spec.limits.tenants.[tenant_name].retention are meant to be used for individual (pod level) or group of workload (namespace level) where stream corresponds to the workload.

For setting "globally" and/or "by tenant" log retention, see "Enabling stream-based retention with Loki" . Here is an example to configure in Loki a global log retention of 7 days:

$ oc edit LokiStack logging-loki -n openshift-logging
--- OUTPUT OMMITED ---
spec:
  limits:
    global: 
      retention:  
        days: 7 ---> Set the value as per requirement.
  managementState: Managed
--- OUTPUT OMMITED ---

Set retention by threshold and/or size

Refer to implement threshold and/or size based retention for Loki in RHOCP 4.

Considerations when using NooBaa as object storage backend

See the article how to configure Lifecycle policy on noobaa bucket.
>Note: there is a known bug in older versions This content is not included.Bug 2279964 - [GSS][ODF 4.12.7] noobaa bucket lifecycle policy does not delete all objects that explains that although object storage global retention is configured in noobaa, not all objects are deleted and causes storage bucket size to grow as objects are not being purged. This is fixed with this errata starting in ODF 4.16. In older versions, the only effective option is to set retention on Loki side.

Root Cause

It is recommended to set retention on the object storage provider side for deletion of the objects as it is more efficient in terms of costs. Bug This content is not included.OBSDOCS-784 was created to get the above information added to the official documentation.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.