The gunicorn processes are always showing high memory utilization after upgrading to Satellite 6.10 and above

Environment

• Red Hat Satellite 6.10 and above

Issue

• After upgrading the Red Hat Satellite server to 6.10.z or 6.11.z, from version 6.10.2 or below, the pulpcore-related gunicorn processes are always consuming much higher system resources.

• After upgrading the Red Hat Satellite server to 6.13 from version 6.12 or below, the pulpcore-related gunicorn processes are consuming much more memory and frequently trigger Out Of Memory (OOM).

• Alternatively, pulpcore-worker processes are affected the same way (when publishing Content Views with filters or doing an incremental update of a CV)

• The memory utilization remains at peak while performing any content-management activities and would not be released even after the activities are completed.

• This over-utilization of system memory often results in the server performing poorly or causing Out Of Memory (OOM) kills of other processes.

Resolution

This issue has been reported via This content is not included.Bugzilla 2211957 and This content is not included.Bugzilla 2231496. Simply upgrade the Satellite to the latest 6.13.z version to fix the issue.

For Satellite 6.13.0 to 6.13.3
If upgrade is not possible at the moment, you can follow the steps below to apply the patch manually.

• Edit /usr/lib/python3.9/site-packages/pulpcore/app/access_policy.py file in the Satellite

       class AccessPolicyFromDB(AccessPolicy):
        """
        An AccessPolicy that loads statements from an `AccessPolicy` model instance.
        """
  
        @staticmethod
        #@lru_cache    <=========================== comment out this line (at around line 15)
        def get_access_policy(view):
  

• Restart all Pulp core services.

  systemctl restart pulpcore* && sleep 30 && hammer ping
  

For Satellite 6.11:
Meanwhile, To improve the behavior ( and even solve the issue ), the following steps can be performed:

• Ensure to have the right set of performance tuning applied on the affected satellite server.

• Clean up the unnecessary rpm-changelogs from the satellite server as mentioned in the This content is not included.Bugzilla 2122344.
  (While running the command, It is recommended to not perform any repository or content-view-related tasks)

  # sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' /usr/bin/pulpcore-manager rpm-trim-changelogs
  # satellite-maintain service restart
  

  NOTE: These steps are auto-executed when upgrading from a Red Hat Satellite 6.11 to 6.12.

• Continue with daily tasks and monitor the overall behavior and system resource utilization by the satellite server.

  • The problem reported in This content is not included.Bugzilla 2121288 can affect the performance of pulp tasks as well.
  • Hence, If the approaches suggested earlier were not good enough to improve the server behavior, consider applying the suggestions from this solution article.

For Satellite 6.10:
If you are unable to upgrade your Satellite to 6.11 at the moment, the following steps can be performed to mitigate the issue:

• Ensure to have the right set of performance tuning applied on the affected satellite server.

• Run the following command to reduce the bulk_load_size to 250 and restart the Satellite.

  # hammer settings set --name "bulk_load_size" --value 250
  # satellite-maintain service restart
  

• Continue with daily tasks and monitor the overall behaviour and system resource utilization by the satellite server.

• You can switch it back to the default(2000) once the Satellite is upgraded to 6.11 and you have performed the steps for Satellite 6.11 above.

Reach out to Red Hat Technical Support in case of any further concerns.

Root Cause

Please visit the This content is not included.Bugzilla 2122344 for a detailed explanation of the primary root cause of gunicorn processes high memory usage.

For high usage of pulpcore-worker processes: when publishing a Content View with filters, or when doing an incremental update of a CV, katello / pulp needs to generate new metadata for the specific subset of content of the repos. That includes API queries to pulp to a (large set of) packages with all details. That is very memory demanding. While trimming packages changelog is a trivial way to restrict the memory usage.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.