High memory requirements on Satellite or Capsule when sending SCAP reports

Solution Verified - Updated

Environment

  • Red Hat Satellite 6
  • Red Hat Capsule 6

Issue

  • Satellite or Capsule encounters high memory demands or even hits OOM killer activity.

  • Around that time, Clients were sending OpenSCAP reports in bulk to the Satellite/Capsule

Resolution

  • Ideally, deploy the SCAP client using the ansible role (or the puppet module) in Satellite, as it already does add some splay.

  • If that is not possible, try to split the bulk activity of generating+sending SCAP reports into multiple chunks, or span the activity to a longer period of time. Particular options to achieve that:

     - Duplicate Satellite's OpenSCAP Policy, change its schedule and split the systems across both Policies 
 - Move some Hosts to a different Capsule that does not handle OpenSCAP reports

  • If scans are triggered directly from the Satellite Web UI or via Remote Execution (REX) playbooks, do not target all hosts at once. Set a strict concurrency/batch limit on the Remote Execution job so that hosts run scans and submit reports in serialized batches instead of simultaneously.

  • If scans are triggered via local cron jobs on client machines (for example, under /etc/cron.d/foreman_scap_client_cron), verify that a randomized sleep/splay delay is configured before the client executes:

    0 1 * * 1 root /bin/sleep 379; /usr/bin/foreman_scap_client ds 1 2>&1 | logger -t foreman_scap_client

  • For more KB articles/solutions related to Red Hat Satellite 6.x OpenSCAP Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x OpenSCAP Issues

Root Cause

  • Each OpenSCAP client report is being processed by a dedicated new process spawned from smart-proxy. Memory consumption of such process is often 0.5GB+ and takes tens of seconds to complete. Therefore, memory demands of concurrent bulk SCAP report processing can easily require (tens of) gigabytes of memory.

Diagnostic Steps

  • OOM killer killing process either (cryptic) diagnostic_con*or directly smart-proxy:

    Dec  6 01:13:45 nz11rsat001v kernel: Out of memory: Killed process 1003621 (diagnostic_con*) total-vm:5291728kB, anon-rss:3496688kB, file-rss:128kB, shmem-rss:0kB, UID:993 pgtables:9452kB oom_score_adj:0

  • Inspect the system log (/var/log/messages or journalctl -xe) around the time of the system hang for kernel OOM-killer invocations involvingdiagnostic_con*, smart-proxy, postgresql, or sidekiq processes:

    Aug 18 12:08:42 ttgssdc0vrhyr03 kernel: [  pid  ]   uid  tgid total_vm rss pgtables_bytes swapents oom_score_adj name
  Aug 18 12:08:42 ttgssdc0vrhyr03 kernel: [ 279480]   496 279480   191178 89323  1007616     3245             0 diagnostic_con*
  Aug 18 12:08:42 ttgssdc0vrhyr03 kernel: [ 279491]   496 279491   175961 64532   811008     3245             0 diagnostic_con*

  • /var/log/foreman-proxy/proxy.log has tens to hundreds of below requests within a short time:

    2025-12-13T01:05:01 7ce63ffd [I] Started POST /compliance/arf/2 
2025-12-13T01:05:01 adb37c5c [I] Started POST /compliance/arf/2 
2025-12-13T01:05:04 50f0d47e [I] Started POST /compliance/arf/2 
2025-12-13T01:05:08 5d45232d [I] Started POST /compliance/arf/2 
2025-12-13T01:05:09 eff6c219 [I] Started POST /compliance/arf/1 
2025-12-13T01:05:11 c68d16df [I] Started POST /compliance/arf/1 
..

  • ps output (caught at the "busy times") shows tens to hundreds of smart-proxy processes (usually there is just one):

    foreman+ 1172374  0.0  1.9 875096 636568 ?       -    Dec09   3:39 /usr/bin/ruby /usr/share/foreman-proxy/bin/smart-proxy
foreman+ 1399229  3.6 10.4 3771708 3412372 ?     -    01:06   0:08 /usr/bin/ruby /usr/share/foreman-proxy/bin/smart-proxy
foreman+ 1399275  4.1  8.5 3004536 2784488 ?     -    01:07   0:09 /usr/bin/ruby /usr/share/foreman-proxy/bin/smart-proxy
foreman+ 1399320  2.7  6.6 2590872 2177096 ?     -    01:07   0:06 /usr/bin/ruby /usr/share/foreman-proxy/bin/smart-proxy
..

  • Inspect /var/log/foreman/production.log on the Satellite server for a cluster of concurrent POST requests to the compliance report endpoint at the exact time of the memory peak or hang:

    2026-08-18T12:08:49 [I|app] Started POST "/api/v2/compliance/arf_reports/..."   
2026-08-18T12:08:50 [I|app] Started POST "/api/v2/compliance/arf_reports/..."

  • pstree shows the processes were spawned from the main smart-proxy and they sometimes call bunzip:

             |-smart-proxy(1172374)-+-diagnostic_con*(1399229)
         |                      |-diagnostic_con*(1399275)
         |                      |-diagnostic_con*(1399320)
         |                      |-diagnostic_con*(1399340)
         |                      |-diagnostic_con*(1399447)-+-bunzip2(1399448)
         |                      |                          |-{diagnostic_con*}(1399449)
         |                      |                          |-{diagnostic_con*}(1399450)
         |                      |                          `-{diagnostic_con*}(1399451)
         |                      |-diagnostic_con*(1400388)
         |                      |-diagnostic_con*(1400486)
         |                      |-diagnostic_con*(1400860)-+-bunzip2(1400861)
         |                      |                          |-{diagnostic_con*}(1400863)
         |                      |                          |-{diagnostic_con*}(1400864)
         |                      |                          `-{diagnostic_con*}(1400865)
..
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.