Satellite 6 upgrade check through satellite-maintain failed on candlepin disk space usage

Environment

• Red Hat Satellite 6.x

Issue

• Upgrading Red Hat Satellite 6 fails on disk space check for candlepin.

• Scenario Checks before upgrading to Satellite 6.x.z failed.

      The following steps ended up in failing state:
      [available-space-cp]
  
      Resolve the failed steps and rerun
      the command. In case the failures are false positives,
      use --whitelist="available-space-cp"
  

Resolution

• Increase the disk space used by candlepin i.e under /var/lib/candlepin and make sure the disk usage is free by more than 90%

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

Root Cause

• candlepin or rather the Artemis-service used by candlepin started blocking requests as soon as the disk-usage rose above 90% (even though this meant that 200GB were still free).

• Added below checks for upgrade and health check with satellite-maintain

      # cat /usr/share/gems/gems/foreman_maintain-0.7.8/definitions/checks/disk/available_space_candlepin.rb 
  
       module Checks
       module Disk
       class AvailableSpaceCandlepin < ForemanMaintain::Check
        metadata do
          label :available_space_cp
          description 'Check to make sure /var/lib/candlepin has enough space'
          tags :pre_upgrade
          confine do
            feature(:candlepin) && check_min_version('candlepin', '3.1')
          end
          end
  
        MAX_USAGE_IN_PERCENT = 90
  
        def run
          assert(enough_space?, "System has more than #{MAX_USAGE_IN_PERCENT}% space used"\
                 " on #{feature(:candlepin).work_dir}.\n"\
                 'See https://bugzilla.redhat.com/show_bug.cgi?id=1898605')
        end
  
        def enough_space?
          io_obj = ForemanMaintain::Utils::Disk::IODevice.new(feature(:candlepin).work_dir)
          io_obj.space_used_in_percent < MAX_USAGE_IN_PERCENT
        end
       end
       end
       end
  

Diagnostic Steps

• To check the disk space used by the candlepin partition.

   # df -T /var/lib/candlepin
  

• To check the candlpin status through curl.

   # curl -s -k https://localhost:8443/candlepin/status | json_reformat
  

• To check specifically candlepin disk space uses before upgrade, use below command.

   # satellite-maintain health check --label available-space-cp
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.