Custom repository is not available on the client system even after attaching the right subscription.

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.12 and below.

Issue

  • Even after attaching the custom product subscription, the custom repository is not available on the client system.
  • Entitlement certificate present on the client system is missing the content section.

Resolution

The Red Hat Satellite Engineering team fixed the reported issue via various Bugzillas.

Workaround for Satellite 6.12 and below

Fix just one repository


Get the ID of the repository: 
    # echo "select id,name from katello_root_repositories where name ilike '%Centos%'"|su - postgres -c "psql foreman"

Feed the ID to the following rake (in this case it was 119):

# foreman-rake console << EOF
root = Katello::RootRepository.find(119);
ForemanTasks.async_task(::Actions::Candlepin::Product::ContentAdd, owner: root.product.organization.label, product_id: root.product.cp_id, content_id: root.content_id)
EOF

Fix all repositories

Create file $(rpm -ql rubygem-katello tfm-rubygem-katello | grep /lib/katello/tasks$)/find_broken_custom_repos.rake:

namespace :katello do
  def commit?
    ENV['COMMIT'] == 'true'
  end
  desc "Check candlepin for missing custom repository content. Use COMMIT=true to apply the fix."
  task :correct_candlepin_custom_content => ["environment", "check_ping"] do
    logger = Logger.new(STDOUT)
    repos = ::Katello::Repository.yum_type.in_default_view.select{|r| r.class.respond_to?(:custom) and r.custom? }

    logger.info("Checking #{repos.count} custom repos..")
    products = {}
    repos.each do |repo|
      unless products.key?(repo.product.cp_id)
        products[repo.product.cp_id] = Katello::Resources::Candlepin::Product.get(repo.organization.label, repo.product.cp_id).pluck(:productContent).first.pluck(:content).pluck(:id)
      end
      product = products[repo.product.cp_id]
      unless product.include?(repo.root.content_id)
        logger.warn("MISSING CONTENT: Product: \"#{repo.product.name}\" (cp_id #{repo.product.cp_id}), Repository: \"#{repo.name}\" (cp_id #{repo.root.content_id})")
        if commit?
          puts "Fixing Repository #{repo.name}"
          ForemanTasks.async_task(::Actions::Candlepin::Product::ContentAdd, owner: repo.organization.label, product_id: repo.product.cp_id, content_id: repo.root.content_id)
        else
          logger.info("Re-run the command with COMMIT=true to fix the repository.")
        end
      end
    end
    logger.info("Done")
  end
end

and execute it via:

foreman-rake katello:correct_candlepin_custom_content

If the output contains something like:

I, [2022-12-07T22:19:45.688398 #1651862]  INFO -- : Checking 110 custom repos..
W, [2022-12-07T22:19:46.051515 #1651862]  WARN -- : MISSING CONTENT: Product: "CUSTOM__ZOO_1" (cp_id 201568164235), Repository: "CUSTOM__ZOO_1-7" (cp_id 1670444191888)
I, [2022-12-07T22:19:46.051604 #1651862]  INFO -- : Re-run the command with COMMIT=true to fix the repository.
I, [2022-12-07T22:19:46.324856 #1651862]  INFO -- : Done

then re-run the script with COMMIT=true option after resetting entity version of all custom products (a preventive safe step, see "In case the above command(s) fail"):

su - postgres -c "psql candlepin -c \"UPDATE cp2_products SET entity_version = NULL WHERE locked = 0;\""
foreman-rake katello:correct_candlepin_custom_content COMMIT=true

Then the script will fire one or multiple Content Add task(s) you can see in WebUI.

Note: after a Satellite upgrade, the rake script start to be ignored. You must re-create it in the given location() after each and every Satellite upgrade. () the location is always evaluated to the currently valid directory.

Note: The path specified above includes an asterisk, which is designed to adapt to file paths that include version numbers that could differ from one Satellite server to another. However, these version numbers, and therefore the file paths that reflect them, can change when the commands "satellite-installer" or "satellite-maintain" are run. This is fine, but it could mean that future attempts to run this script might require one to download and save this script again.

In case the above command(s) fail:

A runtime error happens

The triggered task(s) can fail with error like:

Runtime Error could not execute statement at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,433

If so, look for detailed error in /var/log/candlepin/error.log which would be something like:

Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "cp2_product_entity_version"
Detail: Key (product_id, entity_version)=(924474952511, -242729466) already exists.

In this case the solution is to delete the duplicate registry and let the ContentAdd task recreate it. Use the below command to delete the registry:

# echo "delete from cp2_products where product_id = '924474952511' and entity_version = '-242729466'" |su - postgres -c "psql candlepin"

NOTE : The product_id and entity_version would differ in your case.
Then you can try resuming the paused task or simply cleanup the paused ones create a new one (using the ContentAdd rake command(s) above again).

A LoadError is raised:

The command fails with:

rake aborted!
LoadError: cannot load such file -- /usr/share/gems/gems/katello-4.5.0.24/lib/katello/tasks/find_broken_custom_repos.rake

Make sure the file find_broken_custom_repos.rake does have at least 0644 permissions set.

A JobExecutionException is raised:

The command fails with:

org.candlepin.async.JobExecutionException: Unexpected exception occured while executing transactional block

Look for detailed error in /var/log/candlepin/error.log which would be something like:

2022-05-17 15:22:38,962 [thread=Thread-131 (ActiveMQ-client-global-threads)] [job=00565e1d80d1383a0180d1f8657a0099, job_key=ImportJob, org=<organization_name>, csid=XXX-XXX-XXX-XXX-XXX] ERROR org.candlepin.controller.refresher.visitors.ProductNodeVisitor - Entity version collision detected: Product [uuid: null, id: RH00050, name: Red Hat Enterprise Linux for Virtual Datacenters, Standard] != Product [uuid: 00565e1d803903fc01803afa615a00cc, id: RH00050, name: Red Hat Enterprise Linux for Virtual Datacenters, Standard]

or like:

2022-05-17 15:22:38,962 [thread=Thread-131 (ActiveMQ-client-global-threads)] [job=00565e1d80d1383a0180d1f8657a0099, job_key=ImportJob, org=<organization_name>, csid=XXX-XXX-XXX-XXX-XXX] ERROR org.candlepin.controller.refresher.visitors.ProductNodeVisitor - Entity version collision detected: Product [uuid: 8a5701b9825e4bbf0182817a061c27f4, id: 936846185834, name: Extra Packages for Enterprise Linux (EPEL)] != Product [uuid: null, id: 936846185834, name: Extra Packages for Enterprise Linux (EPEL)]

In this case refer solution - or, after doing a preventive backup of Satellite, run directly (use proper uuid per the error log:

su - postgres -c "psql candlepin -c \"SELECT * FROM cp2_products WHERE uuid = '00565e1d803903fc01803afa615a00cc'\""

If that returns one record with the product name, then fix it:

su - postgres -c "psql candlepin -c \"UPDATE cp2_products SET entity_version = NULL WHERE uuid = '00565e1d803903fc01803afa615a00cc'\""

Then, apply the same workaround with ContentAdd again.

404 Not Found for Product is raised

When hitting

RestClient::NotFound: Katello::Resources::Candlepin::Product: 404 Not Found {"displayMessage":"Product with ID \"483023584370\" could not be found.","requestUuid":"2d4ff7fa-6da8-4a3b-a775-8052c0fd64e3"} (GET /candlepin/owners/ORGANIZATION/products/483023584370/?)

then ensure that candlepin does not have such product (customize all below commands by the product ID from above error string):

su - postgres -c "psql candlepin -c \"SELECT * FROM cp2_products WHERE product_id = '483023584370';\""

If nothing is returned, then:

su - postgres -c "psql foreman -c \"SELECT * FROM katello_products WHERE cp_id = '483023584370';\""

tells you what product is missing in candlepin. Let re-create the object there: run foreman-rake console and once it loads:

product = ::Katello::Product.where(:cp_id => '483023584370').first

ForemanTasks.sync_task(::Actions::Candlepin::Product::Create,
                       :owner => product.organization.label,
                       :name => product.name,
                       :id => product.cp_id,
                       :multiplier => 1,
                       :attributes => [{:name => "arch", :value => "ALL"}])

ForemanTasks.sync_task(::Actions::Candlepin::Product::CreateUnlimitedSubscription,
                       :owner_key => product.organization.label,
                       :product_id => product.cp_id,
                       :start_time => nil)

Then, re-run again the rake script to add missing content.

Diagnostic Steps

  • Check the entitlement certificate present on the client system:

      # ls /etc/pki/entitlement/
  # rct cat-cert /etc/pki/entitlement/XXXXXXXXXX.pem

  • The content section from the entitlement certificate of the affected product is missing.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.