"RestClient::Gone: Katello::Resources::Candlepin::Consumer: 410 Gone" errors in Red Hat Satellite 6 logs?

Solution Unverified - Updated

Environment

Red Hat Satellite 6

Issue

An error similar to the following appears in the Red Hat Satellite's /var/log/foreman/production.log along with a significant backtrace.

"RestClient::Gone: Katello::Resources::Candlepin::Consumer: 410 Gone"

Resolution

Track down source of the call external to the Red Hat Satellite (see Diagnostic Steps) and remedy whatever service is making the call to the Satellite.

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

Root Cause

There are numerous situations that could lead to these errors. A number have in common that a request is made of a specific Candlepin consumer ID that no longer exists on the Satellite.

Diagnostic Steps

Tracking down the source of the error

  • A page-long trace beginning with a 410 Gone error in the Satellite's /var/log/foreman/production.log. In the following example, the UUID of the specific Content Host is 1c38a940-d849-466e-91ee-169561409fc4
2018-12-14T14:43:47 [I|app|d3886] Processing by Katello::Api::Rhsm::CandlepinProxiesController#serials as JSON
2018-12-14T14:43:47 [I|app|d3886]   Parameters: {"id"=>"1c38a940-d849-466e-91ee-169561409fc4"}
2018-12-14T14:43:47 [I|app|d3886] Current user: foreman_admin (administrator)
2018-12-14T14:43:47 [E|app|d3886] RestClient::Gone: Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit 1c38a940-d849-466e-91ee-169561409fc4 has been deleted","requestUuid":"a537b59e-7ee7-4251-947f-27c964b73909","deletedId":"1c38a940-d849-466e-91ee-169561409fc4"} (GET /candlepin/consumers/1c38a940-d849-466e-91ee-169561409fc4)
Body: {"displayMessage":"Unit 1c38a940-d849-466e-91ee-169561409fc4 has been deleted","requestUuid":"a537b59e-7ee7-4251-947f-27c964b73909","deletedId":"1c38a940-d849-466e-91ee-169561409fc4"}
  • Shortly before the error in the logs, there will be a line logging the API call using that same UUID
2018-12-14T14:43:47 [I|app|] Started GET "/rhsm/consumers/1c38a940-d849-466e-91ee-169561409fc4/certificates/serials" for 192.168.1.7 at 2018-12-14 14:43:47 -0500

In the simplest Satellite setup with no capsules, this will be the IP of the server that is making the request.
In a more complex setup with Red Hat Satellite Capsules, this IP address will be that of the capsule that is passing along the request.

  • Searching for the UUID in the Capsule's /var/log/httpd/katello-reverse-proxy_access_ssl.log will give you the IP address of the Content Host that originally made the call as well as the command that generated it.
# grep 1c38a940-d849-466e-91ee-169561409fc4 /var/log/httpd/katello-reverse-proxy_access_ssl.log
192.168.1.25 - - [14/Dec/2018:14:43:47 -0500] "GET /rhsm/consumers/1c38a940-d849-466e-91ee-169561409fc4/certificates/serials HTTP/1.1" 410 183 "-" "RHSM/1.0 (cmd=yum)"
192.168.1.25 - - [14/Dec/2018:14:43:49 -0500] "GET /rhsm/consumers/1c38a940-d849-466e-91ee-169561409fc4/certificates/serials HTTP/1.1" 410 183 "-" "RHSM/1.0 (cmd=yum)"
192.168.1.25 - - [14/Dec/2018:14:55:24 -0500] "GET /rhsm/consumers/1c38a940-d849-466e-91ee-169561409fc4/certificates/serials HTTP/1.1" 410 183 "-" "RHSM/1.0 (cmd=rhsmcertd-worker)"
SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.