"cookie store unavailable: hash" and "500 Internal Server Error" errors after restarting Satellite services on Red Hat Satellite 6.x server

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.17
  • Multiple external Red Hat Capsule 6.17 servers.

Issue

Issue

  • After restarting Satellite services on Red Hat Satellite 6.17.6.2 server, authenticated API calls (such as RHSM check-ins or virt-who uploads) fail with "500 Internal Server Error" and the message "cookie store unavailable: hash, error: key not found: hash" is recorded in the production logs.
  • The Satellite server was updated from 6.17.5 to 6.17.6.2 few days before the issue was observed.

Resolution

This issue has been reported to Satellite Engineering via the This content is not included.Recurring cookie store unavailable: :hash, error: key not found: :hash in Satellite under bulk RHSM load Jira ticket.

As a workaround:

  1. First, apply the steps in the Diagnostic Steps section of this solution article to confirm that the resolution suggested here applies to the reported issue.

  2. Stop all the Satellite services on all external Capsule servers:

    # satellite-maintain service stop

  3. Restart all services on the Satellite server to initialize a clean session store:

    # satellite-maintain service restart

  4. Start all Satellite services on the external Capsule servers to establish fresh authentication sessions:

    # satellite-maintain service start

Note:
If the issue is encountered after the Satellite server has been upgraded from one to the next major Satellite release, the KB solution After upgrading Red Hat Satellite server from the 6.15 to the 6.16 release, thousands of "cookie store unavailable: :hash, error: key not found: :hash" errors detected in Satellite's production.log might be more relevant.

Diagnostic Steps

  1. Check /var/log/foreman/production.log for a high volume of 500 errors associated with the following API calls:
  • GET "/rhsm/consumers/<UUID>/certificates/serials"
  • GET "/rhsm/status"
  1. Verify if the logs contain the following specific warning and stack trace:
[W app] cookie store unavailable: hash, error: key not found: hash
/usr/share/gems/gems/http-cookie-1.0.8/lib/http/cookie_jar.rb:40:in rescue in get_impl' /usr/share/gems/gems/http-cookie-1.0.8/lib/http/cookie_jar.rb:37:in get_impl'
/usr/share/gems/gems/http-cookie-1.0.8/lib/http/cookie_jar.rb:74:in initialize' /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:339:in process_cookie_args!'
  1. Confirm if the errors appeared immediately following a service restart or an update of the Satellite server while Capsule services remained running.
SBR
Product(s)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.