After upgrading Red Hat Satellite server from the 6.15 to the 6.16 release, thousands of "cookie store unavailable: :hash, error: key not found: :hash" errors detected in Satellite's production.log

Solution Verified - Updated 20 Feb 2026

Environment

Red Hat Satellite 6.15 server.
Multiple external Red Hat Capsule 6.15 servers in loadbalanced configuration.
10k+ managed hosts registered to Satellite/Capsule servers.

Issue

Getting lots of errors like the following one after Satellite upgrade from the 6.15 to the 6.16 release:
```
cookie store unavailable: :hash, error: key not found: :hash
```
Operations success rate on the Satellite server was critically low due to these errors.

When proceeding to upgrade the external Capsules, the following error was encountered:

Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-7885 [ProxyAPI::ProxyException]: Unable to fetch logs ([ArgumentError]: cookie store unavailable: :hash, error: key not found: :hash) for Capsule https://loadbalancer.example.com:9090/logs)

Hosts registered to the Satellite/Capsule servers report the following error in /var/log/rhsm/rhsm.log:

2025-04-16 23:15:39,832 [WARNING] dnf.py:3689895:MainThread @cache.py:745 - Unable to query for content access updates: HTTP error (500 - Internal Server Error): cookie store unavailable: :hash, error: key not found: :hash
2025-04-16 23:15:52,004 [INFO] dnf.py:3690631:MainThread @entcertlib.py:108 - certs updated:
Total updates: 0
Found (local) serial# [3458679433474532973]
Expected (UEP) serial# [3458679433474532973]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2025-04-16 23:16:47,408 [INFO] dnf.py:3694003:MainThread @entcertlib.py:108 - certs updated:
Total updates: 0
Found (local) serial# [3458679433474532973]
Expected (UEP) serial# [3458679433474532973]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>
2025-04-16 23:16:47,562 [WARNING] dnf.py:3694003:MainThread @cache.py:745 - Unable to query for content access updates: HTTP error (500 - Internal Server Error): cookie store unavailable: :hash, error: key not found: :hash
2025-04-16 23:16:51,103 [INFO] dnf.py:3694075:MainThread @entcertlib.py:108 - certs updated:
Total updates: 0

The count of the cookie store unavailable: :hash, error: key not found: :hash errors in /var/log/foreman/production.log on the Satellite server kept growing drastically:
```
# cat /var/log/foreman/production.log| grep "cookie store unavailable" | wc -l
18003
```
and after one hour:
```
# cat /var/log/foreman/production.log| grep "cookie store unavailable" | wc -l
32071
```

Resolution

Roll back the Satellite server upgrade to the 6.15 release, by restoring from a backup/snapshot taken before starting the upgrade.
Upgrade the Satellite server during a low load time window (for example on Friday evening).
After the upgrade of the Satellite server completes successfully, re-disable the virt-who service on the Satellite server (see the Root Cause section for more details).
Monitor the Satellite server for any unexpected behavior for the next 24 hours.
If no issues are detected, proceed with upgrading the external Capsules one at a time, with a time window of 1 to 2 hours between each Capsule upgraded (at least for the first 2 or 3 Capsules). During this time, monitor the Satellite server and the upgraded Capsules for any unexpected behavior.
If things are stable, then proceed with the rest of the Capsule upgrades without the need to extensively monitor the situation after each Capsule upgraded.

Note:
If the issue is encountered after restarting the Satellite services on the Satellite server few days after an update of the Satellite server to latest minor release, the KB solution "cookie store unavailable: hash" and "500 Internal Server Error" errors after restarting Satellite services on Red Hat Satellite 6.x server might be more relevant.

For more KB articles/solutions related to Red Hat Satellite 6.x Installation/Upgrade/Update Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Installation/Upgrade/Update Issues.

Root Cause

The virt-who service was manually disabled on the Satellite as a workaround for the issue reported in This content is not included.Bug 2173870 - When using the customer data (json) with 13 diff conf files, we can see some weird behavior when updating the hypervisors.
When the Satellite server was upgraded from the 6.15 to the 6.16 release, the virt-who service was enabled by the Satellite installer. This caused multiple Actions::Katello::Host::Hypervisors tasks to be launched simultaneously. These tasks resulted in a massive number of GET "/rhsm/consumers/<uuid>/certificates/serials" API calls to be generated, effectively putting the Satellite server under a form of denial of service (DoS) attack. Upgrading the Capsule servers shortly after upgrading the Satellite server emphasized the effect.

Diagnostic Steps

In /var/log/foreman/production.log on the Satellite server:

2025-04-16T21:53:38 [I|app|5095ab0f] Started GET "/rhsm/consumers/a5c04966-0ddf-4c87-a26a-0afdb6e72f16/content_overrides" for 80.74.208.72 at 2025-04-16 21:53:38 +0200
2025-04-16T21:53:38 [W|app|5095ab0f] cookie store unavailable: :hash, error: key not found: :hash
 5095ab0f | /usr/share/gems/gems/http-cookie-1.0.6/lib/http/cookie_jar.rb:40:in `rescue in get_impl'
 5095ab0f | /usr/share/gems/gems/http-cookie-1.0.6/lib/http/cookie_jar.rb:37:in `get_impl'
 5095ab0f | /usr/share/gems/gems/http-cookie-1.0.6/lib/http/cookie_jar.rb:74:in `initialize'
 .
 .
 5095ab0f | /usr/share/gems/gems/puma-6.4.3/lib/puma/server.rb:245:in `block in run'
 5095ab0f | /usr/share/gems/gems/puma-6.4.3/lib/puma/thread_pool.rb:155:in `block in spawn_thread'
 5095ab0f | /usr/share/gems/gems/logging-2.4.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2025-04-16T21:53:38 [I|app|5095ab0f] Completed 500 Internal Server Error in 13ms (Views: 0.1ms | ActiveRecord: 1.6ms | Allocations: 8218)

In /var/log/foreman-maintain/foreman-maintain.log on an external Capsule server, the following error was logged while upgrading the Capsule:

  I, [2025-04-16 20:10:09+0200 #103837]  INFO -- : === Scenario 'Procedures before migrating' finished ===
  I, [2025-04-16 20:10:09+0200 #103837]  INFO -- : === Scenario 'Migration scripts' started ===
  I, [2025-04-16 20:10:09+0200 #103837]  INFO -- : --- Execution step 'Setup repositories' [repositories-setup] started ---
  E, [2025-04-16 20:10:11+0200 #103837] ERROR -- : Failed executing subscription-manager repos --disable=*, exit status 70:
   Unable to sync syspurpose with server: HTTP error (500 - Internal Server Error): cookie store unavailable: :hash, error: key not found: :hash

  cookie store unavailable: :hash, error: key not found: :hash (HTTP error code 500: Internal Server Error) (ForemanMaintain::Error::ExecutionError)
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/utils/command_runner.rb:52:in `execution_error'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/concerns/system_helpers.rb:69:in `execute!'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/repository_manager/el.rb:12:in `rhsm_disable_repos'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/concerns/downstream.rb:21:in `setup_repositories'
  /usr/share/gems/gems/foreman_maintain-1.7.13/definitions/procedures/repositories/setup.rb:17:in `block in run'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/reporter/cli_reporter.rb:146:in `with_spinner'
  /usr/share/ruby/forwardable.rb:235:in `with_spinner'
  /usr/share/ruby/forwardable.rb:235:in `with_spinner'
  /usr/share/gems/gems/foreman_maintain-1.7.13/definitions/procedures/repositories/setup.rb:15:in `run'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/executable.rb:142:in `__run__'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/runner/execution.rb:87:in `block (2 levels) in run'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/runner/execution.rb:111:in `capture_errors'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/runner/execution.rb:86:in `block in run'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/runner/execution.rb:105:in `with_metadata_calculation'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/runner/execution.rb:85:in `run'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/runner.rb:134:in `run_step'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/runner.rb:123:in `run_steps'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/runner.rb:116:in `execute_scenario_steps'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/runner.rb:61:in `run_scenario'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/upgrade_runner.rb:128:in `block in run_phase'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/upgrade_runner.rb:173:in `with_non_empty_scenario'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/upgrade_runner.rb:123:in `run_phase'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/upgrade_runner.rb:70:in `block in run'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/upgrade_runner.rb:64:in `each'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/upgrade_runner.rb:64:in `run'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/cli/upgrade_command.rb:64:in `block in execute'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/cli/upgrade_command.rb:23:in `try_upgrade'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/cli/upgrade_command.rb:63:in `execute'
  /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:66:in `run'
  /usr/share/gems/gems/clamp-1.3.2/lib/clamp/subcommand/execution.rb:18:in `execute'
  /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:66:in `run'
  /usr/share/gems/gems/clamp-1.3.2/lib/clamp/subcommand/execution.rb:18:in `execute'
  /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:66:in `run'
  /usr/share/gems/gems/foreman_maintain-1.7.13/lib/foreman_maintain/cli.rb:40:in `run'
  /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:140:in `run'
  /usr/share/gems/gems/foreman_maintain-1.7.13/bin/foreman-maintain:15:in `<top (required)>'
  /bin/satellite-maintain:23:in `load'
  /bin/satellite-maintain:23:in `<main>'

In /var/log/foreman/production.log on the Satellite server, the majority of the ArgumentError: cookie store unavailable: :hash, error: key not found: :hash errors were reported by API calls like these:

  2025-04-16T19:53:34 [I|app|1e6e4400] Started GET "/rhsm/consumers/c91bc76a-82ee-4b88-8429-6fa86cdb056e/certificates/serials" for 10.88.79.111 at 2025-04-16 19:53:34 +0200

  2025-04-16T19:53:34 [I|app|d1b21777] Started GET "/rhsm/consumers/a3c81757-439c-4058-9034-3c300267b40c/certificates/serials" for 10.88.79.111 at 2025-04-16 19:53:34 +0200

  2025-04-16T19:53:34 [I|app|33fcfd1a] Started GET "/rhsm/" for 10.88.79.111 at 2025-04-16 19:53:34 +0200

  2025-04-16T19:53:35 [I|app|4c4c9610] Started GET "/rhsm/consumers/91ed2f88-e957-45cd-b6f2-52e59f16418d/certificates/serials" for 10.88.79.111 at 2025-04-16 19:53:35 +0200

  2025-04-16T19:53:35 [I|app|e9eacbe4] Started GET "/rhsm/consumers/91ed2f88-e957-45cd-b6f2-52e59f16418d" for 10.88.79.111 at 2025-04-16 19:53:35 +0200
  
  2025-04-16T19:53:34 [I|app|f46dc028] Started GET "/rhsm/consumers/fb08c31f-ba2e-415d-8e02-08a5b6c775a3/certificates/serials" for 10.88.79.109 at 2025-04-16 19:53:34 +0200

  2025-04-16T19:53:34 [I|app|ed79e39d] Started GET "/rhsm/consumers/7ed2dbb8-bda8-4ff3-9c20-08823c548010/certificates/serials" for 10.88.79.110 at 2025-04-16 19:53:34 +0200

  2025-04-16T19:53:34 [I|app|1cb4e390] Started GET "/rhsm/consumers/80c96a9d-d2a7-4a30-b7cc-bedaae289827/certificates/serials" for 10.87.142.5 at 2025-04-16 19:53:34 +0200

SBR

SysMgmt

Product(s)

Red Hat Satellite

Components

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.