[Satellite6] Calling hammer commands concurrently raises 500 ISE error on apidoc / apipie

Environment

Satellite 6.12 or newer

Issue

On a fresh installation of Sat6.12 or after an update/upgrade of it, calling hammer commands concurrently sometimes raises 500 Internal Server Error.

Resolution

Run a hammer command sequentially and ensure ~/.cache/apipie_bindings/*/v2/*json file exists / has recent mtime.

For more KB articles/solutions related to Red Hat Satellite 6.x hammer Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x hammer-related Issues.

Root Cause

hammer - as a first step of communication with Satellite - checks if it knows up-to-date version of Satellite API. If not, it fetches the whole APIdoc from Satellite and stores it in ~/.cache/apipie_bindings/*/v2/*json file.

Due to a recent change in Satellite 6.12, APIdoc on Satellite supports sequential requests only, not parallel. And concurrent requests there end up in 500 error that hammer exhibits to the user.

Running a hammer request sequentially will let it fetch current APIdoc and store it in its cache. Then subsequent - even concurrent - hammer requests just validate the cache is up-to-date and dont fetch the APIdoc at all.

This applies both to new installations, as well as updates/upgrades where APIdoc changes.

This behaviour is tracked under This content is not included.this bugzilla for possible improvement of hammer.

Diagnostic Steps

Some of concurrent hammer requests fail with error:

Could not load the API description from the server: 
500 Internal Server Error
  - is the server down?
  - was 'foreman-rake apipie:cache' run on the server when using apipie cache? (typical production settings)
Warning: An error occured while loading module hammer_cli_foreman.

Running the foreman-rake command does not help.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.