[Satellite6] Upgrade of Satellite Server results in a FAIL for pulp and pulp_auth

Environment

• Red Hat Enterprise Linux
• Red Hat Satellite 6.1.x

Issue

• Upgraded the Satellite Server and the hammer ping fails.

[root@satellite-server ~]# hammer ping
candlepin:      
    Status:          ok
    Server Response: Duration: 589ms
candlepin_auth: 
    Status:          ok
    Server Response: Duration: 495ms
pulp:           
    Status:          FAIL
    Server Response: Message: 500 Internal Server Error
pulp_auth:      
    Status:          FAIL
    Server Response: Message: Skipped pulp_auth check after failed pulp check
elasticsearch:  
    Status:          ok
    Server Response: Duration: 52ms
foreman_tasks:  
    Status:          ok
    Server Response: Duration: 4ms

Resolution

• Check out the contents of all the json files and ensure while modifying the file nothing is missed.

# ll /etc/pulp/server/plugins.conf.d/*.json
-rw-r--r--. 1 root root 107 Nov 20 15:17 /etc/pulp/server/plugins.conf.d/docker_importer.json
-rw-r--r--. 1 root root 107 Nov 20 15:17 /etc/pulp/server/plugins.conf.d/iso_importer.json
-rw-r--r--. 1 root root 107 Nov 20 15:17 /etc/pulp/server/plugins.conf.d/puppet_importer.json
-rw-r--r--. 1 root root 107 Nov 20 15:17 /etc/pulp/server/plugins.conf.d/yum_importer.json

• The contents of all the json files should look like the following.

# cat /etc/pulp/server/plugins.conf.d/docker_importer.json
{
    "proxy_host": "",

    "proxy_port": null,
    
    "proxy_username": "",
    "proxy_password": ""
}

# cat /etc/pulp/server/plugins.conf.d/iso_importer.json 
{
    "proxy_host": "",

    "proxy_port": null,
    
    "proxy_username": "",
    "proxy_password": ""
}

# cat /etc/pulp/server/plugins.conf.d/puppet_importer.json 
{
    "proxy_host": "",

    "proxy_port": null,
    
    "proxy_username": "",
    "proxy_password": ""
}

# cat /etc/pulp/server/plugins.conf.d/yum_importer.json 
{
    "proxy_host": "",

    "proxy_port": null,
    
    "proxy_username": "",
    "proxy_password": ""
}

Note Even if a comma or null is missed in any of the above files then it is possible that you hit the following error.

# systemctl status pulp_resource_manager
● pulp_resource_manager.service - Pulp Resource Manager
   Loaded: loaded (/usr/lib/systemd/system/pulp_resource_manager.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2015-11-27 23:51:28 PST; 2min 13s ago
  Process: 28925 ExecStart=/usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%%h -Q resource_manager -c 1 --events --umask 18 (code=exited, status=1/FAILURE)
 Main PID: 28925 (code=exited, status=1/FAILURE)

Nov 27 23:51:28 satellite.example.com celery[28925]: File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
Nov 27 23:51:28 satellite.example.com celery[28925]: return _default_decoder.decode(s)
Nov 27 23:51:28 satellite.example.com celery[28925]: File "/usr/lib64/python2.7/json/decoder.py", line 365, in decode
Nov 27 23:51:28 satellite.example.com celery[28925]: obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Nov 27 23:51:28 satellite.example.com celery[28925]: File "/usr/lib64/python2.7/json/decoder.py", line 383, in raw_decode
Nov 27 23:51:28 satellite.example.com celery[28925]: raise ValueError("No JSON object could be decoded")
Nov 27 23:51:28 satellite.example.com celery[28925]: ValueError: No JSON object could be decoded
Nov 27 23:51:28 satellite.example.com systemd[1]: pulp_resource_manager.service: main process exited, code=exited, status=1/FAILURE
Nov 27 23:51:28 satellite.example.com systemd[1]: Unit pulp_resource_manager.service entered failed state.
Nov 27 23:51:28 satellite.example.com systemd[1]: pulp_resource_manager.service failed.

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

Root Cause

• json files got modified manually and the file is missing important parameters.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.