[Satellite 6] pulp services are down because some of the pulp plugins failed to initialize

Solution Verified - Updated

Environment

  • Satellite 6

Issue

  • pulp services are down due to:
ERROR: One or more plugins failed to initialize. If a new type has been added, run pulp-manage-db to load the type into the database and restart the application. Error message: Expecting property name: line 11 column 1 (char 136)

Resolution

  • Fix the syntax (preferably using satellite-installer) and restart the services via foreman-maintain service restart.

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

Root Cause

  • The files under /etc/pulp/server/plugins.conf.d/ were not in a correct format:
# cat etc/pulp/server/plugins.conf.d/yum_importer.json
{
        "proxy_host": "http://XX.XXX.XX.XX",

        "proxy_port": XXXX,


    "proxy_username": null,


    "proxy_password": ********


    "max_speed": null,

    "gpg_sign_metadata": false
}

# cat /etc/pulp/server/plugins.conf.d/yum_importer.json | json_reformat
lexical error: invalid char in json text.
                                        {         "proxy_host": "http:
                     (right here) ------^

Diagnostic Steps

  • pulp services are down:
# hammer ping
candlepin:
    Status:          ok
    Server Response: Duration: 23ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 22ms
pulp:
    Status:          FAIL
    Server Response:
pulp_auth:
    Status: FAIL
foreman_tasks:
    Status:          ok
    Server Response: Duration: 4ms
  • Following errors can be seen in /var/log/messages:
Nov  1 14:37:23 satellite.example.com pulp: pulp.server.webservices.application:CRITICAL: *************************************************************
Nov  1 14:37:23 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17176-84480) The Pulp server encountered an unexpected failure during initialization
Nov  1 14:37:23 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17176-84480) Traceback (most recent call last):
Nov  1 14:37:23 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17176-84480)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 111, in wsgi_application
Nov  1 14:37:23 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17176-84480)     _initialize_web_services()
Nov  1 14:37:23 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17176-84480)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 66, in _initialize_web_services
Nov  1 14:37:23 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17176-84480)     initialization.initialize()
Nov  1 14:37:23 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17176-84480)   File "/usr/lib/python2.7/site-packages/pulp/server/initialization.py", line 31, in initialize
Nov  1 14:37:23 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17176-84480)     db_connection.initialize()
Nov  1 14:37:23 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17176-84480)   File "/usr/lib/python2.7/site-packages/pulp/server/db/connection.py", line 61, in initialize
Nov  1 14:37:23 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17176-84480)     raise RuntimeError("The database is already initialized. It is an error to call this "
Nov  1 14:37:23 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17176-84480) RuntimeError: The database is already initialized. It is an error to call this function a second time.
Nov  1 14:37:23 satellite.example.com pulp: pulp.server.webservices.application:CRITICAL: *************************************************************

Nov  1 14:31:15 satellite.example.com pulp: pulp.plugins.loader.manager:INFO: Loaded plugin group_export_distributor for types: rpm,srpm,drpm,erratum,distribution,package_category,p
ackage_group
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.initialization:ERROR: One or more plugins failed to initialize. If a new type has been added, run pulp-manage-db to load the 
type into the database and restart the application. Error message: Expecting property name: line 11 column 1 (char 136)
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:CRITICAL: *************************************************************
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040) The Pulp server encountered an unexpected failure during initialization
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040) Traceback (most recent call last):
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py"
, line 111, in wsgi_application
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)     _initialize_web_services()
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py"
, line 66, in _initialize_web_services
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)     initialization.initialize()
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)   File "/usr/lib/python2.7/site-packages/pulp/server/initialization.py", line 36
, in initialize
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)     plugin_api.initialize()
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)   File "/usr/lib/python2.7/site-packages/pulp/plugins/loader/api.py", line 55, i
n initialize
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)     loading.load_plugins_from_entry_point(*entry_point)
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)   File "/usr/lib/python2.7/site-packages/pulp/plugins/loader/loading.py", line 60, in load_plugins_from_entry_point
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)     cls, cfg = entry_point.load()()
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)   File "/usr/lib/python2.7/site-packages/pulp_ostree/plugins/importers/web.py", line 18, in entry_point
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)     config = read_json_config(constants.IMPORTER_CONFIG_FILE_PATH)
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)   File "/usr/lib/python2.7/site-packages/pulp/common/config.py", line 681, in read_json_config
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)     config = json.load(f)
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)   File "/usr/lib64/python2.7/json/__init__.py", line 290, in load
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)     **kw)
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)   File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)     return _default_decoder.decode(s)
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)   File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)   File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040)     obj, end = self.scan_once(s, idx)
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:ERROR: (17178-51040) ValueError: Expecting property name: line 11 column 1 (char 136)
Nov  1 14:31:15 satellite.example.com pulp: pulp.server.webservices.application:CRITICAL: *************************************************************
SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.