Katello-service status returns "Some services failed to start" in Satellite 6
Environment
- Red Hat Satellite 6.x
Issue
- While enabling a new repository, the satellite returns error :
There was an issue with the backend service pulp: Request Timeout
- When registering a Content Host to the Red Hat Satellite server facing service error
# subscription-manager register --org="Default_Organization" --activationkey="Act1" --force
Unable to register system, not all services available
Resolution
- To resolve the issue if mongod service fails, following commands need to be executed.
Delete the mongod.lock file, repair mongod db and start the service.
# rm /var/lib/mongodb/mongod.lock
# sudo -u mongodb mongod --dbpath /var/lib/mongodb --repair
Red Hat Enterprise Linux 6
# service mongodb start
Red Hat Enterprise Linux 7
# systemctl start mongod
Restart the katello-service
# katello-service restart
- Check if all services are running now
# katello-service status
For more KB articles/solutions related to Red Hat Satellite 6.x MongoDB Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x MongoDB-related Issues
Root Cause
-
mongod service was down
-
Katello-service was not running properly stating on the Satellite server "Some services failed to status: pulp_workers,pulp_resource_manager,pulp_celerybeat"
Diagnostic Steps
- katello-service status
Redirecting to /bin/systemctl status mongod.service
● mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
Active: failed (Result: timeout) since Mon 2016-02-01 14:48:09 CET; 1 day 23h ago
Process: 6876 ExecStart=/usr/bin/mongod $OPTIONS run (code=killed, signal=TERM)
Main PID: 17702 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/mongod.service
Feb 01 14:46:39 stefan.sat6.test systemd[1]: Starting High-performance, schema-free document-oriented database...
Feb 01 14:46:39 stefan.sat6.test mongod[6876]: about to fork child process, waiting until server is ready for connections.
Feb 01 14:46:39 stefan.sat6.test mongod[6876]: forked process: 6878
Feb 01 14:46:39 stefan.sat6.test mongod[6876]: all output going to: /var/lib/mongodb/mongodb.log
Feb 01 14:48:09 stefan.sat6.test systemd[1]: mongod.service start operation timed out. Terminating.
Feb 01 14:48:09 stefan.sat6.test systemd[1]: Failed to start High-performance, schema-free document-oriented database.
Feb 01 14:48:09 stefan.sat6.test systemd[1]: Unit mongod.service entered failed state.
Feb 01 14:48:09 stefan.sat6.test systemd[1]: mongod.service failed.
Redirecting to /bin/systemctl status qpidd.service
● qpidd.service - An AMQP message broker daemon.
Loaded: loaded (/usr/lib/systemd/system/qpidd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-02-01 14:07:19 CET; 2 days ago
Docs: man:qpidd(1)
http://qpid.apache.org/
Main PID: 4228 (qpidd)
CGroup: /system.slice/qpidd.service
└─4228 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf
Feb 01 14:07:19 stefan.sat6.test systemd[1]: Started An AMQP message broker daemon..
Feb 01 14:07:19 stefan.sat6.test systemd[1]: Starting An AMQP message broker daemon....
Feb 01 14:48:55 stefan.sat6.test systemd[1]: Started An AMQP message broker daemon..
Redirecting to /bin/systemctl status qdrouterd.service
● qdrouterd.service - Qpid Dispatch router daemon
Loaded: loaded (/usr/lib/systemd/system/qdrouterd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-02-01 14:07:19 CET; 2 days ago
Main PID: 4247 (qdrouterd)
CGroup: /system.slice/qdrouterd.service
└─4247 /usr/sbin/qdrouterd -c /etc/qpid-dispatch/qdrouterd.conf
Feb 01 14:07:33 stefan.sat6.test qdrouterd[4247]: Mon Feb 1 14:07:33 2016 CONN_MGR (info) Configured Listener: 0.0.0.0:5647 role=normal
Feb 01 14:07:33 stefan.sat6.test qdrouterd[4247]: Mon Feb 1 14:07:33 2016 CONN_MGR (info) Configured Listener: 0.0.0.0:5646 role=inter-router
Feb 01 14:07:33 stefan.sat6.test qdrouterd[4247]: Mon Feb 1 14:07:33 2016 CONN_MGR (info) Configured on-demand connector: stefan.sat6.test:5671 name=broker
Feb 01 14:07:33 stefan.sat6.test qdrouterd[4247]: Mon Feb 1 14:07:33 2016 ROUTER (info) Configured Link-route-pattern: prefix=pulp. dir=both connector=broker
Feb 01 14:07:33 stefan.sat6.test qdrouterd[4247]: Mon Feb 1 14:07:33 2016 ROUTER (info) Configured Link-route-pattern: prefix=qmf. dir=both connector=broker
Feb 01 14:07:33 stefan.sat6.test qdrouterd[4247]: Mon Feb 1 14:07:33 2016 CONN_MGR (info) Starting on-demand connector: broker
Feb 01 14:07:33 stefan.sat6.test qdrouterd[4247]: Mon Feb 1 14:07:33 2016 SERVER (info) Operational, 4 Threads Running
Feb 01 14:07:40 stefan.sat6.test qdrouterd[4247]: Mon Feb 1 14:07:40 2016 ROUTER (info) Activating Prefix 'pulp.' for routed links to 'broker'
Feb 01 14:07:40 stefan.sat6.test qdrouterd[4247]: Mon Feb 1 14:07:40 2016 ROUTER (info) Activating Prefix 'qmf.' for routed links to 'broker'
Feb 01 14:48:55 stefan.sat6.test systemd[1]: Started Qpid Dispatch router daemon.
Redirecting to /bin/systemctl status tomcat.service
● tomcat.service - Apache Tomcat Web Application Container
Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-02-01 14:07:19 CET; 2 days ago
Process: 4068 ExecStop=/usr/libexec/tomcat/server stop (code=exited, status=0/SUCCESS)
Main PID: 4268 (java)
CGroup: /system.slice/tomcat.service
└─4268 /usr/lib/jvm/jre/bin/java -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
Feb 01 14:09:44 stefan.sat6.test server[4268]: INFO: Deployment of web application directory /var/lib/tomcat/webapps/gutterball has finished in 5,429 ms
Feb 01 14:09:44 stefan.sat6.test server[4268]: Feb 01, 2016 2:09:44 PM org.apache.coyote.AbstractProtocol start
Feb 01 14:09:44 stefan.sat6.test server[4268]: INFO: Starting ProtocolHandler ["http-bio-8080"]
Feb 01 14:09:44 stefan.sat6.test server[4268]: Feb 01, 2016 2:09:44 PM org.apache.coyote.AbstractProtocol start
Feb 01 14:09:44 stefan.sat6.test server[4268]: INFO: Starting ProtocolHandler ["http-bio-8443"]
Feb 01 14:09:44 stefan.sat6.test server[4268]: Feb 01, 2016 2:09:44 PM org.apache.coyote.AbstractProtocol start
Feb 01 14:09:44 stefan.sat6.test server[4268]: INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Feb 01 14:09:44 stefan.sat6.test server[4268]: Feb 01, 2016 2:09:44 PM org.apache.catalina.startup.Catalina start
Feb 01 14:09:44 stefan.sat6.test server[4268]: INFO: Server startup in 127552 ms
Feb 01 14:48:55 stefan.sat6.test systemd[1]: Started Apache Tomcat Web Application Container.
● elasticsearch.service - LSB: Starts elasticsearch
Loaded: loaded (/etc/rc.d/init.d/elasticsearch)
Active: active (running) since Mon 2016-02-01 14:07:24 CET; 2 days ago
Docs: man:systemd-sysv-generator(8)
Process: 4020 ExecStop=/etc/rc.d/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
Process: 4321 ExecStart=/etc/rc.d/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/elasticsearch.service
└─4337 /bin/java -Xms256m -Xmx256m -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/java/elasticsearch -cp :/usr/share/java/elasticsearch/lib/elasticsearch-0.90.10.redhat-1.jar:/usr/share/java/lucene4-core.jar:/usr/share/java/lucene4-contrib/*:/usr/share/java/sigar/sigar.jar -Des.config=/etc/elasticsearch/elasticsearch.yml -Des.path.conf=/etc/elasticsearch -Des.path.home=/usr/share/java/elasticsearch -Des.path.logs=/var/log/elasticsearch -Des.path.data=/var/lib/elasticsearch -Des.path.work=/tmp/elasticsearch -Des.path.plugins=/usr/share/java/elasticsearch/plugins org.elasticsearch.bootstrap.ElasticSearch
Feb 01 14:07:19 stefan.sat6.test systemd[1]: Starting LSB: Starts elasticsearch...
Feb 01 14:07:22 stefan.sat6.test runuser[4329]: pam_unix(runuser:session): session opened for user elasticsearch by (uid=0)
Feb 01 14:07:23 stefan.sat6.test runuser[4329]: pam_unix(runuser:session): session closed for user elasticsearch
Feb 01 14:07:23 stefan.sat6.test elasticsearch[4321]: Starting elasticsearch: [ OK ]
Feb 01 14:07:24 stefan.sat6.test systemd[1]: Started LSB: Starts elasticsearch.
Feb 01 14:48:56 stefan.sat6.test systemd[1]: Started LSB: Starts elasticsearch.
Redirecting to /bin/systemctl status foreman-proxy.service
● foreman-proxy.service - Foreman Proxy
Loaded: loaded (/usr/lib/systemd/system/foreman-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2016-02-02 03:50:03 CET; 1 day 10h ago
Process: 6229 ExecStart=/usr/share/foreman-proxy/bin/smart-proxy (code=exited, status=0/SUCCESS)
Main PID: 6238 (ruby)
CGroup: /system.slice/foreman-proxy.service
└─6238 ruby /usr/share/foreman-proxy/bin/smart-proxy
Feb 02 03:50:02 stefan.sat6.test systemd[1]: Starting Foreman Proxy...
Feb 02 03:50:03 stefan.sat6.test systemd[1]: PID file /run/foreman-proxy/foreman-proxy.pid not readable (yet?) after start.
Feb 02 03:50:03 stefan.sat6.test systemd[1]: Started Foreman Proxy.
Redirecting to /bin/systemctl status pulp_celerybeat.service
● pulp_celerybeat.service - Pulp's Celerybeat
Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-02-01 14:08:24 CET; 2 days ago
Main PID: 4565 (celery)
CGroup: /system.slice/pulp_celerybeat.service
└─4565 /usr/bin/python /usr/bin/celery beat --scheduler=pulp.server.async.scheduler.Scheduler
Feb 03 14:33:31 stefan.sat6.test pulp[4565]: pulp.server.db.connection:ERROR: (4565-86080) ... Waiting 32 seconds and trying again.
Feb 03 14:34:01 stefan.sat6.test pulp[4565]: pulp.server.async.scheduler:ERROR: Cannot get collection from uninitialized database
Feb 03 14:34:03 stefan.sat6.test pulp[4565]: pulp.server.db.connection:ERROR: (4565-86080) Could not connect to MongoDB at localhost:27017:
Feb 03 14:34:03 stefan.sat6.test pulp[4565]: pulp.server.db.connection:ERROR: (4565-86080) Cannot connect to database default :
Feb 03 14:34:03 stefan.sat6.test pulp[4565]: pulp.server.db.connection:ERROR: (4565-86080) could not connect to localhost:27017: [Errno 111] Connection refused
Feb 03 14:34:03 stefan.sat6.test pulp[4565]: pulp.server.db.connection:ERROR: (4565-86080) ... Waiting 32 seconds and trying again.
Feb 03 14:34:35 stefan.sat6.test pulp[4565]: pulp.server.db.connection:ERROR: (4565-86080) Could not connect to MongoDB at localhost:27017:
Feb 03 14:34:35 stefan.sat6.test pulp[4565]: pulp.server.db.connection:ERROR: (4565-86080) Cannot connect to database default :
Feb 03 14:34:35 stefan.sat6.test pulp[4565]: pulp.server.db.connection:ERROR: (4565-86080) could not connect to localhost:27017: [Errno 111] Connection refused
Feb 03 14:34:35 stefan.sat6.test pulp[4565]: pulp.server.db.connection:ERROR: (4565-86080) ... Waiting 32 seconds and trying again.
Redirecting to /bin/systemctl status pulp_resource_manager.service
● pulp_resource_manager.service - Pulp Resource Manager
Loaded: loaded (/usr/lib/systemd/system/pulp_resource_manager.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-02-01 14:08:24 CET; 2 days ago
Main PID: 4584 (celery)
CGroup: /system.slice/pulp_resource_manager.service
└─4584 /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18
Feb 01 14:08:24 stefan.sat6.test systemd[1]: Started Pulp Resource Manager.
Feb 01 14:08:24 stefan.sat6.test systemd[1]: Starting Pulp Resource Manager...
Feb 01 14:08:48 stefan.sat6.test celery[4584]: No handlers could be found for logger "pulp.server.db.connection"
Feb 01 14:48:57 stefan.sat6.test systemd[1]: Started Pulp Resource Manager.
Redirecting to /bin/systemctl status pulp_workers.service
● pulp_workers.service - Pulp Celery Workers
Loaded: loaded (/usr/lib/systemd/system/pulp_workers.service; enabled; vendor preset: disabled)
Active: active (exited) since Mon 2016-02-01 14:08:24 CET; 2 days ago
Process: 3927 ExecStop=/usr/libexec/pulp-manage-workers stop (code=exited, status=0/SUCCESS)
Process: 4603 ExecStart=/usr/libexec/pulp-manage-workers start (code=exited, status=0/SUCCESS)
Main PID: 4603 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/pulp_workers.service
Feb 01 14:08:24 stefan.sat6.test systemd[1]: Starting Pulp Celery Workers...
Feb 01 14:08:24 stefan.sat6.test systemd[1]: Started Pulp Celery Workers.
Feb 01 14:48:57 stefan.sat6.test systemd[1]: Started Pulp Celery Workers.
Redirecting to /bin/systemctl status httpd.service
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-02-01 14:08:33 CET; 2 days ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 3880 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
Process: 3022 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
Main PID: 4632 (httpd)
Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec"
CGroup: /system.slice/httpd.service
├─ 4632 /usr/sbin/httpd -DFOREGROUND
├─ 4713 PassengerWatchdog
├─ 4716 PassengerHelperAgent
├─ 4724 PassengerLoggingAgent
├─ 4731 (wsgi:pulp) -DFOREGROUND
├─ 4732 /usr/sbin/httpd -DFOREGROUND
├─ 4733 /usr/sbin/httpd -DFOREGROUND
├─ 4734 /usr/sbin/httpd -DFOREGROUND
├─ 4735 /usr/sbin/httpd -DFOREGROUND
├─ 4736 /usr/sbin/httpd -DFOREGROUND
├─ 4737 /usr/sbin/httpd -DFOREGROUND
├─ 4738 /usr/sbin/httpd -DFOREGROUND
├─ 4739 /usr/sbin/httpd -DFOREGROUND
├─ 5995 Passenger RackApp: /etc/puppet/rack
├─ 6775 /usr/sbin/httpd -DFOREGROUND
├─ 6776 /usr/sbin/httpd -DFOREGROUND
├─ 6777 /usr/sbin/httpd -DFOREGROUND
├─ 6838 /usr/sbin/httpd -DFOREGROUND
├─20191 Passenger RackApp: /usr/share/foreman
├─25701 Passenger RackApp: /usr/share/foreman
├─25720 Passenger RackApp: /usr/share/foreman
└─25841 Passenger RackApp: /usr/share/foreman
Feb 03 14:33:31 stefan.sat6.test pulp[4731]: pulp.server.db.connection:ERROR: (4731-52032) could not connect to localhost:27017: [Errno 111] Connection refused
Feb 03 14:33:31 stefan.sat6.test pulp[4731]: pulp.server.db.connection:ERROR: (4731-52032) ... Waiting 32 seconds and trying again.
Feb 03 14:34:03 stefan.sat6.test pulp[4731]: pulp.server.db.connection:ERROR: (4731-52032) Could not connect to MongoDB at localhost:27017:
Feb 03 14:34:03 stefan.sat6.test pulp[4731]: pulp.server.db.connection:ERROR: (4731-52032) Cannot connect to database default :
Feb 03 14:34:03 stefan.sat6.test pulp[4731]: pulp.server.db.connection:ERROR: (4731-52032) could not connect to localhost:27017: [Errno 111] Connection refused
Feb 03 14:34:03 stefan.sat6.test pulp[4731]: pulp.server.db.connection:ERROR: (4731-52032) ... Waiting 32 seconds and trying again.
Feb 03 14:34:35 stefan.sat6.test pulp[4731]: pulp.server.db.connection:ERROR: (4731-52032) Could not connect to MongoDB at localhost:27017:
Feb 03 14:34:35 stefan.sat6.test pulp[4731]: pulp.server.db.connection:ERROR: (4731-52032) Cannot connect to database default :
Feb 03 14:34:35 stefan.sat6.test pulp[4731]: pulp.server.db.connection:ERROR: (4731-52032) could not connect to localhost:27017: [Errno 111] Connection refused
Feb 03 14:34:35 stefan.sat6.test pulp[4731]: pulp.server.db.connection:ERROR: (4731-52032) ... Waiting 32 seconds and trying again.
Redirecting to /bin/systemctl status foreman-tasks.service
● foreman-tasks.service - Foreman jobs daemon
Loaded: loaded (/usr/lib/systemd/system/foreman-tasks.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-02-01 14:09:35 CET; 2 days ago
Docs: https://github.com/theforeman/foreman-tasks
Process: 3836 ExecStop=/usr/bin/foreman-tasks stop (code=exited, status=0/SUCCESS)
Process: 4831 ExecStart=/usr/bin/foreman-tasks start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/foreman-tasks.service
├─4855 dynflow_executor_monitor
└─4857 dynflow_executor
Feb 01 14:08:38 stefan.sat6.test systemd[1]: Starting Foreman jobs daemon...
Feb 01 14:08:44 stefan.sat6.test foreman-tasks[4831]: Dynflow Executor: start in progress
Feb 01 14:08:45 stefan.sat6.test foreman-tasks[4831]: dynflow_executor: process with pid 4857 started.
Feb 01 14:08:45 stefan.sat6.test foreman-tasks[4831]: Waiting for the executor to be ready...
Feb 01 14:09:34 stefan.sat6.test foreman-tasks[4831]: ..................................................executor started successfully
Feb 01 14:09:35 stefan.sat6.test systemd[1]: Started Foreman jobs daemon.
Feb 01 14:49:02 stefan.sat6.test systemd[1]: Started Foreman jobs daemon.
Some services failed to status: mongod
- Check katello-service status
# katello-service status
satellite.example.com qpidd[21742]: 2018-07-31 13:50:12 [System] error JournalInactive:katello_event_queue couldn't setup next timer firing: satellite.example.com qpidd[21742]: 2018-07-31 13:50:12 [System] error JournalInactive:katello_event_queue couldn't setup next timer firing: -499167497ns[500ms]
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
/usr/sbin/service-wait qdrouterd status
Redirecting to /bin/systemctl status qdrouterd.service
goferd[21623]: [ERROR][Thread-4] katelloplugin:195 - gaierror: [Errno -2] Name or service not known
Aug 17 08:04:41 satellite.example.com goferd[21623]: [WARNING][Thread-4] katelloplugin:132 - [Errno -2] Name or service not known
Some services failed to status: pulp_workers,pulp_resource_manager,pulp_celerybeat
SBR
Product(s)
Components
Category
Tags
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.