Hammer ping failing or pulp and pulp_auth failing
Environment
Satellite 6
Issue
When starting Satellite Service via foreman-maintain service start, the response of hammer ping is pulp and pulp_auth failing.
When setroubleshoot package is not installed the customer can't see much information related to the SELinux issue on the logs.
Resolution
In order to identify the issue it's recommended to install the package setroubleshoot, after that, the customer will be able to see on /var/log/messages the clear information about the block and the complete command to get the additional information and also how to fix the blocking.
For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues
Root Cause
When SELinux is enforcing, for some reason if the context was not applied correctly, some modules can fail during the start. Here we can see the issue affecting pulp, however, the procedure will be similar, independent of the component which is failing.
Diagnostic Steps
Here you can see the initial behavior. To guarantee the issue was related to SELinux, the configuration was changed from Enforcing to Permissive, after that the service was restarted and everything started as expected.
# hammer ping
candlepin:
Status: ok
Server Response: Duration: 13ms
candlepin_auth:
Status: ok
Server Response: Duration: 11ms
pulp:
Status: FAIL
Server Response:
pulp_auth:
Status: FAIL
foreman_tasks:
Status: ok
Server Response: Duration: 428ms
In the beginning, no information on /var/log/messages, that said, the package setroubleshoot was installed and after that we were able to see some additional information as we can see below:
$ grep setroubleshoot messages
Feb 28 12:19:10 server_fqdn setroubleshoot: SELinux is preventing /usr/bin/python2.7 from getattr access on the filesystem /sys/fs/cgroup. For complete SELinux messages run: sealert -l db20e03e-11e3-4e36-8184-ce3252775bac
Feb 28 12:19:22 server_fqdn setroubleshoot: SELinux is preventing /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/bin/java from unlink access on the file 18415. For complete SELinux messages run: sealert -l 170c1ac4-7bd7-42c5-8cce-fc9717eeb701
...
Proceeding with the presented command sealert -l db20e03e-11e3-4e36-8184-ce3252775bac the customer will be able to see the complete information as we can see below
# sealert -l db20e03e-11e3-4e36-8184-ce3252775bac
SELinux is preventing /usr/bin/python2.7 from write access on the directory /dev/shm.
***** Plugin catchall (100. confidence) suggests **************************
If you believe that python2.7 should be allowed write access on the shm directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'celery' --raw | audit2allow -M my-celery
# semodule -i my-celery.pp
Additional Information:
Source Context system_u:system_r:celery_t:s0
Target Context system_u:object_r:tmpfs_t:s0
Target Objects /dev/shm [ dir ]
Source celery
Source Path /usr/bin/python2.7
Port <Unknown>
Host server_fqdn
Source RPM Packages python-2.7.5-76.el7.x86_64
Target RPM Packages
Policy RPM selinux-policy-3.13.1-229.el7_6.5.noarch
Selinux Enabled True
Policy Type targeted
Enforcing Mode Permissive
Host Name server_fqdn
Platform Linux server_fqdn 3.10.0-957.el7.x86_64 #1 SMP
Thu Oct 4 20:48:51 UTC 2018 x86_64 x86_64
Alert Count 1
First Seen 2020-02-28 17:23:28 EST
Last Seen 2020-02-28 17:23:28 EST
Local ID 0d4e87a2-1fb6-4b27-b1d6-5477f044a01f
Raw Audit Messages
type=AVC msg=audit(1582928608.763:8688): avc: denied { write } for pid=17432 comm="celery" name="/" dev="tmpfs" ino=2259 scontext=system_u:system_r:celery_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1582928608.763:8688): avc: denied { add_name } for pid=17432 comm="celery" name="sem.LQ42pz" scontext=system_u:system_r:celery_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1582928608.763:8688): avc: denied { create } for pid=17432 comm="celery" name="sem.LQ42pz" scontext=system_u:system_r:celery_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file permissive=1
type=AVC msg=audit(1582928608.763:8688): avc: denied { read write open } for pid=17432 comm="celery" path="/dev/shm/sem.LQ42pz" dev="tmpfs" ino=3012669 scontext=system_u:system_r:celery_t:s0 tcontext=system_u:object_r:tmpfs_t:s0 tclass=file permissive=1
type=SYSCALL msg=audit(1582928608.763:8688): arch=x86_64 syscall=open success=yes exit=EBADF a0=7ffd01d81fe0 a1=c2 a2=180 a3=5e5992e0 items=0 ppid=1 pid=17432 auid=4294967295 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295 comm=celery exe=/usr/bin/python2.7 subj=system_u:system_r:celery_t:s0 key=(null)
Hash: celery,celery_t,tmpfs_t,dir,write
Note, the command to fix is already on the description above and we can see the snippet below:
...
# ausearch -c 'celery' --raw | audit2allow -M my-celery
# semodule -i my-celery.pp
...
Now, let's check a real scenario, analyzing the /var/log/message logs
# grep setroubleshoot messages | grep -o sealert.* | sort | uniq -c
4 sealert -l 0d4e87a2-1fb6-4b27-b1d6-5477f044a01f
1 sealert -l 170c1ac4-7bd7-42c5-8cce-fc9717eeb701
2 sealert -l 61435e81-fc4b-45da-8fd6-015d98c3ea49
2 sealert -l 784afd15-d791-409b-a672-a987e70bfcdf
2 sealert -l 89889614-f03d-4478-b16c-d0a71928e0a8
1 sealert -l 90f2a22d-7dcc-4dbb-9313-7d94c1d943de
2 sealert -l a7b1ace6-619c-4a43-ad4b-926178d39d93
74 sealert -l db20e03e-11e3-4e36-8184-ce3252775bac
After checking all of them, the way to fix was
# ausearch -c 'celery' --raw | audit2allow -M my-celery
# semodule -i my-celery.pp
# ausearch -c 'java' --raw | audit2allow -M my-java
# semodule -i my-java.pp
At the end, the selinux was changed back to Enforcing with no issues anymore.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.