Why unable to start Gluster-Swift services using 'systemctl' on Red Hat Gluster Storage 3.1 ISO ?
Environment
- Red Hat Gluster Storage 3.1
- Red Hat Enterprise Linux 6.7
- Red Hat Enterprise Linux 7.1
Issue
-
Why unable to start Gluster-Swift services using 'systemctl' on Red Hat Gluster Storage 3.1 ISO ?
-
Did 'systemctl' supported for gluster swift services other than 'swift-init' for starting gluster swift ?
-
Unable to start gluster swift services with 'systemctl' on Red Hat Gluster Storage 3.1 based on Red Hat Enterprise Linux 7.1 ?
~snippet from terminal,
[root@gluster-swift ~]# systemctl status openstack-swift-proxy openstack-swift-account openstack-swift-container openstack-swift-object
openstack-swift-proxy.service - OpenStack Object Storage (swift) - Proxy Server
Loaded: loaded (/usr/lib/systemd/system/openstack-swift-proxy.service; disabled)
Active: failed (Result: exit-code) since Thu 2015-11-05 13:52:59 GMT; 11s ago
Process: 13049 ExecStart=/usr/bin/swift-proxy-server /etc/swift/proxy-server.conf (code=exited, status=1/FAILURE)
Main PID: 13049 (code=exited, status=1/FAILURE)
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-proxy-server[13049]: Traceback (most recent call last):
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-proxy-server[13049]: File "/usr/bin/swift-proxy-server", line 23, in <module>
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-proxy-server[13049]: sys.exit(run_wsgi(conf_file, 'proxy-server', default_port=8080, **options))
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-proxy-server[13049]: File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 383, in run_wsgi
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-proxy-server[13049]: drop_privileges(conf.get('user', 'swift'))
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-proxy-server[13049]: File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 1219, in drop_privileges
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-proxy-server[13049]: os.setgid(user[3])
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-proxy-server[13049]: OSError: [Errno 1] Operation not permitted <<================
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net systemd[1]: openstack-swift-proxy.service: main process exited, code=exited, status=1/FAILURE
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net systemd[1]: Unit openstack-swift-proxy.service entered failed state.
openstack-swift-account.service - OpenStack Object Storage (swift) - Account Server
Loaded: loaded (/usr/lib/systemd/system/openstack-swift-account.service; disabled)
Active: failed (Result: exit-code) since Thu 2015-11-05 13:52:59 GMT; 11s ago
Process: 13050 ExecStart=/usr/bin/swift-account-server /etc/swift/account-server.conf (code=exited, status=1/FAILURE)
Main PID: 13050 (code=exited, status=1/FAILURE)
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-account-server[13050]: Traceback (most recent call last):
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-account-server[13050]: File "/usr/bin/swift-account-server", line 24, in <module>
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-account-server[13050]: 'account-server', default_port=6002, **options))
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-account-server[13050]: File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 383, in run_wsgi
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-account-server[13050]: drop_privileges(conf.get('user', 'swift'))
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-account-server[13050]: File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 1219, in drop_privileges
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-account-server[13050]: os.setgid(user[3])
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-account-server[13050]: OSError: [Errno 1] Operation not permitted
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net systemd[1]: openstack-swift-account.service: main process exited, code=exited, status=1/FAILURE
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net systemd[1]: Unit openstack-swift-account.service entered failed state.
openstack-swift-container.service - OpenStack Object Storage (swift) - Container Server
Loaded: loaded (/usr/lib/systemd/system/openstack-swift-container.service; disabled)
Active: failed (Result: exit-code) since Thu 2015-11-05 13:52:59 GMT; 11s ago
Process: 13051 ExecStart=/usr/bin/swift-container-server /etc/swift/container-server.conf (code=exited, status=1/FAILURE)
Main PID: 13051 (code=exited, status=1/FAILURE)
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-container-server[13051]: Traceback (most recent call last):
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-container-server[13051]: File "/usr/bin/swift-container-server", line 24, in <module>
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-container-server[13051]: 'container-server', default_port=6001, **options))
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-container-server[13051]: File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 383, in run_wsgi
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-container-server[13051]: drop_privileges(conf.get('user', 'swift'))
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-container-server[13051]: File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 1219, in drop_privileges
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-container-server[13051]: os.setgid(user[3])
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-container-server[13051]: OSError: [Errno 1] Operation not permitted <<====================
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net systemd[1]: openstack-swift-container.service: main process exited, code=exited, status=1/FAILURE
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net systemd[1]: Unit openstack-swift-container.service entered failed state.
openstack-swift-object.service - OpenStack Object Storage (swift) - Object Server
Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object.service; disabled)
Active: failed (Result: exit-code) since Thu 2015-11-05 13:52:59 GMT; 11s ago
Process: 13052 ExecStart=/usr/bin/swift-object-server /etc/swift/object-server.conf (code=exited, status=1/FAILURE)
Main PID: 13052 (code=exited, status=1/FAILURE)
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-object-server[13052]: Traceback (most recent call last):
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-object-server[13052]: File "/usr/bin/swift-object-server", line 27, in <module>
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-object-server[13052]: **options))
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-object-server[13052]: File "/usr/lib/python2.7/site-packages/swift/common/wsgi.py", line 383, in run_wsgi
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-object-server[13052]: drop_privileges(conf.get('user', 'swift')) <<====================
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-object-server[13052]: File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 1219, in drop_privileges
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-object-server[13052]: os.setgid(user[3])
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net swift-object-server[13052]: OSError: [Errno 1] Operation not permitted
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net systemd[1]: openstack-swift-object.service: main process exited, code=exited, status=1/FAILURE
Nov 05 13:52:59 hh-rhs-ap41.lnx.lr.net systemd[1]: Unit openstack-swift-object.service entered failed state.
Resolution
Pre-requisite for configuring Object Store (Gluster-swift):
-
openstack-swift-*andswiftonfilepackage version should be similar. -
The gluster swift services should run as "root" user i.e,
account-server.conf:user = root
container-server.conf:user = root
object-expirer.conf:user = root
object-expirer.conf:user = root
object-server.conf:user = root
proxy-server.conf:user = root
Note: The gluster swift services should run as "root" user as compared to RHEL openstack swift where services run as "swift" user.
Configure Object Store (Gluster-swift) in Red Hat Gluster Storage:
- Refer documentation for This content is not included.Configuring Object Store in your Red Hat Gluster Storage
Starting and Stopping Gluster Swift Operations:
To start gluster swift services, execute following command,
# swift-init main start
To stop gluster swift services, execute following command,
# swift-init main stop
To restart the gluster swift services, whenever the configuration files are modified, execute following command,
# swift-init main restart
To identify the gluster swift services status, execute following command,
# swift-init main status
Starting the Gluster Swift services at boot on Red Hat Enterprise Linux 6.7:
# chkconfig memcached on
# chkconfig openstack-swift-proxy on
# chkconfig openstack-swift-account on
# chkconfig openstack-swift-container on
# chkconfig openstack-swift-object on
# chkconfig openstack-swift-object-expirer on
Starting the Gluster Swift services at boot on Red Hat Enterprise Linux 7.1:
Pre-requisite:
-
SElinux should be in permissive mode
-
The gluster swift services should run as "root" user as compared to RHEL openstack swift where services run as "swift" user.
Start the gluster swift services before & after reboot as,
# systemctl start openstack-swift-proxy.service openstack-swift-account.service openstack-swift-container.service openstack-swift-object.service openstack-swift-object-expirer.service
# systemctl enable openstack-swift-proxy.service openstack-swift-account.service openstack-swift-container.service openstack-swift-object.service openstack-swift-object-expirer.service
# systemctl status openstack-swift-proxy.service openstack-swift-account.service openstack-swift-container.service openstack-swift-object.service openstack-swift-object-expirer.service
Note: systemctl can be used as an alternative to 'swif-init main start' for gluster swift services operations in Red Hat Enterprise Linux 7.1 for starting gluster swift services at boot.
Diagnostic Steps
Using 'systemctl' as an alternate solution to 'swift-init' to start the gluster swift services on Red Hat Enterprise Linux 7.1 as,
[root@gluster-swift ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.1 (Maipo)
[root@gluster-swift ~]# cat /etc/redhat-storage-release
Red Hat Gluster Storage Server 3.1 Update 1
Gluster swift packages,
[root@gluster-swift ~]# rpm -qa |grep swift
openstack-swift-account-1.13.1-6.el7ost.noarch
openstack-swift-plugin-swift3-1.7-3.el7ost.noarch
openstack-swift-proxy-1.13.1-6.el7ost.noarch
openstack-swift-1.13.1-6.el7ost.noarch
openstack-swift-container-1.13.1-6.el7ost.noarch
swiftonfile-1.13.1-6.el7rhgs.noarch
openstack-swift-object-1.13.1-6.el7ost.noarch
Gluster swift configuration files,
[root@gluster-swift swift]# ll /etc/swift/
total 64
-rw-r--r--. 1 root root 481 Nov 5 18:22 account.builder
-rw-r--r--. 1 root root 186 Nov 5 18:22 account.ring.gz
drwxr-xr-x. 2 root root 6 Aug 26 21:32 account-server
-rw-r--r--. 1 root root 1287 Oct 6 11:36 account-server.conf
drwxr-xr-x. 2 root root 4096 Nov 5 18:22 backups
-rw-r--r--. 1 root root 1253 Nov 10 15:10 cert.crt
-rw-r--r--. 1 root root 1704 Nov 10 15:10 cert.key
-rw-r--r--. 1 root root 481 Nov 5 18:22 container.builder
-rw-r--r--. 1 root root 187 Nov 5 18:22 container.ring.gz
drwxr-xr-x. 2 root root 6 Aug 26 21:32 container-server
-rw-r--r--. 1 root root 1298 Oct 6 11:36 container-server.conf
-rw-r--r--. 1 root root 1272 Oct 6 11:36 fs.conf
-rw-r--r--. 1 root root 481 Nov 5 18:22 object.builder
-rw-r--r--. 1 root root 1384 Oct 6 11:36 object-expirer.conf
-rw-r--r--. 1 root root 183 Nov 5 18:22 object.ring.gz
drwxr-xr-x. 2 root root 6 Aug 26 21:32 object-server
-rw-r--r--. 1 root root 2211 Oct 6 11:36 object-server.conf
drwxr-xr-x. 2 root root 6 Aug 26 21:32 proxy-server
-rw-r--r--. 1 root root 2761 Oct 6 11:36 proxy-server.conf
-rw-r--r--. 1 root root 2545 Oct 6 11:36 swift.conf
SELinux should be permissive as,
[root@gluster-swift swift]# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28
[root@gluster-swift system]# pwd
/usr/lib/systemd/system
Append the swift user to root user in the openstack-swift-* services files as,
[root@gluster-swift system]# sed -i s/User=swift/User=root/ openstack-swift-proxy.service openstack-swift-account.service openstack-swift-container.service openstack-swift-object.service openstack-swift-object-expirer.service
[root@gluster-swift system]# grep -ri "user=root" openstack-swift-*
openstack-swift-account.service:User=root
openstack-swift-container.service:User=root
openstack-swift-object-expirer.service:User=root
openstack-swift-object.service:User=root
openstack-swift-proxy.service:User=root
Start the gluster swift services before reboot as,
[root@gluster-swift system]# systemctl start openstack-swift-proxy.service openstack-swift-account.service openstack-swift-container.service openstack-swift-object.service openstack-swift-object-expirer.service
Start the gluster swift services at boot as,
[root@gluster-swift system]# systemctl enable openstack-swift-proxy.service openstack-swift-account.service openstack-swift-container.service openstack-swift-object.service openstack-swift-object-expirer.service
ln -s '/usr/lib/systemd/system/openstack-swift-account.service' '/etc/systemd/system/multi-user.target.wants/openstack-swift-account.service'
ln -s '/usr/lib/systemd/system/openstack-swift-container.service' '/etc/systemd/system/multi-user.target.wants/openstack-swift-container.service'
ln -s '/usr/lib/systemd/system/openstack-swift-object.service' '/etc/systemd/system/multi-user.target.wants/openstack-swift-object.service'
ln -s '/usr/lib/systemd/system/openstack-swift-object-expirer.service' '/etc/systemd/system/multi-user.target.wants/openstack-swift-object-expirer.service'
[root@gluster-swift system]# systemctl status openstack-swift-proxy.service openstack-swift-account.service openstack-swift-container.service openstack-swift-object.service openstack-swift-object-expirer.service
openstack-swift-proxy.service - OpenStack Object Storage (swift) - Proxy Server
Loaded: loaded (/usr/lib/systemd/system/openstack-swift-proxy.service; enabled)
Active: active (running) since Tue 2015-11-10 15:47:15 IST; 22min ago
Main PID: 1087 (swift-proxy-ser)
CGroup: /system.slice/openstack-swift-proxy.service
├─1087 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
└─1265 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
Nov 10 15:47:15 gluster-swift.redhat.com systemd[1]: Starting OpenStack Object Storage (swift) - Proxy Server...
Nov 10 15:47:15 gluster-swift.redhat.com systemd[1]: Started OpenStack Object Storage (swift) - Proxy Server.
Nov 10 16:07:31 gluster-swift.redhat.com systemd[1]: Started OpenStack Object Storage (swift) - Proxy Server.
Nov 10 16:08:07 gluster-swift.redhat.com systemd[1]: Started OpenStack Object Storage (swift) - Proxy Server.
openstack-swift-account.service - OpenStack Object Storage (swift) - Account Server
Loaded: loaded (/usr/lib/systemd/system/openstack-swift-account.service; enabled)
Active: active (running) since Tue 2015-11-10 16:07:31 IST; 2min 3s ago
Main PID: 3153 (swift-account-s)
CGroup: /system.slice/openstack-swift-account.service
├─3153 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
└─3174 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
Nov 10 16:07:31 gluster-swift.redhat.com systemd[1]: Starting OpenStack Object Storage (swift) - Account Server...
Nov 10 16:07:31 gluster-swift.redhat.com systemd[1]: Started OpenStack Object Storage (swift) - Account Server.
Nov 10 16:08:07 gluster-swift.redhat.com systemd[1]: Started OpenStack Object Storage (swift) - Account Server.
openstack-swift-container.service - OpenStack Object Storage (swift) - Container Server
Loaded: loaded (/usr/lib/systemd/system/openstack-swift-container.service; enabled)
Active: active (running) since Tue 2015-11-10 16:07:31 IST; 2min 3s ago
Main PID: 3154 (swift-container)
CGroup: /system.slice/openstack-swift-container.service
├─3154 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
└─3175 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
Nov 10 16:07:31 gluster-swift.redhat.com systemd[1]: Starting OpenStack Object Storage (swift) - Container Server...
Nov 10 16:07:31 gluster-swift.redhat.com systemd[1]: Started OpenStack Object Storage (swift) - Container Server.
Nov 10 16:08:07 gluster-swift.redhat.com systemd[1]: Started OpenStack Object Storage (swift) - Container Server.
openstack-swift-object.service - OpenStack Object Storage (swift) - Object Server
Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object.service; enabled)
Active: active (running) since Tue 2015-11-10 16:07:31 IST; 2min 3s ago
Main PID: 3155 (swift-object-se)
CGroup: /system.slice/openstack-swift-object.service
├─3155 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
└─3173 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
Nov 10 16:07:31 gluster-swift.redhat.com systemd[1]: Starting OpenStack Object Storage (swift) - Object Server...
Nov 10 16:07:31 gluster-swift.redhat.com systemd[1]: Started OpenStack Object Storage (swift) - Object Server.
Nov 10 16:08:07 gluster-swift.redhat.com systemd[1]: Started OpenStack Object Storage (swift) - Object Server.
openstack-swift-object-expirer.service - OpenStack Object Storage (swift) - Object Expirer
Loaded: loaded (/usr/lib/systemd/system/openstack-swift-object-expirer.service; enabled)
Active: active (running) since Tue 2015-11-10 16:07:31 IST; 2min 3s ago
Main PID: 3156 (swift-object-ex)
CGroup: /system.slice/openstack-swift-object-expirer.service
└─3156 /usr/bin/python /usr/bin/swift-object-expirer /etc/swift/object-expirer.conf
Nov 10 16:07:31 gluster-swift.redhat.com systemd[1]: Starting OpenStack Object Storage (swift) - Object Expirer...
Nov 10 16:07:31 gluster-swift.redhat.com systemd[1]: Started OpenStack Object Storage (swift) - Object Expirer.
Nov 10 16:08:07 gluster-swift.redhat.com systemd[1]: Started OpenStack Object Storage (swift) - Object Expirer.
Hint: Some lines were ellipsized, use -l to show in full.
Verify gluster swift services running as,
[root@gluster-swift ~]# ps -ef |grep swift
root 1098 1 0 17:25 ? 00:00:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
root 1099 1 0 17:25 ? 00:00:00 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
root 1100 1 0 17:25 ? 00:00:00 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
root 1101 1 0 17:25 ? 00:00:00 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
root 1102 1 0 17:25 ? 00:00:00 /usr/bin/python /usr/bin/swift-object-expirer /etc/swift/object-expirer.conf
root 1231 1101 0 17:25 ? 00:00:00 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf
root 1232 1100 0 17:25 ? 00:00:00 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf
root 1233 1099 0 17:25 ? 00:00:00 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf
root 1237 1098 0 17:25 ? 00:00:00 /usr/bin/python /usr/bin/swift-proxy-server /etc/swift/proxy-server.conf
SBR
Product(s)
Category
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.