Understanding why a legacy SysV service on RHEL7 gets terminated too early during system shutdown

Updated 11 Sept 2017

Introduction

The systemd init daemon has built-in support for legacy SysV initscripts. For every initscript a corresponding systemd unit is generated on the fly during the system boot. This generated unit is a wrapper around the legacy script. Additional dependencies or resource limits can be applied using configuration drop-ins. For more information on drop-ins, see How to use drop-ins.

Considerations about `su`

If an initscript contains the su command, this command leads to the creation of a new Pluggable Authentication Module (PAM) session.
On Red Hat Enterprise Linux (RHEL) 7, processes of every PAM session run under a corresponding systemd scope unit: a scope unit is a transient unit type used to manage groups of the externally created process.

Scope units are not configured through unit files, but are created using the DBus API from within the pam_systemd PAM module.
A process that was initially part of the systemd unit corresponding to the initscript is after a call to su migrated to a newly created scope unit.

Assignment of a process to a particular cgroup influences the order of termination of the processes during system shutdown.
The processes running under scope units are terminated earlier than the processes running under systemd units.

Consequently, if the legacy SysV service initscript contains the su command, the service will be terminated early during system shutdown.

Resolution

To avoid early shutdown, adapt the initscript by replacing calls to su by calls to runuser, a binary provided by the util-linux package.
Additionally, make sure that runuser is not used with the -, -l or --login flag.

Indeed, when starting the shell as a login shell, the command will also lead to the creation of a new Pluggable Authentication Module (PAM) session.
Note that the command parameters for runuser are compatible with the parameters for su.

Example: installing the problematic SysV service

The attached archive example.zip provides an example legacy SysV service.

Prerequisite

Download the attached archive example.zip. Decompress this archive to the example folder. This folder contains four files:

deploy.yml
example-app
example-app-init
Vagrantfile

Install the vagrant package to be able to build a virtual machine from the Vagrantfile file using the vagrant up command. Refer to Content from www.softwarecollections.org is not included.Vagrant 1.8 by Software Collections for information on how to install the vagrant package.
Install the ansible package to be able to deploy the example-app application on the newly created virtual machine. Refer to This content is not included.Install Ansible on RHEL for information on how to install the ansible package.

Diagnostic steps

Change directory to the example folder
Run the vagrant up command

This command looks into the Vagrantfile file and creates a virtual machine with the example application named example-app.
Run the vagrant ssh -c "systemctl status example-app.service" command

By running this command, you will connect to the newly created virtual machine and you can examine the status of the example-app service.
Observe that no process tree is displayed in the systemctl output
Run the vagrant ssh -c "systemd-cgls --all" command
Note that example-app process is not running under the example-app.service service unit and this service unit does not contain any processes.

Example: fixing the problematic SysV service

Prerequisite

Change the example-app-init initscript file by replacing previously used su command by the runuser command as follows:

# BAD
#(su $user -c "$exec " &) &

# GOOD
(runuser $user -c "$exec " &) &

Diagnostic steps

Change directory to the example folder
Run the vagrant destroy && vagrant up command

By running this command, you will redeploy the virtual machine with the example application named example-app.
Run the vagrant ssh -c "systemctl status example-app.service" command

By running this command, you will connect to the redeployed virtual machine and you can examine the status of the example-app service.
You can observe a process tree displayed in systemctl output
Run the vagrant ssh -c "systemd-cgls --all" command
Note that the example-app process is now running under the example-app.service service unit.