My system shuts down before completing the boot process

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 7 and later

Issue

  • System shuts down before completing the boot process

  • Attempting to boot results in the following:

    Boot failure messages 1

    Boot failure messages 2

Resolution

Follow the procedure described in the Diagnostic Steps section.
If this is a match, proceed further, otherwise open a case on the Customer Portal referencing this solution.

The solution is to enlarge the /var/log/audit file system or clean ancient audit logs.
Usually the issue happens after hardening for CIS, we hence recommend that you discuss this with your Security Team before taking any action.

Root Cause

CIS hardening requires that the system gets shut down if /var/log/audit space is low, in order to avoid losing audit logs which may be used for forensics.

When this happens, the system is brought down but no message is printed on the console, making it very hard to troubleshoot.
This content is not included.BZ 2207869 - When audit has "admin_space_left_action = halt", the system is halted with no visible message on the console has been filed on RHEL9 to enhance this.

Diagnostic Steps

  1. Boot the system, halting at Grub menu to append audit=0 to the kernel command line

  2. Check the amount of space consumed by the audit logs

    $ grep -e audit -e Mounted df
Filesystem              1K-blocks     Used Available Use% Mounted on
/dev/mapper/rootvg-auditlv01       1038336  987208     51128  96% /var/log/audit

    In the example above, audit logs are stored on a dedicated partition.

  3. Check the audit configuration

    $ grep ^admin_space_left /etc/audit/auditd.conf
admin_space_left = 50
admin_space_left_action = HALT

    In the example above, audit is configured to shut down the system if space is lower than 50MB, which is the case, from step 1.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.