What log files should I gather to troubleshoot a kickstart or manual installation issue with Anaconda?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 9
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 5

Issue

  • What log files will be helpful to look at when Anaconda fails in some way or another?
  • How do I gather log files from the installation environment?
  • Why is anaconda failing to install with a message about being unable to find certain items?
  • How can I debug an Anaconda issue?

Resolution

Prelude

Basic Log File Inspection

  • Switch to tty2 by pressing Ctrl-Alt-F2

  • Inspect and/or capture the following log files:

    • /tmp/anaconda-tb-*
    • /tmp/syslog
    • /tmp/*.log
      - /tmp/anaconda.log
      - /tmp/ifcfg.log
      - /tmp/program.log
      - /tmp/storage.log
      - /tmp/packaging.log
      - /tmp/yum.log
      - /tmp/dnf.librepo.log
      - /tmp/hawkey.log
      - /tmp/dbus.log
      - NOTE: There are some difference in the generated log file by RHEL version.

  • Basic troubleshooting of items necessary to perform installation can also be viewed from here:

    • To confirm storage devices are present:

      # fdisk -l
# cat /proc/partitions
# ls -alR /dev | less
# lvm lvscan

    • To test basic network connectivity:

      # ping 192.168.1.129 (destination ip address)
# ping myfqdn.com (to confirm networking and DNS, often required for installations via Satellite)

Capturing Extensive Info For Debugging

If providing information to Red Hat Support in order to gain assistance troubleshooting an issue, it might prove useful to capture additional info from the environment of the failed installation. To automate this task, a script can be used.

  • Save the following as a file called LogCapture on a system which the failed installation can ssh to:

    #!/bin/bash
cd /tmp; mkdir cmd-outputs
W() { local OutputFile=$(tr ' /' '_' <<<"$@"); $@ >cmd-outputs/$OutputFile; }
if [[ ! -f ks.cfg ]] && [[ ! -f /run/install/ks.cfg ]]; then echo No kickstart present >ks.cfg; fi
ls anaconda-tb-* &>/dev/null || kill -USR2 $(</var/run/anaconda.pid)
W cat /mnt/sysimage/root/anaconda-ks.cfg
W cat /run/install/ks.cfg 
W date
W dmesg
W dmidecode
W lspci -vvnn
W fdisk -cul
W parted -l
W ls -lR /dev
W dmsetup ls --tree
W lvm pvs
W lvm vgs
W lvm lvs
W cat /proc/partitions
W mount
W df -h
W cat /proc/meminfo
W cat /proc/cpuinfo
W ps axf
W lsof
W ip -s li
W ip a
W ip r
date=$(date +%F)
tar cvjf install-logs-$date.tbz *log cmd-outputs anaconda-tb-* ks.cfg
echo -e "\nFinished gathering data\nUpload /tmp/install-logs-$date.tbz to another system\n"

  • On the broken installation, switch to tty2 (Ctrl-Alt-F2) and then, if necessary, bring up networking
    Note: eth0 in the following examples should be replaced with the appropriate/desired interface, as shown by the ip link command

    • If there is a dhcp server on one of the system's NICs, one simple step is necessary to get on the network:

      # dhclient -v eth0

    • Static addressing is almost as simple, but as always, it requires knowing what address & netmask to use, e.g.:

      # ifconfig eth0 192.168.122.35/24 up

  • Use rsync or scp to grab the LogCapture script from the remote host it was downloaded to, e.g.:

    # rsync -v User@Remote:LogCapture /tmp

    Remember: rsync and scp both use ssh, so ensure the remote machine allows incoming ssh traffic from the halfway-installed system.

  • Run the LogCapture script on the system which failed installation in order to capture command outputs and tar them up with logs:

    # bash /tmp/LogCapture

  • Finally, use rsync or scp to get the generated /tmp/install-logs-*.tbz file off the broken install system and back to a remote system

    # rsync -v /tmp/install-logs-*.tbz User@Remote:

Issues networking with NetworkManager

  • If the issue relates to network management (such as IP address, subnet, routing, default gateway, DNS, or DHCP) and NetworkManager is used such as with RHEL7 and onwards, it would be helpful to have NetworkManager's logging level increased.

    • Have the %pre section of the Kickstart script write a file at /etc/NetworkManager/conf.d/logging.conf with the contents:
    [logging]
level=TRACE

    Example:

    %pre
mkdir -p /etc/NetworkManager/conf.d
cat >> /etc/NetworkManager/conf.d/logging.conf <<EOF
[logging]
level=TRACE
EOF
systemctl restart NetworkManager
%end

Additional Resources: For even more on Anaconda logging, including some tricks for when ssh isn't feasible, see Content from fedoraproject.org is not included.the official Anaconda project wiki page.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.