Additional and alternative steps for gathering a sosreport for a Red Hat Enterprise Linux support case

Issue

• Having problems or issues gathering a sosreport
• System crashes during boot up
• System hangs during boot up
• System otherwise fails to boot
• System cannot be booted, for example due to what might be a hardware failure
• Gathering a sosreport hangs or otherwise does not complete

Environment

• Red Hat Enterprise Linux 6,7

Resolution

• System is not booting
• Boot Crash: If the system is crashing or panicing on boot, see gathering a vmcore and sosreport for boot crash or panics,
• Boot Hang: If the system is hanging on boot, see gathering a vmcore and sosreport for boot time hangs
• Boot Fails: If the system otherwise fails to complete boot, for example during boot the message
  mount: error mounting /dev/root on /sysroot as extN: Invalid argument
  occurs, then see gathering a sosreport for non-booting systems
• SOS Hang: Gathering a sosreport hangs or otherwise is not completing, see selective sosreport module specification,
• Alternative Info: If above does not help, then see minimum alternative/manual information gathering

Gathering a vmcore and sosreport for boot crash or panics

First Steps

• Take and post a screen shot of the error messages or last messages printed on the console at boot time
  • If running rhgb (Red Hat Graphical Boot) option and you are at the splash/progress screen, try hitting the esc key (escape) to flip to the logging view.
  • If the above didn't work or there is no useful information present, then you will need to remove rhgb and reboot. Our full recommendation is to remove both rhgb and quiet.
    • See How to display more verbose boot-related messages during system startup. for details on removing rhgb and quiet boot options when booting.
    • Using the same steps as above add the following boot options: debug, printk, as well as
      • log_buf_len=8M: ensure that a log buffer length option is present with 8Mb buffer or larger, that is log_buf_len=8M, if not present of smaller buffer size then add/update this option
      • prink.time=1: ensure that a prink.time=1 option is present, if not, then add it.
• Gather a sosreport from rescue mode. Refer to the following documentation for further information.
  • "How to generate sosreport from the rescue environment"
• Please attach a copy of the current initramfs boot image file being used to boot to the case.
• Provide a list of any recent changes to the system or its attached parts (like storage switch or firmware upgrades)
• If a vmcore was created, use the rescue environment to retrieve the core to post to the case
• Is there any older version of the kernel installed on this system?
  • If so, are you able to boot with the older kernel?
    * If so, boot that kernel and gather a sosreport, post that along with a copy of the initramfs boot image used to boot this version of the kernel as well as the initramfs from associated with the newer version that does not complete booting.

Next Steps
It is highly likely that a vmcore will be needed for either crash/panics or hangs at boot time. If kdump has not been setup and tested on the system, follow the steps atHow to troubleshoot kernel crashes, hangs, or reboots with kdump on Red Hat Enterprise Linux to do so at the earliest possible time. There is also a This content is not included.Kdump Helper that can walk you through the steps in setting up kdump.

Gathering a vmcore and sosreport for boot time hangs

First Steps

• Take and post a screen shot of the error messages or last messages printed on the console at boot time
  • If running rhgb (Red Hat Graphical Boot) option and you are at the splash/progress screen, try hitting the esc key (escape) to flip to the logging view.
  • If the above didn't work or there is no useful information present, then you will need to remove rhgb and reboot. Our full recommendation is to remove both rhgb and quiet.
    • See How to display more verbose boot-related messages during system startup. for details on removing rhgb and quiet boot options when booting.
    • Using the same steps as above add the following boot options: debug, printk, as well as
      • log_buf_len=8M: ensure that a log buffer length option is present with 8Mb buffer or larger, that is log_buf_len=8M, if not present of smaller buffer size then add/update this option
      • prink.time=1: ensure that a prink.time=1 option is present, if not, then add it.
• Gather a sosreport from rescue mode. Refer to the following documentation for further information.
  • "How to generate sosreport from the rescue environment"
• Please attach a copy of the current initramfs boot image file being used to boot to the case.
• Provide a list of any recent changes to the system or its attached parts (like storage switch or firmware upgrades)
• If a vmcore was created, use the rescue environment to retrieve the core to post to the case
• Is there any older version of the kernel installed on this system?
  • If so, are you able to boot with the older kernel?
    * If so, boot that kernel and gather a sosreport, post that along with a copy of the initramfs boot image used to boot this version of the kernel as well as the initramfs from associated with the newer version that does not complete booting.

Next Steps
It is highly likely that a vmcore will be needed for either crash/panics or hangs at boot time. If kdump has not been setup and tested on the system, follow the steps atHow to troubleshoot kernel crashes, hangs, or reboots with kdump on Red Hat Enterprise Linux to do so at the earliest possible time. There is also a This content is not included.Kdump Helper that can walk you through the steps in setting up kdump.

Gathering sosreport on non-booting systems

If the system did not complete booting up correctly but did not crash, panic, or hang then follow the following instructions. An example of this scenario is the boot disk could not be found or the root file system failed to mount, or there were critical disks missing such that LVM volumes end up not being available during boot.

First Steps

• Take and post a screen shot of the error messages or last messages printed on the console at boot time
  • If running rhgb (Red Hat Graphical Boot) option and you are at the splash/progress screen, try hitting the esc key (escape) to flip to the logging view.
  • If the above didn't work or there is no useful information present, then you will need to remove rhgb and reboot. Our full recommendation is to remove both rhgb and quiet.
    • See How to display more verbose boot-related messages during system startup. for details on removing rhgb and quiet boot options when booting.
    • Using the same steps as above add the following boot options: debug, printk, as well as
      • log_buf_len=8M: ensure that a log buffer length option is present with 8Mb buffer or larger, that is log_buf_len=8M, if not present of smaller buffer size then add/update this option
      • prink.time=1: ensure that a prink.time=1 option is present, if not, then add it.
• Gather a sosreport from rescue mode. Refer to the following documentation for further information.
  • "How to generate sosreport from the rescue environment"
• Please attach a copy of the current initramfs boot image file being used to boot to the case.
• Provide a list of any recent changes to the system or its attached parts (like storage switch or firmware upgrades)
• If a vmcore was created, use the rescue environment to retrieve the core to post to the case
• Is there any older version of the kernel installed on this system?
  • If so, are you able to boot with the older kernel?
    * If so, boot that kernel and gather a sosreport, post that along with a copy of the initramfs boot image used to boot this version of the kernel as well as the initramfs from associated with the newer version that does not complete booting.

Next Steps
It is highly likely that a vmcore will be needed for either crash/panics or hangs at boot time. If kdump has not been setup and tested on the system, follow the steps atHow to troubleshoot kernel crashes, hangs, or reboots with kdump on Red Hat Enterprise Linux to do so at the earliest possible time. There is also a This content is not included.Kdump Helper that can walk you through the steps in setting up kdump.

Selective sosreport module specification

If the sosreport hangs or otherwise does not complete - for example sosreport ends up in D state for a very long time - then a selective sosreport run can be made.

First Steps

• Collect the partial sosreport information already gathered before sosreport hung:

  • How to gather data from a Red Hat Enterprise Linux system for troubleshooting if the sosreport process goes to hung state?

• Check there is enough free space within /tmp, if not you can use the --tmp-dir /path/directory option on the sosreport command line to redirect output to a filesystem that has enough free space.

• Re-run sosreport with additional logging to identify which plugin is running at hang time, then exclude it:

  • Why does the sosreport command hang?, these steps are also outlined in
  • What is a sosreport and how to create one in Red Hat Enterprise Linux 4.6 and later?

Minimum alternative/manual information gathering options

If sosreport use results in too many issues, then one alternative is to run a set of individual information gathering commands. These commands only gather very minimal information and are not a replacement for sosreport. The purpose of this set of commands is to at least gather some minimal level of information for the support case until such time as the cause of the sosreport hang/failure to complete has been addressed. Typically the issues plaguing sosreport will also be effecting other applications on the system.

See "Sosreport fails. What data should I provide in its place?" for the command details.

...