What are unsigned modules and how do they appear in a kernel backtrace?

Solution Unverified - Updated

Environment

  • Red Hat Enterprise Linux 9
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 4
  • Red Hat Enterprise Linux 3

Issue

  • What does it mean for a kernel to contain "unsigned modules" and how can we tell they are present?

Resolution

Red Hat signs all kernel modules that it ships. The signature is a cryptographic key, which the kernel uses to verify modules when loaded.

Unsigned modules have been recompiled by the customer or (more likely) are provided by a party other than Red Hat and do not include a cryptographic key.

There are two different methods used by Red Hat to sign modules. Unsigned modules will not have this information present.

  • RHEL 7 and later kernels, each of the following commands can be used to determine if a signature is present.

     # modinfo lpfc
 filename:       /lib/modules/4.18.0-372.26.1.el8_6.x86_64/kernel/drivers/scsi/lpfc/lpfc.ko.xz
 version:        0:14.0.0.4
 :
 vermagic:       4.18.0-372.26.1.el8_6.x86_64 SMP mod_unload modversions 
 sig_id:         PKCS#7
 signer:         Red Hat Enterprise Linux kernel signing key
 sig_key:        67:C6:8A:84:D4:0D:B0:66:D0:54:B5:2F:D0:9F:EA:A2:1D:EE:DD
 sig_hashalgo:   sha256
 signature:      A1:18:22:A5:10:2C:DD:3A:9D:FE:55:ED:F5:0B:87:25:D1:5F:A2:15:
 :
	                    7F:78:6A:F8

 # modinfo -F signer lpfc
 Red Hat Enterprise Linux kernel signing key

 # unxz lpfc.ko.xz
 # strings lpfc.ko | grep "~Module signature appended~"
 g~Module signature appended~

 >> The following unsigned.ko module does not have the expected signatures or tell-tales as shown above.

 # modinfo unsigned
 filename: /lib/modules/4.18.0-372.26.1.el8_6.x86_64/extra/unsigned.ko
 version: 0.6.3-1.2
 :
 vermagic:       4.18.0-372.26.1.el8_6.x86_64 SMP mod_unload modversions 
 :

 # modinfo -F signer unsigned
 #

 # strings unsigned.ko  | grep "~Module signature appended~"
 #

  • RHEL 6 and earlier kernels, an ELF .module_sig section is utilized to hold the signature

     # objdump -s lpfc.ko | grep .module_sig -A 12
 :
 Contents of section .module_sig:
  0000 883f0305 005d5a9a c688ff63 04ebc6e5  .?...]Z....c....
  0010 001102f9 48009e39 49ebb835 488cb726  ....H..9I..5H..&
  0020 477731c7 d0812c9f 1c188300 9f632b8f  Gw1...,......c+.
  0030 e096a614 4cb8c580 22230104 83591701  ....L..."#...Y..
  0040 4e 
                                  N
 # readelf -x '.module_sig' lpfc.ko
 Hex dump of section '.module_sig':
  0x00000000 883f0305 005d5a9a c688ff63 04ebc6e5 .?...]Z....c....
  0x00000010 001102f9 48009e39 49ebb835 488cb726 ....H..9I..5H..&
  0x00000020 477731c7 d0812c9f 1c188300 9f632b8f Gw1...,......c+.
  0x00000030 e096a614 4cb8c580 22230104 83591701 ....L..."#...Y..
  0x00000040 4e                                  N

 >> unsigned modules do no have the .module_sig ELF section present (or it contains bad data)

 # objdump -s unsigned.ko | grep .module_sig -A 12
 #

 # readelf -x '.module_sig' unsigned.ko
 readelf: unsigned.ko: Warning: Section '.module_sig' was not dumped because it does not exist!

In addition to the above, RHEL 7 and later kernels will also flag the unsigned module being present in /proc/sys/kernel/tainted. The value is a bit mask. Moreover, kernels will display information about unsigned modules in a couple different ways.

Unsigned Modules Displayed in Backtraces

Backtraces printed in the logs when the kernel Oopses or Panics all contain a section listing kernel modules that are present and whether they are unsigned.'(U)'nsigned. In this case the system is likely running a third party proprietary stack of infiniband drivers.

  • RHEL 7 and later, taint E (unsigned module) and other module taing flags included:

Modules linked in: udp_diag unix_diag af_packet_diag netlink_diag tcp_diag inet_diag ebtable_filt
er ebtables devlink overlay(T) scini(POE) 8021q garp mrp bonding vxlan ip6_udp_tunnel udp_tunnel openvswitch nf_nat_ipv6 nf_nat_ipv4 nf_nat nls_utf8 isofs nf_log_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6table_raw ip6_tables iptable_raw nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_comment xt_multiport xt_conntrack iptable_filter skx_edac nfit libnvdimm intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass pcspkr ses enclosure sg joydev mei_me mei lpc_ich hpilo hpwdt ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter nf_conntrack br_netfilter bridge stp llc ip_tables xfs dm_service_time sd_mod crc_t10dif crct10dif_generic
  • RHEL 6 and earlier, '(U)' output to flag unsigned modules:

Modules linked in: ipmi_si mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler
dell_rbu rdma_ucm(U) ib_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) bonding ib_ipoib(U) ib_cm(U) ib_sa(U) ipv6 
ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) mlx4_ib(U) ib_mthca(U) ib_mad(U) ib_core(U) 
power_meter sg shpchp tg3 mlx4_en(U) mlx4_core(U) dcdbas sb_edac edac_core iTCO_wdt iTCO_vendor_support 
ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif ahci wmi megaraid_sas dm_mirror dm_region_hash 
dm_log dm_mod [last unloaded: ipmi_si]

Taint Mask Displayed

While the (U) syntax is used within module listing output for backtraces in RHEL 6 and earlier, the output of /proc/sys/kernel/tainted uses 'E' to flag for unsigned modules starting in RHEL7. The interpretation of the full tainted mask will appear in the dmesg and messages files simiilar to the following where the 'P' (proprietary module loaded), 'O' (out of tree module loaded), and 'E' (unsigned module) taint flags get set if so defined within the running kernel.

This is only be present in RHEL7 and later as the TAINT_UNSIGNED_MODULE (bit13) was not added until that versions release. [FN.1] 


var/log/dmesg:[   14.463869] emcp: module verification failed: signature and/or required key missing - tainting kernel
var/log/dmesg.kernel:[   14.455500] emcp: loading out-of-tree module taints kernel.
var/log/dmesg.kernel:[   14.455520] emcp: module license 'Proprietary' taints kernel.
var/log/dmesg.kernel:[   14.455524] Disabling lock debugging due to kernel taint
var/log/dmesg.kernel:[   14.463869] emcp: module verification failed: signature and/or required key missing - tainting kernel
var/log/dmesg.kernel:[1560475.896997] CPU: 27 PID: 8147  Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-1160.21.1.el7.x86_64 #1/pre>

Determination of Module Tainting

While the easiest way to tell if a loaded module tainted the kernel is to look at the dmesg and messages event files. This works for all versions.

As of RHEL 7 and later, determing if a loaded module taints the kernel can be easily checked by another means:

# grep -Hv "zzz" /sys/modules/*/taint | grep -v ":$"
/sys/module/abcl/taint:POE
/sys/module/abccommon/taint:POE
/sys/module/abcfs/taint:POE

Footnotes

  • There was a TAINT_UNSIGNED_MODULE (bit06) flag added to RHEL5 -- but it was only used in relation to output of module listings at oops and panic time. It was not displayed as part of taint flag output. This flag was removed as of RHEL 6.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.