Memory hot-plug failed and 'unaligned hotplug range' error logged by the kernel

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux
  • A virtual machine running on a VMware hypervisor

Issue

  • Memory hot-plug failed.
  • unaligned hotplug range error is logged by the kernel.
  • Can the size of the memory block changed?

Resolution

  • Try to hotplug memory in integer multiples of block size reported by the kernel in Memory block size log line.

  • The memory block size can be obtained via the lsmem command:

       # lsmem 
   RANGE                                  SIZE  STATE REMOVABLE  BLOCK 
   0x0000000000000000-0x0000000097ffffff  2.4G online       yes   0-18 
   0x0000000100000000-0x000000085fffffff 29.5G online       yes 32-267 

   Memory block size:       128M 
   Total online memory:    31.9G 
   Total offline memory:      0B
  • In the above example, the block size is 128 MB, so it is allowed to add 1024 MB (128 MB × 8), but a hot-add of 1088 MB (128 MB × 8.5) would fail.

Root Cause

  • Relevant kernel logs:

    [    0.125429] x86/mm: Memory block size: 2048MB
(...)
[1247984.219210] Block size [0x80000000] unaligned hotplug range: start 0x1280000000, size 0xec0000000
[1247984.219218] acpi PNP0C80:04: add_memory failed
[1247984.219334] acpi PNP0C80:04: acpi_memory_enable_device() error
[1247984.219357] acpi PNP0C80:04: Enumeration failure

  • Memory hotplug failed in check_hotplug_memory_range() function:

    file mm/memory_hotplug.c:
1020 static int check_hotplug_memory_range(u64 start, u64 size)
1021 {
1022         /* memory range must be block size aligned */
1023         if (!size || !IS_ALIGNED(start, memory_block_size_bytes()) ||
1024             !IS_ALIGNED(size, memory_block_size_bytes())) {
1025                 pr_err("Block size [%#lx] unaligned hotplug range: start %#llx, size %#llx",
1026                        memory_block_size_bytes(), start, size);
1027                 return -EINVAL;
1028         }
1029 
1030         return 0;
1031 }

  • The check_hotplug_memory_range() function checks the alignment of the start address and the size of hotplugged memory block. Let's examine which condition failed:

    block size = 0x0080000000 = 2147483648 = 2048 MB
    start = 0x1280000000 = 79456894976
    size = 0x0ec0000000 = 63350767616 = 60416 MB

    • Alignment can be verified by dividing by the block size. If the result is an integer, the value is aligned.

      • Hotplug memory start address is aligned:
        79456894976 ÷ 2147483648 = 37

      • Hotplug memory size is not aligned and it caused memory hotadd failure:
        63350767616 ÷ 2147483648 = 29.5

  • The valid memory hotplug sizes for this example, which are close to the requested 60416 MB, are 59392 MB (29 × 2048 MB) or 61440 MB (30 × 2048 MB)

  • Below the function which calculates the size of memory block:

    file arch/x86/mm/init_64.c:
1431 static unsigned long probe_memory_block_size(void)
1432 {
1433         unsigned long boot_mem_end = max_pfn << PAGE_SHIFT;
1434         unsigned long bz;
1435 
1436         /* If memory block size has been set, then use it */
1437         bz = set_memory_block_size;
1438         if (bz)
1439                 goto done;
1440 
1441         /* Use regular block if RAM is smaller than MEM_SIZE_FOR_LARGE_BLOCK */
1442         if (boot_mem_end < MEM_SIZE_FOR_LARGE_BLOCK) {
1443                 bz = MIN_MEMORY_BLOCK_SIZE;
1444                 goto done;
1445         }
1446 
1447         /* Find the largest allowed block size that aligns to memory end */
1448         for (bz = MAX_BLOCK_SIZE; bz > MIN_MEMORY_BLOCK_SIZE; bz >>= 1) {
1449                 if (IS_ALIGNED(boot_mem_end, bz))
1450                         break;
1451         }
1452 done:
1453         pr_info("x86/mm: Memory block size: %ldMB\n", bz >> 20);
1454 
1455         return bz;
1456 }

  • In case of systems with less than MEM_SIZE_FOR_LARGE_BLOCK (64 GB) the system will use blocks of MIN_MEMORY_BLOCK_SIZE (128 MB) size. If the system has more memory than MEM_SIZE_FOR_LARGE_BLOCK then the kernel will find the largest possible memory block and use it.

  • The memory block size is determined by a heuristic that considers the layout of the underlying hardware & cannot be changed from the OS side manually.

Diagnostic Steps

  • Check the kernel log for lines like this:
    [1247984.219210] Block size [0x80000000] unaligned hotplug range: start 0x1280000000, size 0xec0000000
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.