MTV Migrations fails as virt-v2v appliance runs out of memory

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform 4.x
  • Migration Toolkit for Virtualization 2.x

Issue

  • Conversion of Virtual Machine to OpenShift Virtualization fails with out of memory error.
  • oom-killer is invoked during migration of Virtual Machine to OpenShift Virtualization

Resolution

There is no resolution at this stage until the appliance memory size is officially configurable. But there are 2 workarounds:

Convert on RHEL and then upload to OpenShift Virtualization

Follow Converting a VMware vCenter Windows virtual machine to KVM and set the environment variable to a bigger size when running the command:

$ LIBGUESTFS_MEMSIZE=16384 virt-v2v -ic vpx://.....

Then use the OpenShift Console to upload the disks as PVCs or use virtctl to upload the disks from the command line. After that create the VM using the existing disks.

Use Gatekeeper to adjust the MTV virt-v2v Pod with more memory for the appliance.

If your subscription provides support for Gatekeeper, install the Gatekeeper Operator (with mutation enabled) and use these the Assigns objects below to modify the virt-v2v Pods on the migration target namespaces, the appliance will be created using twice the amount of memory specified by below in LIBGUESTFS_MEMSIZE, so the memory of appliance in this example will be 24GiB:

apiVersion: mutations.gatekeeper.sh/v1
kind: Assign
metadata:
  name: increase-virt-v2v-memory
spec:
  applyTo:
  - groups: [""]
    kinds: ["Pod"]
    versions: ["v1"]
  match:
    labelSelector:
      matchLabels:
        forklift.app: virt-v2v
    scope: Namespaced
    namespaces:
    - "target-namespace-of-vm"                          <----- replace this 
  location: "spec.containers[name:virt-v2v].env[name:LIBGUESTFS_MEMSIZE]"
  parameters:
    assign:
      value:
        name: LIBGUESTFS_MEMSIZE
        value: "12288"
---
apiVersion: mutations.gatekeeper.sh/v1
kind: Assign
metadata:
  name: increase-virt-v2v-limits
spec:
  applyTo:
  - groups: [""]
    kinds: ["Pod"]
    versions: ["v1"]
  match:
    labelSelector:
      matchLabels:
        forklift.app: virt-v2v
    scope: Namespaced
    namespaces:
    - "target-namespace-of-vm"                          <----- replace this 
  location: "spec.containers[name:virt-v2v].resources.limits.memory"
  parameters:
    assign:
      value: "32Gi"

NOTE: this does not work for VMs being migrated to target namespaces that start with openshift-*.

Root Cause

Diagnostic Steps

  • In the virt-v2v pod logs, some proccess related to the conversion triggers oom. Here is an example where it happens during xfs_repair:
[ 1604.204253] xfs_repair invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO|__GFP_COMP), order=0, oom_score_adj=0
...
[ 1604.217146] Mem-Info:
[ 1604.217333] active_anon:578436 inactive_anon:19702 isolated_anon:0
[ 1604.217333]  active_file:0 inactive_file:46 isolated_file:0
[ 1604.217333]  unevictable:0 dirty:0 writeback:0
[ 1604.217333]  slab_reclaimable:2325 slab_unreclaimable:8192
[ 1604.217333]  mapped:24510 shmem:64063 pagetables:1216
[ 1604.217333]  sec_pagetables:0 bounce:0
[ 1604.217333]  kernel_misc_reclaimable:0
[ 1604.217333]  free:13697 free_pcp:278 free_cma:0
[ 1604.220250] Node 0 active_anon:2313744kB inactive_anon:78808kB active_file:0kB inactive_file:184kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:98040kB dirty:0kB writeback:0kB shmem:256252kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:768000kB writeback_tmp:0kB kernel_stack:2672kB pagetables:4864kB sec_pagetables:0kB all_unreclaimable? yes Balloon:0kB
[ 1604.222711] Node 0 DMA free:10048kB boost:0kB min:276kB low:344kB high:412kB reserved_highatomic:0KB active_anon:4768kB inactive_anon:0kB active_file:0kB inactive_file:16kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1604.224720] lowmem_reserve[]: 0 2445 2445 2445 2445
[ 1604.225111] Node 0 DMA32 free:44740kB boost:0kB min:44776kB low:55968kB high:67160kB reserved_highatomic:0KB active_anon:2308976kB inactive_anon:78808kB active_file:0kB inactive_file:184kB unevictable:0kB writepending:0kB present:2604916kB managed:2504284kB mlocked:0kB bounce:0kB free_pcp:1112kB local_pcp:248kB free_cma:0kB
[ 1604.227266] lowmem_reserve[]: 0 0 0 0 0
[ 1604.227576] Node 0 DMA: 2*4kB (M) 1*8kB (M) 2*16kB (ME) 1*32kB (E) 2*64kB (ME) 1*128kB (E) 2*256kB (ME) 2*512kB (UE) 2*1024kB (ME) 1*2048kB (E) 1*4096kB (M) = 10064kB
[ 1604.228719] Node 0 DMA32: 195*4kB (UME) 323*8kB (UME) 286*16kB (UME) 140*32kB (UME) 53*64kB (UME) 15*128kB (UE) 8*256kB (U) 3*512kB (UM) 3*1024kB (UM) 2*2048kB (UE) 4*4096kB (UM) = 44868kB
[ 1604.230006] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 1604.230677] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 1604.231323] 64079 total pagecache pages
[ 1604.231635] 0 pages in swap cache
[ 1604.231904] Free swap  = 0kB
[ 1604.232134] Total swap = 0kB
[ 1604.232369] 655227 pages RAM
[ 1604.232608] 0 pages HighMem/MovableOnly
[ 1604.232915] 25316 pages reserved
[ 1604.233172] 0 pages cma reserved
[ 1604.233437] 0 pages hwpoisoned
[ 1604.233685] Tasks state (memory values in pages):
[ 1604.234057] [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[ 1604.234919] [    298]     0   298     8375      477      477        0         0    90112        0             0 systemd-udevd
[ 1604.235788] [    344]     0   344     1082       90       89        1         0    49152        0             0 dhcpcd
[ 1604.236615] [    432]     0   432    89668     1782     1767       15         0   110592        0             0 guestfsd
[ 1604.237446] [    817]     0   817   609201   555903   531455        0     24448  4542464        0             0 xfs_repair                                                                                                                 
[ 1604.238280] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=xfs_repair,pid=817,uid=0
[ 1604.239260] Out of memory: Killed process 817 (xfs_repair) total-vm:2436804kB, anon-rss:2125820kB, file-rss:0kB, shmem-rss:97792kB, UID:0 pgtables:4436kB oom_score_adj:0
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.