Recommended practice to follow before Openshift SDN network plugin migration to OVNKubernetes plugin.

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.12
    • 4.13
    • 4.14
    • 4.15

Issue

  • What are all things one needs to check before one starts migration from OpenshiftSDN to OVNKubernetes network plugin?
  • What pre-checks need to be done before one starts OpenshiftSDN to OVNK migration?

Resolution

One can proactively check the following in the environment.

  • A cluster installed on infrastructure configured that is capable of the OVN-Kubernetes network plugin.

  • The cluster is in a known good state, without any errors. All operators must be in a healthy state, nodes must be ready.

  • Ensure cluster is not in middle of upgrade. All nodes in the cluster must be on the same version.

  • Ensure there are no IP conflicts for openshift cluster nodes in the network.

  • Take the This page is not included, but the link has been rewritten to point to the nearest parent document.etcd backup of your cluster.

  • Migration takes care of MTU, i.e MTU in SDN set to 1450, will set to 1400 in OVN.

  • Ensure no nodes have any duplicate system ID in the /etc/openvswitch/system-id.conf.

  • On all cloud platforms, on-prem platforms after updating software, a security group rule must be in place to allow UDP packets on port 6081 for all nodes, for on-prem/baremetal platforms make sure UDP port 6081 must be opened in the firewall on priority.

  • OVN-Kubernetes, the default network provider in OpenShift Container Platform 4.14 and later versions, uses the following IP address ranges internally: 100.64.0.0/16, 169.254.169.0/29, 100.88.0.0/16, fd98::/64, fd69::/125, and the fd97::/64. If your cluster uses OVN-Kubernetes, do not include any IP address ranges in any other CIDR definitions in your cluster.

  • In case, where the NODEIP_HINT param is used to point the secondary interface IP for configuring the OCP control plane nodes or any machine config is used for achieving the same , then make sure that ovn interface must have a default gateway (until bug This content is not included.OCPBUGS-31591 is released) in order to work properly with OVNK plugin. To examine the default gateway for br-ex, examine the kernel routing table from a node: 

sh-5.1# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.123.1   0.0.0.0         UG        0 0          0 br-ex

  • If there is no NODEIP_HINT configured, ovn will set up its bridge i.e br-ex on default interface.

  • When a cluster depends on static routes and/or routing policies in the host network so that pods can reach some destinations, one needs to set routingViaHost to true and ipForwarding to Global in gatewayConfig during migration (to offload routing decision to host kernel).

    • In order to achieve this during the SDN to OVN Kubernetes migration, you can perform the following patch at the same time that you customize MTU or IPv4 subnet during migration (at the time of writing this solution, it is the step 7 of the offline migration documentation) 

      oc patch Network.operator.openshift.io cluster --type=merge \
  --patch '{
    "spec":{
      "defaultNetwork":{
        "ovnKubernetesConfig":{
          "gatewayConfig": {
            "ipForwarding": "Global",
            "routingViaHost": true 
    }}}}}'

  • If running OpenShift Data Foundations (ODF), refer to this kcs for health checking prior to migration

  • Please open a proactive case prior to your migration for Red Hat assistance clarifying that it is for a OpenShift SDN to OVN-Kubernetes network plugin migration, and (as applicable) mention if using ODF storage for additional validation of storage pools. Include a must-gather from the cluster in the case.

Diagnostic Steps

This loop can be run pre and post migration.
Output with pre migration can be use as reference and can be useful to compare it with post migration results. 

for podIP in $(oc get pod -l dns.operator.openshift.io/daemonset-dns=default -o wide --no-headers -n openshift-dns| awk '{print $6}') ; do for nodes in $(oc get nodes --no-headers | awk '{print $1}') ; do echo Querying from $nodes to dns pod $podIP ; oc debug node/$nodes -- chroot /host dig @$podIP -p 5353 kubernetes.default.svc.cluster.local. +short 2>/dev/null ; echo ; done ; done

To check the webhook configuration:

oc get mutatingwebhookconfiguration

oc get validatingwebhookconfiguration
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.