How to list OVN database contents with ovn-kubernetes in Red Hat OpenShift Container Platform 4.x?

Environment

Red Hat OpenShift Container Platform

• OCP v4.8 - OCP v4.13

Issue

How to list OVN database contents with ovn-kubernetes in Red Hat OpenShift Container Platform 4.x?

Resolution

Querying data from the leader pod

It is recommended to query up to date information directly from the OVN Raft leader.

Finding the leader pod

First, create the following bash function to make it easier to determine the leader pod:

function get_leader_pod {
  local socket=/var/run/ovn/ovnnb_db.ctl
  local db=OVN_Northbound
  if [ "$1" == "south" ]; then
    socket=/var/run/ovn/ovnsb_db.ctl
    db=OVN_Southbound
  fi 
  for f in $(oc -n openshift-ovn-kubernetes get pods -l app=ovnkube-master -o jsonpath="{.items[*].metadata.name}")
  do
    f_role=$(oc -n openshift-ovn-kubernetes exec "${f}" -c northd -- ovs-appctl -t ${socket} cluster/status ${db} | grep -E "^Role: ")
    echo "${f_role}" | grep -q leader && { echo ${f}; return $(/bin/true); }
  done
  return $(/bin/false)
}

For convenience, add this function to ~/.bashrc.
The function will determine the Northbound Raft leader by default. In order to determine the Southbound leader, run get_leader_pod south.

One liner to find leader.

for OVNMASTER in $(oc -n openshift-ovn-kubernetes get pods -l app=ovnkube-master -o custom-columns=NAME:.metadata.name --no-headers); \
   do echo "········································" ; \
   echo "· OVNKube Master: $OVNMASTER ·" ; \
   echo "········································" ; \
   echo 'North' `oc -n openshift-ovn-kubernetes rsh -Tc northd $OVNMASTER ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound | grep ^Role` ; \
   echo 'South' `oc -n openshift-ovn-kubernetes rsh -Tc northd $OVNMASTER ovn-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound | grep ^Role`; \
   echo "····················"; \
   done

Northbound database

Get the leader ovnkube-master pod and list contents of the NB database:

POD=$(get_leader_pod)
oc exec -n openshift-ovn-kubernetes -it $POD -- ovn-nbctl show

Southbound database

Get the leader ovnkube-master pod and list contents of the SB database:

POD=$(get_leader_pod south)
oc exec -n openshift-ovn-kubernetes -it $POD -- ovn-sbctl show

Querying data from other pods

The --no-leader-only flag allows read only transactions against any server in the OVN cluster. Thus it is particularly useful for querying and analysis of stale data.

List all ovnkube-master pods:

$ oc get pods -n openshift-ovn-kubernetes  -o wide -l app=ovnkube-master -o name
pod/ovnkube-master-2d762
pod/ovnkube-master-tmhfp
pod/ovnkube-master-x2v6x

Pick any of the pods and run either the ovn-sbctl or ovn-nbctl command with the --no-leader-only parameter:

POD=ovnkube-master-2d762 
oc exec -n openshift-ovn-kubernetes -it $POD -c northd -- ovn-sbctl --no-leader-only show  
oc exec -n openshift-ovn-kubernetes -it $POD -c northd -- ovn-nbctl --no-leader-only show  

Along the same lines, one can also query the cluster status on each individual node to asses the cluster member's health and synchronization status:

POD=ovnkube-master-2d762
oc exec -n openshift-ovn-kubernetes -it $POD -c northd -- ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound
oc exec -n openshift-ovn-kubernetes -it $POD -c northd -- ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound 

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.