How to validate that Egress IP is working/assigned properly in OpenShift 4.x

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (OCP) 4.x
  • OVN-Kubernetes network plugin
  • OpenShift SDN network plugin

Issue

  • EgressIP is not visibly assigned to nodes, how to confirm that it is working?
  • How to test EgressIP (EIP) on ovn-kubernetes CNI?
  • How to test EgressIP (EIP) on SDN CNI?
  • Run a local validation check that traffic is using egress to connect externally?
  • How can I be sure that my egress IP address is assigned to a node and that my pods are using it?
  • How to troubleshoot intermittent connectivity issue with EgressIP ?

INDEX

Resolution

Testing Egress configuration if using OVN-Kubernetes on OpenShift 4.14+:

Link to documentation

In 4.14, OVN-Kubernetes was updated to a new distributed architecture, see Root Cause below. As a result, we are primarily interested in the OVN status of two nodes: the CLIENT node where the pod using egress IP resides, and the HOST node where Egress IP is assigned. (Named arbitrarily, but to make it easier to follow).

  1. Start by determining your target nodes for the test. In the below example, our CLIENT node is worker-0 and our HOST node is worker-1. We need to determine the IP for the client pod, the worker nodes IP's, and the egress IP:
$ oc -n test get pod -o wide

NAME                       READY   STATUS    RESTARTS   AGE   IP             NODE                     NOMINATED NODE   READINESS GATES
ruby-ex-5997c4c5fc-svgqh   1/1     Running   0          16s   10.135.1.179   worker-0.ocp.example.net <none>           <none>


$ oc get eip

NAME        EGRESSIPS         ASSIGNED NODE              ASSIGNED EGRESSIPS
egressip1   192.168.194.250   worker-1.ocp.example.net   192.168.194.250

$ oc get node worker-0.ocp.example.net worker-1.ocp.example.net -o 'custom-columns=NAME:metadata.name,INTERNAL-IP:.status.addresses[?(.type=="InternalIP")].address'

NAME                       INTERNAL-IP
worker-0.ocp.example.net   192.168.194.200
worker-1.ocp.example.net   192.168.194.41
  1. Next we need to determine which pods are relevant to our investigation in the namespace: openshift-ovn-kubernetes
$ oc get pod -o wide | awk {'print $1, $7'} | grep -E 'worker-0|worker-1'

ovnkube-node-btkrd worker-1.ocp.example.net
ovnkube-node-rjkv2 worker-0.ocp.example.net
  1. [HOST NODE] - EgressIPs are implemented via Source Network Adress Translation (SNAT) in the internal OVN northbound database. We can find the SNATs entries that match the pod IP in the ovnkube-node pod on the HOST node. There should be a SNAT entry for the podIP: (example below, assuming pod IP is 10.135.1.179):
$ oc -n openshift-ovn-kubernetes exec -ti ovnkube-node-btkrd -c northd -- ovn-nbctl find nat logical_ip="10.135.1.179"

_uuid               : 12d0b4a4-0421-4575-aed0-708ae0316f9e
allowed_ext_ips     : []
exempted_ext_ips    : []
external_ids        : {name=egressip1}
external_ip         : "192.168.194.250"
external_mac        : []
external_port_range : ""
gateway_port        : []
logical_ip          : "10.135.1.179"
logical_port        : k8s-worker-1.ocp.example.net
options             : {stateless="false"}
type                : snat
  1. [HOST NODE] - We also check on the HOST node that the NAT entry above is attached to the gateway router of the node (named GR_${NODE_NAME}), because it is on the list of the nat field:
# Run the below command to find any routers that have the NAT entry above in its "nat" field (replace the UUID of the example with the one you obtained from your NAT entry). The output must show the gateway router (named `GR_${NODE_NAME}`) like in the example below, so if it is empty or it shows a different router, there is a problem (or typo):

$ oc -n openshift-ovn-kubernetes exec -ti ovnkube-node-btkrd -c northd -- ovn-nbctl --column name,nat find logical_router 'nat{>}[12d0b4a4-0421-4575-aed0-708ae0316f9e]'

name                : GR_worker-1.ocp.example.net
nat                 : [12d0b4a4-0421-4575-aed0-708ae0316f9e, 174b293c-81ad-42af-a05f-298d40f810c5, 17de8148-3448-4c32-8c47-623944ed65c4, 37e160d3-81db-4f41-9079-cc10acefbb0f, 68abd805-e5ce-4623-a3ab-79f2005438ec, 7714a021-e80c-4a33-92e6-039d7fee9557, 85f3fcb9-eb96-4e0e-b0ba-cd3446ff2f66, 8d65c467-3e0d-466e-b492-53a5a4dbf932, 99b24fb5-99f7-4656-a906-057875d52dd6, b2015477-6586-48d3-ad46-082f6eff2678, cbe59fa3-f4e0-4980-b2a9-4102367685b8, e6a4496e-8ad0-48f3-a070-b9df5ddc464a]
  1. [CLIENT NODE] - Check also the logical_router_policy of the pod to confirm it's affinity and reroute rule with an egressIP. The command should be run on the ovnkube-node pod on the CLIENT node (the one hosting our egress pod), replacing the IP address of the pod.
#The following command checks the logical_router_policy for a pod (this is responsible for forwarding the pod traffic to the egress node), there should be one entry per pod IP. Note the reroute action for the logical router, sending to an internal address for redirect to our target HOST node for egressIP SNAT handling on arrival:

$ oc -n openshift-ovn-kubernetes exec -ti ovnkube-node-rjkv2 -c northd -- ovn-nbctl find logical_router_policy match="\"ip4.src == 10.135.1.179\""

_uuid               : a14b0f81-47d0-48f3-966c-9ef81222081b
action              : reroute
external_ids        : {name=egressip1}
match               : "ip4.src == 10.135.1.179"
nexthop             : []
nexthops            : ["100.88.0.6"]
options             : {}
priority            : 100

# Here, the 100.88.0.6 IP is the IP of the worker 1(egress node) logical switch port on the transit switch, which basically means that packets would go to worker1 through the interconnection of the per-node OVN instances
  1. [CLIENT NODE] - Check also on the CLIENT node that the logical router policy above is attached to the ovn_cluster_router, as expected
# This command looks for any logical router that includes the routing policy above on its `policies` field. Replace the UUID from the example with the one you obtained from the logical_router_policy above. It must show the ovn_cluster_router like in the example below. If it shows no router or different routers, then there is a problem (or typo)

$  oc -n openshift-ovn-kubernetes exec -ti ovnkube-node-rjkv2 -c northd -- ovn-nbctl --column name,policies find logical_router 'policies{>}[a14b0f81-47d0-48f3-966c-9ef81222081b]'

name                : ovn_cluster_router
policies            : [3175017e-356c-45b6-a86d-89bb51745e81, a14b0f81-47d0-48f3-966c-9ef81222081b, c5712b1d-d0a0-421f-bfae-ebdb45e79ab2, cbdc3fef-be90-4789-a0f2-fb16bc7e3274, f04f6199-423e-4e9d-8fe5-74dfcd090106]
  1. [CLIENT & HOST] - Check also that the 100.88.0.6 is the right IP for the routing policy. You should do this check on both the CLIENT and the HOST nodes:

For the CLIENT node

# In the CLIENT node, the transit switch for the HOST node should have that address and be attached to the transit switch

$ oc -n openshift-ovn-kubernetes exec -ti ovnkube-node-rjkv2 -c northd -- ovn-nbctl lsp-list transit_switch

070f6df5-c9ba-4e90-b64c-88690a9e78f9 (tstor-ctlplane-0.ocp.example.net)
b32c50ab-8bf9-4c2f-bbdf-6dfd2ff4ce93 (tstor-ctlplane-1.ocp.example.net)
59782184-6095-4637-952a-abecd7affa44 (tstor-ctlplane-2.ocp.example.net)
d3b947bb-8585-48e1-81d8-de860a72dc8d (tstor-worker-0.ocp.example.net)
cadc2002-4895-449b-b54f-092b3b7badb8 (tstor-worker-1.ocp.example.net) ## It is attached to transit switch

$ oc -n openshift-ovn-kubernetes exec -ti ovnkube-node-rjkv2 -c northd -- ovn-nbctl lsp-get-addresses cadc2002-4895-449b-b54f-092b3b7badb8

0a:58:64:58:00:06 100.88.0.6/16 ## It has the right addresses

For the HOST node


# In the HOST node, it must be the IP of the logical router port that is attached to both the ovn_cluster_router and the logical switch port that is on the transient switch

# First, we find the logical router port and check its address

$ oc -n openshift-ovn-kubernetes exec -ti ovnkube-node-btkrd -c northd -- ovn-nbctl --column _uuid,mac,name,networks find logical_router_port name=rtots-worker-1.ocp.example.net

_uuid               : 646c6e4f-dc29-49c7-823b-60c4931a82af
mac                 : "0a:58:64:58:00:06"
name                : rtots-worker-1.ocp.example.net
networks            : ["100.88.0.6/16"]

# Now, we check if it is attached to the ovn_cluster_router

$ oc -n openshift-ovn-kubernetes exec -ti ovnkube-node-btkrd -c northd -- ovn-nbctl lrp-list ovn_cluster_router

636c1bd1-d734-4c87-a30f-bac41de95b0a (rtoj-ovn_cluster_router)
b7731a06-c769-491e-a954-99a1d32ba672 (rtos-worker-1.ocp.example.net)
646c6e4f-dc29-49c7-823b-60c4931a82af (rtots-worker-1.ocp.example.net) ## Here it is

# Now, we check its peer logical switch port

$  oc -n openshift-ovn-kubernetes exec -ti ovnkube-node-btkrd -c northd -- ovn-nbctl --column _uuid,external_ids,name,options,type find logical_switch_port 'options:router-port=rtots-worker-1.ocp.example.net'

_uuid               : 615dc47f-32a1-43a0-8708-ea4db1c1f7fb
external_ids        : {node=worker-1.ocp.example.net}
name                : tstor-worker-1.ocp.example.net
options             : {requested-tnl-key="6", router-port=rtots-worker-1.ocp.example.net}
type                : router

# And we check if it is in the transient switch
# Note that the different names rtots versus tstor are expected in the output

$ oc -n openshift-ovn-kubernetes exec -ti ovnkube-node-btkrd -c northd -- ovn-nbctl lsp-list transit_switch

989ca038-7e3f-4043-a0e7-ee9fa0dbbcf4 (tstor-ctlplane-0.ocp.example.net)
229979a6-8c66-4c1f-9ab9-5e309bcf8d7c (tstor-ctlplane-1.ocp.example.net)
6735161c-cf74-4f5b-a39b-9d7b68933e89 (tstor-ctlplane-2.ocp.example.net)
9513d930-e988-489e-b26f-d19753cc42be (tstor-worker-0.ocp.example.net)
615dc47f-32a1-43a0-8708-ea4db1c1f7fb (tstor-worker-1.ocp.example.net) ## Here it is

Testing Egress configuration if using OVN-kubernetes on OpenShift 4.13 and earlier:

Link to documentation

  • OpenShift will organize your egressIP via an EgressIP object (eip) you can query to validate which node is currently elected to host that egressIP address, and then you can cross-reference it with the ovn-kubernetes database running on the cluster and confirm that your pods are snatted properly to this address.
    In order to query the OVN databases; we will first need to learn which pod is currently running as the leader; all the database requests will then be sent to this pod for validation.
#Get OVN Leader pod:
$ for f in $(oc -n openshift-ovn-kubernetes get pods -l app=ovnkube-master -o jsonpath={.items[*].metadata.name})  ; do echo -e "\n${f}\n" ; oc -n openshift-ovn-kubernetes exec "${f}" -c northd -- ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound  ; done

//example output: 
ovnkube-master-7lw96

ad98
Name: OVN_Northbound
Cluster ID: 819a (819aebee-c723-4537-bd91-a8e38d0e8f15)
Server ID: ad98 (ad983c20-41c0-4ece-aeac-e61bd68c5fc5)
Address: ssl:10.0.94.197:9643
Status: cluster member
Role: leader
Term: 5
Leader: self
Vote: self
  • Get the EgressIP object and observe Assigned Node:
$ oc get eip
NAME       EGRESSIPS    ASSIGNED NODE      ASSIGNED EGRESSIPS
eip-test   10.0.91.88   worker-2           10.0.91.88


$ oc get eip -o yaml
apiVersion: k8s.ovn.org/v1
kind: EgressIP
metadata:
  name: eip-test
spec:
  egressIPs:
  - 10.0.91.88
  namespaceSelector:
    matchLabels:
      env: egress

$ oc get nodes -o wide | grep worker-2
worker-2   Ready    worker   25h   v1.24.6+deccab3   10.0.91.87   ...

#note node IP: 10.0.91.87
  • List the SNAT entries (Source-NAT) of pods using the egress IP address you wish to query: (use the leader pod you found earlier)

**NOTE: using any ovnkube-master pod; you can insert the option --no-leader-only and query the database even if the ovnkube-master is not the leader, provided the pod is in 6/6 ready status:
ovn-nbctl --no-leader-only show ...

$ oc -n openshift-ovn-kubernetes exec -ti ovnkube-master-7lw96  -c northd -- ovn-nbctl show | grep -B1 -A3 "10.0.91.88"
    nat ac5ee984-d78b-4a1a-885a-9f6b98552a29
        external ip: "10.0.91.88"
        logical ip: "10.128.2.20"
        type: "snat"
    nat b100888d-585d-40ee-81a0-e5aa09be3065

$ oc get pods -o wide -n <namespace>
httpd-ex-c794bd8df-7p7g4   1/1     Running     0          9m59s   10.128.2.20   worker-0   none  none

#observe logical IP of pod: 10.128.2.20, is snatted to externalIP (egressIP) -- 10.0.91.88
  • Query for node affiliation in the database tables and snat rules
$ oc -n openshift-ovn-kubernetes exec -ti <ovn-master-that-is-leader> -c northd -- ovn-nbctl --format=csv find nat external_ids:name=<egress-ip-name>

#example:
$ oc -n openshift-ovn-kubernetes exec -ti ovnkube-master-7lw96 -c northd -- ovn-nbctl --format=csv find nat external_ids:name=eip-test

_uuid,allowed_ext_ips,exempted_ext_ips,external_ids,external_ip,external_mac,external_port_range,gateway_port,logical_ip,logical_port,options,type
ac5ee984-d78b-4a1a-885a-9f6b98552a29,[],[],{name=eip-test},"""10.0.91.88""",[],"""""",[],"""10.128.2.20""",k8s-worker-2,"{stateless=""false""}",snat

  • Observe above the worker node (worker-2) is labeled as bound to 10.0.91.88 egressIP object, being routed by snatted pod IP 10.128.2.20

  • Note that this pod runs on worker-0 as we have previously seen. This example assumes only one pod is managed by egress IP; you may need to grep the output further if you have an enormous amount of egress addresses being managed.

  • You can also use --format=table or --format=json instead of --format=csv in the above query to get an easier to read output depending on your needs/use-case.

  • Check also the logical_router_policy of the pod to confirm it's affinity and reroute rule with an egressIP:

#The following command checks the logical_router_policy for a pod(this is responsible for forwarding the pod traffic to the egress node), there should be one entry per pod IP:

$ oc -n openshift-ovn-kubernetes exec -ti <ovn-master-that-is-leader> -c northd -- ovn-nbctl --no-leader-only find logical_router_policy match="\"ip4.src == POD_IP\""

$ oc project openshift-ovn-kubernetes
$ oc rsh <ovnkube-master-pod>
$ ovn-nbctl --no-leader-only find logical_router_policy match="\"ip4.src == 10.128.2.20\""
_uuid               : 60b2dc86-cae9-4c68-85c8-1c75f798165e
action              : reroute
external_ids        : {name=eip-test}
match               : "ip4.src == 10.128.2.20"
nexthop             : []
nexthops            : ["100.64.0.5"]
options             : {}
priority            : 100

  • To find SNATs that match the pod IP use the following syntax in the ovnkube-master pod:
    $ oc -n openshift-ovn-kubernetes exec -ti <ovn-master-that-is-leader> -c northd -- ovn-nbctl --no-leader-only find nat logical_ip="POD_IP"

  • If the pod is on the same node as the egressIP then there should be only one SNAT present, otherwise there should be two (example below, assuming pod IP is 10.128.2.20):


$ oc -n openshift-ovn-kubernetes exec -ti <ovn-master-that-is-leader> -c northd -- ovn-nbctl --no-leader-only find nat logical_ip="10.128.2.20"
_uuid               : 7ab7e900-1e72-4120-97ca-607725b4a2fe
allowed_ext_ips     : []
exempted_ext_ips    : []
external_ids        : {}
external_ip         : "10.0.94.86"
external_mac        : []
external_port_range : ""
gateway_port        : []
logical_ip          : "10.128.2.20"
logical_port        : []
options             : {stateless="false"}
type                : snat

_uuid               : e2196a29-c98c-4dac-9874-3250b003b934
allowed_ext_ips     : []
exempted_ext_ips    : []
external_ids        : {name=eip-test}
external_ip         : "10.0.91.88"
external_mac        : []
external_port_range : ""
gateway_port        : []
logical_ip          : "10.128.2.20"
logical_port        : k8s-worker-2
options             : {stateless="false"}
type                : snat
  • Troubleshooting further:
    • Are your desired egress host nodes labeled properly with k8s.ovn.org/egress-assignable: ""
    • Are your pod or namespace objects properly labeled to match the egressIP selector value

Testing for egressIP affiliation on OpenShift-SDN

Link to our documentation

  • Because OpenShift SDN uses the hostnetwork and netnamespace objects for managing egress, we'll need to start by querying these to determine which nodes are hosting the egress addresses in use:

  • Get hostnetwork/netnamespace:

$ oc get hostsubnet 
NAME       HOST       HOST IP         SUBNET          EGRESS CIDRS   EGRESS IPS
master-0   master-0   10.0.134.251    10.129.0.0/23                  
master-1   master-1   10.0.138.46     10.130.0.0/23                  
master-2   master-2   10.0.156.243    10.131.0.0/23                  
worker-1   worker-1   10.0.168.102    10.128.2.0/23                  ["10.0.168.103"]
worker-2   worker-2   10.0.93.243     10.129.2.0/23                  
worker-3   worker-3   10.0.253.40     10.128.0.0/23

$ oc get netnamespace | grep <project>
NAME                                                NETID      EGRESS IPS
testing                                             7179866    ["10.0.168.103"]
  • Counter to how this works in OVN, Worker-2 has been allocated as our egressIP host, and we can see in the ip a output of the node that a secondary IP is now attached to the node explicitly:
$ oc debug node/worker-2 -- chroot /host sh -c "ip a | grep eip" 
Starting pod/worker-2-debug ...
To use host binaries, run `chroot /host`
    inet 10.0.168.103/18 brd 10.0.191.255 scope global secondary ens5:eip
  • We should also see in the sdn logs for the pod allocated to the node hosting the egressIP the following:
I0401 21:31:30.105308    1819 egressip.go:506] Initialized egress IP capacity: 14 for Node: "worker-2"
  • We can now validate that the pods in your namespace are allocated; check the pod IP and determine worker that hosts it:
NAME         READY   STATUS    RESTARTS   AGE    IP            NODE                                         NOMINATED NODE   READINESS GATES
egress-pod   1/1     Running   0          3m3s   10.131.0.18   worker-1   <none>           <none>
  • Now we're going to check the OVS flows and iptables rulesets for the worker that has allocated an egress IP in use by this pod:
##on node running EgressIP; observe that in the below output we have an explicit SNAT rule, defining that networkID [0x106d0e5a] should be SNAT forwarded to the exit source IP with masquerade:

sh-4.4# iptables-save -c | grep 10.0.168.103
[78:4680] -A OPENSHIFT-MASQUERADE -s 10.128.0.0/14 -m mark --mark 0x106d0e5a -j SNAT --to-source 10.0.168.103
[0:0] -A OPENSHIFT-FIREWALL-ALLOW -d 10.0.168.103/32 -m conntrack --ctstate NEW -j REJECT --reject-with icmp-port-unreachable

##on node running the test pod; observe that the IP of our pod [10.131.0.18] is labeled with the same networkID: [0x106d0e5a].

h-4.4# ovs-ofctl -O OpenFlow13 dump-flows br0 | grep 10.131.0.18
 cookie=0x0, duration=1099.897s, table=20, n_packets=59, n_bytes=2478, priority=100,arp,in_port=19,arp_spa=10.131.0.18,arp_sha=00:00:0a:83:00:12/00:00:ff:ff:ff:ff actions=load:0x6d8e5a->NXM_NX_REG0[],goto_table:30
 cookie=0x0, duration=1099.897s, table=20, n_packets=2269, n_bytes=224482, priority=100,ip,in_port=19,nw_src=10.131.0.18 actions=load:0x6d8e5a->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=1099.897s, table=25, n_packets=1080, n_bytes=98064, priority=100,ip,nw_src=10.131.0.18 actions=load:0x6d8e5a->NXM_NX_REG0[],goto_table:27
 cookie=0x0, duration=1099.897s, table=40, n_packets=59, n_bytes=2478, priority=100,arp,arp_tpa=10.131.0.18 actions=output:19
 cookie=0x0, duration=1099.897s, table=70, n_packets=2280, n_bytes=806558, priority=100,ip,nw_dst=10.131.0.18 actions=load:0x6d8e5a->NXM_NX_REG1[],load:0x13->NXM_NX_REG2[],goto_table:80

**NOTE: the VNID in OVS flows is in decimal

Testing with a local bastion replicator:

  • On a bastion run a simple HTTP server:
 # echo "Hello Egress traffic is Working" > /tmp/test-egress.txt
 # firewall-cmd --zone=public --add-port=8080/tcp
 # python -m http.server --directory /tmp 8080
  • Make sure the firewall is not blocking the traffic for this port from the Openshift nodes.
  • Create a file called egress-pod.yaml with the following content:
$ cat egress-pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: egress-pod
  annotations:
spec:
  containers:
  - command: ['bash', '-c', 'while true; do curl -m 5 http://<ip-of-bastion>:8080/test-egress.txt ; sleep 10; done']
    image: registry.redhat.io/rhel7/rhel-tools
    imagePullPolicy: IfNotPresent
    name: test-egress-pod
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
  • On OCP:
 $ oc new-project test-egress
 $ oc create -f egress-pod.yaml
  • see in the egress-pod logs:
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    24  100    24    0     0    680      0 --:--:-- --:--:-- --:--:--   800
Hello Egress traffic is Working
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
Hello Egress traffic is Working
100    24  100    24    0     0  11379      0 --:--:-- --:--:-- --:--:-- 24000

Capture Traffic with TCPdump:

  • At the pod making the request via the egressIP
  • On the NODE hosting the egressIP
  • At the TARGET host that is speaking with the egressIP
Capture on the pod
Capture on the NODE hosting the egressIP
  • find out the primary host interface. If using OVN-Kubernetes, usually it will be the first physical interface mentioned in ovs-ofctl dump-ports-desc br-ex.
[root@node]# ovs-ofctl dump-ports-desc br-ex
OFPST_PORT_DESC reply (xid=0x2):
 1(enp1s0): addr:02:01:00:00:00:66
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 2(patch-br-ex_nod): addr:e2:f7:61:a0:0d:cd
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-ex): addr:02:01:00:00:00:66
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max

# here the primary host interface is enp1s0
  • On the NODE hosting the egressIP, we want to watch traffic at the primary host interface for the node; and filter for traffic coming from the egressIP: 
# tcpdump -s 0 -n -i <ethX> -w /tmp/$(hostname)-$(date +&quot;%Y-%m-%d-%H-%M-%S&quot;).pcap host <egress-ip>
# where ethX is the primary interface for the node
Capture on the TARGET host that is speaking with the egressIP
  • On the target host that is communicating with the egressIP we will do the same; filtering for the egress IP again:
# tcpdump -s 0 -n -i <ethX> -w /tmp/$(hostname)-$(date "%Y-%m-%d-%H-%M-%S").pcap host <egress-ip>
# where ethX is the primary interface for that host machine

Capturing TCPdumps for Diagnosing Intermittent Timeout/Connectivity Issues:

  • Collect TCPdumps on the three nodes where the application pod is running, where the EgressIP is assigned, and on the destination node using the command below. This command will only capture required packages with this filter.
# tcpdump -nnee -s 0 -i any -y LINUX_SLL2  -w /host/var/tmp/issue_${HOSTNAME}.pcap
  • From the Application POD, run the command below after starting the TCPdump to capture intermittent timeouts.
# for i in {00..10}; do echo "######### Iteration $i - `date` - Port 655$i ##########"; curl -Ivv --connect-timeout 5 <DestIP>:<PORT> --local-port 655$i; done

-- In event destination server is database IP or validate with telnet command --

# for i in {00..10}; do echo "######### Iteration $i - `date` - Port 655$i ##########"; timeout 5 curl -vvv telnet://<DestIP>:<PORT> --local-port 655$i ; sleep 2; done
  • For PCAP analysis, identify the tcpport (i.e. localport) of the failed request from the curl test and use it to filter across all three captured PCAP files.

Disclaimer

The IP addresses, hostnames and MAC addresses, etc provided here are for convenience only and are sourced exclusively from the Red Hat LAB environment. You acknowledge that we are not responsible or liable for any losses or expenses that may result from the use of such IP addresses for illustrative purposes in this article.

Root Cause

  • It can be difficult to immediately confirm upon egress IP deployment whether or not your traffic is routed properly; the above checks can help streamline your rollout and provide validation methodology to ensure your traffic is routed appropriately through the desired nodes and IPs.

  • OVN-Kubernetes was updated in OpenShift 4.14 from a single big HA OVN control plane (using distributed databases through RAFT algorithm) to a distributed handling solution that empowers each node with it's own OVN instance and interconnects them all. Instead of checking in with the nbdb on whichever master node won raft leader-election and just programming the OVN control plane on it, now OVN Kubernetes programs a different OVN control plane on each node separately (this has great benefits because each per-node control plane needs a much lesser number of items on its databases). See This content is not included.this blog post for details.

  • Read more here about EgressIP Failover times and expected behaviors

Diagnostic Steps

  • In OpenShift 4 using ovnkubernetes; it is not immediately apparent that a node IS using the correct IP address as assigned by EgressIP. The IP address does not appear bound to the br-ex interface; and it might not immediately be apparent that the node is routing traffic appropriately.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.