Pods are having issues with DNS after the network policies are applied

Environment

• Red Hat OpenShift Container Platform v4.6

Issue

• After applying the network policies to the project, the pods of that project stop resolving the DNS lookups.

Resolution

• The DNS endpoints are being made with port number 5353:

 oc get endpoints -n openshift-dns -o wide
NAME          ENDPOINTS                                                         
dns-default   10.x.x.16:5353,10.x.x.3:5353,10.x.x.23:5353 + 15 more...   

• Add the port 5353 in the appropriate NetworkPolicy:

 # cat allowing-dns-traffic.yaml

    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: foo-deny-egress
    spec:
      podSelector:
        matchLabels:
          app: foo
      policyTypes:
      - Egress
      egress:
      # allow DNS resolution
      - ports:
        - port: 5353   <--
          protocol: UDP
        - port: 5353    <--
          protocol: TCP

Root Cause

• Open the following ports that are necessary for worker nodes to function properly. Allow inbound and outbound connections to TCP and UDP port 53 and port 5353 for DNS access.
• Please refer HERE for more details.

Diagnostic Steps

• Deployed an test application:

# oc get pods
NAME                     READY   STATUS    RESTARTS   AGE
httpd-779cdbc4c5-wtxzd   1/1     Running   0          13m

• Test DNS resolution:

      # oc rsh httpd-779cdbc4c5-wtxzd
      sh-4.4$ nslookup test.com
      Server:        172.30.0.10
      Address:    172.30.0.10#53
      Non-authoritative answer:
      Name:    test.com
      Address: 69.172.200.235
  
      sh-4.4$ nslookup google.com
      Server:        172.30.0.10
      Address:    172.30.0.10#53
      Non-authoritative answer:
      Name:    google.com
      Address: 172.217.15.78
      Name:    google.com
      Address: 2607:f8b0:4004:810::200e
  

• Applied NetworkPolicy [Allowing DNS traffic]:

 # cat allowing-dns-traffic.yaml
   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: foo-deny-egress
   spec:
     podSelector:
       matchLabels:
         app: foo
     policyTypes:
     - Egress
     egress:
     # allow DNS resolution
     - ports:
       - port: 53
         protocol: UDP
       - port: 53
         protocol: TCP

   # oc get pods -l=app=foo
   NAME                     READY   STATUS    RESTARTS   AGE
   httpd-779cdbc4c5-wtxzd   1/1     Running   0          20m

• Test DNS resolution:

 # oc rsh httpd-779cdbc4c5-wtxzd 
 sh-4.4$ nslookup test.com
 ;; connection timed out; no servers could be reached    <----

 sh-4.4$ nslookup google.com
 ;; connection timed out; no servers could be reached    <----

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.