How to add additional search domains and nameservers to resolv.conf in RHOCP 4

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• /etc/resolv.conf

Issue

• How to add additional search domain for the cluster DNS in order to resolve domains that belong to the cluster base domain in OCP IPI installation.

• The Domain Search List provided by DHCP does not contain all the required domains. How can we add additional?

• Is it possible to modify the domain search in /etc/resolv.conf?

• After an application is migrated to OpenShift they are failing to connect to outside resources due to the application not being configured to use fully qualified domain names.

• Is it possible to modify the nameservers in /etc/resolv.conf?

• Updating nameservers by creating machine configs causing MCP to go into a degraded state causing the below errors:

  E0928 15:37:57.579907    3108 writer.go:135] Marking Degraded due to: unexpected on-disk state validating against rendered-infra-cd76ce447a127cf588ccc839140855f8
  I0928 15:38:05.618367    3108 daemon.go:771] Current config: rendered-infra-19f5b87200b70348e167ddd9f3fe3138
  I0928 15:38:05.618396    3108 daemon.go:772] Desired config: rendered-infra-cd76ce447a127cf588ccc839140855f8
  I0928 15:38:05.631354    3108 update.go:1404] Disk currentConfig rendered-infra-cd76ce447a127cf588ccc839140855f8 overrides node annotation rendered-infra-19f5b87200b70348e167ddd9f3fe3138
  I0928 15:38:05.635347    3108 daemon.go:1014] Validating against pending config rendered-infra-cd76ce447a127cf588ccc839140855f8
  E0928 15:38:06.388077    3108 daemon.go:1413] content mismatch for file /etc/resolv.conf (-want +got):
    bytes.Join({
    - "search example.com",
    +       "# Generated by KNI resolv prepender NM dispatcher script",
    +       "search test.example.com",
    +      "nameserver 5.6.7.8",
            "nameserver 1.2.3.4,
  

Resolution

When using DHCP, the preferred method to modify the "domain search" is to set the required domains in Option 119 of the DHCP Server.

Directly modifying the /etc/resolv.conf manually or via MachineConfig is not supported. Instead of that, use the Kubernetes NMState Operator to change the DNS servers or to add search domains or options.
For examples refer to the NMState document for DNS and review also how to use the Kubernetes NMState Operator to change the DNS server IP and add options in /etc/resolv.conf on CoreOS nodes for additional examples. Refer also to NMState operator cannot implement DNS change for additional clarifications.

Root Cause

Below are some of the challenges for updating /etc/resolv.conf manually without NMState operator:

• The /etc/resolv.conf is managed by NetworkManager, and could be modified.
• In ipi installation (specifically openstack, vspere), the installer creates openshift-openstack-infra which runs some infra pods.
• Can not directly modify /etc/resolv.conf of the node using machine-config (can see floating IP being added in /etc/resolv.conf).
• With machine-config likely can create a race condition where two things try to manage the file.
• If using CI/CD, can probably change the dnspolicy and supply manual changes in pod config accordingly as per the requirement.

Diagnostic Steps

• After creating a machine config for updating the search domains, check below if the changes are getting apply or not:

  $ oc get nodes       //Nodes mark as SchedulingDisabled
  $ oc get mcp          //MCP status changes to progressing
  

• Wait for the nodes to come back to normal state and MCP into available state then check if the changes are updated in node's /etc/resolv.conf:

  $ oc debug node/[node_name]
  [...]
  sh-4.4# chroot /host bash
  [root@node_name /]# cat /etc/resolv.conf
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.