Why are IPv6 DNS lookups done even when IPv6 is disabled?
Environment
- Red Hat Enterprise Linux
- glibc
Issue
- Applications like ssh and telnet use the
getaddrinfo()function withAF_UNSPECand this function invokes both AAAA (IPv6) and A (IPv4) lookups one after the other. This can delay the connection time when DNS servers block or don't handle IPV6 correctly. - How do I prevent IPv6 DNS queries?
getaddrinfo()takes a long time to resolve hostnames- DNS lookups slow in Java due to poor response to IPv6 AAAA records
- How can I change the
getaddrinfo(AF_UNSPEC)system call behaviour in Java, to only look up IPv4 A records when a DNS request is made? - Java application hangs on DNS resolution lookup with backtrace like:
"http-8080-1" daemon prio=10 tid=0x00007fba60008800 nid=0x3d5d runnable [0x00007fba96b3a000]
java.lang.Thread.State: RUNNABLE
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:902)
at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1281)
at java.net.InetAddress.getAllByName0(InetAddress.java:1232)
at java.net.InetAddress.getAllByName(InetAddress.java:1163)
at java.net.InetAddress.getAllByName(InetAddress.java:1099)
at java.net.InetAddress.getByName(InetAddress.java:1049)
Resolution
Resolution - RHEL 8.7 / RHEL 9.1 and later
Update to the following package versions or later:
| Red Hat Enterprise Linux release | Package |
|---|---|
| 9.1 | glibc-2.34-40.el9 |
| 8.7 | glibc-2.28-211.el8 |
The aim of the configuration is to end up with the following /etc/resolv.conf option set:
options no-aaaa
If using NetworkManager, update to the following packages or later:
| Red Hat Enterprise Linux release | Package |
|---|---|
| 9.3 | NetworkManager-1.44.0-3.el9 |
| 8.9 | NetworkManager-1.40.16-9.el8 |
Add the no-aaaa DNS Option connection property and apply it:
nmcli con mod CONNAME +ipv4.dns-options 'no-aaaa'
nmcli con up CONNAME
If using an earlier NetworkManager or initscripts, then edit the /etc/resolv.conf file manually.
resolv.conf file will be overwritten the next time NetworkManager or initscripts modifies the file, so configuration to prevent modifications of resolv.conf is necessary:
If using the legacy initscripts, set PEERDNS=no in each ifcfg file to prevent modification of resolv.conf:
Resolution - Java
For Java applications, preventing IPv6 address lookups can be achieved by passing the following option to the Java command:
-Djava.net.preferIPv4Stack=true
Resolution - Fix the Application
Repair the application so that it does explicitly perform an address lookup of the desired address family.
For IPv4 this is achieved by passing AF_INET as the hints.ai_family to getaddrinfo().
For IPv6 this is achieved by passing AF_INET6 as the hints.ai_family to getaddrinfo().
Ensure the application does not perform an unspecified-family address lookup by passing AF_UNSPEC as the hints.ai_family (or NULL hints) to getaddrinfo().
Workaround - DNS Server
Configure the DNS server to prevent it from ignoring or responding slowly to AAAA queries.
The DNS server could respond with RCODE 3(NXDOMAIN) Name Error as described in Content from www.rfc-editor.org is not included.RFC-1035 - Domain Names - Implementation and Specification, or the pseudo-RCODE NODATA as described in Content from datatracker.ietf.org is not included.RFC-2308 - Negative Caching of DNS Queries which should prevent a long timeout and prevent an IPv6 connection attempt.
Workaround - Local nameserver
Install a local caching-only nameserver which filters AAAA queries.
The BIND nameserver is supplied and supported in Red Hat Enterprise Linux
The following two lines must be added to the options section of /etc/named.conf file. Replace the IP addresses with the address of a real DNS name server. Multiple servers can be listed separated by a semi-colon.
forwarders {192.0.2.111;192.0.2.112;};
filter-aaaa-on-v4 yes;
The /etc/resolv.conf file must be changed to list only one name server - the local system:
nameserver 127.0.0.1
The application will still request both IPv6 and IPv4 addresses but the caching name server will return only the IPv4 addresses it gets from the actual name server. The IPv6 query gets a response with 0 addresses (NODATA). Since a response cache is created this should have the added benefit of speeding up subsequent requests.
Other nameservers like dnsmasq or Unbound may also support being configured in a similar way.
Unsupported Workarounds
The following information has been provided by Red Hat, but is outside the scope of the posted This content is not included.Service Level Agreements and support procedures. The information is provided as-is and any configuration settings or installed applications made from the information in this article could make the Operating System unsupported by Red Hat Global Support Services. The intent of this article is to provide information to accomplish the system's needs. Use of the information in this article at the user's own risk.
Wrapper Library
The library in the attachments of this knowledgebase page does NOT have any support from Red Hat. The attached wrapper library libwgetaddrinfo overrides the behavior of application calls to getaddrinfo() to produce IPv4-only lookups. This could violate standards, including POSIX-2017.1, and is the user's decision to do that.
The library package, along with documentation, is attached to this page. Due to a conflict between the POSIX standard and the requirement for IPv4-only lookups this package is not officially supported by Red Hat. However, it does disable IPv6 lookups regardless of application code behavior.
Most applications that are part of Red Hat Enterprise Linux offer a configuration option to disable IPv6 (or even IPv4) completely. It is advisable that any third-party application provides similar solutions.
Root Cause
The issue reported is the expected default behaviour.
When an application performs a DNS lookup with AF_UNSPEC or NULL hints to getaddrinfo(), both an IPv4 A record and IPv6 AAAA record are requested from DNS, as explained in man getaddrinfo:
ai_family This field specifies the desired address family for the returned
addresses. Valid values for this field include AF_INET and AF_INET6.
The value AF_UNSPEC indicates that getaddrinfo() should return
socket addresses for any address family (either IPv4 or IPv6, for
example) that can be used with node and service.
If IPv4-only DNS lookups are required, then the application must request AF_INET lookups.
If the DNS server does not respond to the AAAA record request, then the application will wait for the DNS timeout and retry the request, specified by the timeout and attempts options in the /etc/resolv.conf file.
Whether the kernel's IPv6 addressing or IPv6 protocol is disabled or not is separate to which DNS lookups are performed.
There is an upstream RFE with glibc to make this configurable, but it has not been completed yet:
This was originally tracked as Red Hat Private Bug 1027452 - glibc: [RFE] Provide mechanism to disable AAAA queries when using AF_UNSPEC on IPv4-only configurations which was closed. It was tracked again on Red Hat Jira Issue RHEL-12776 - RFE: Tunable to make getaddrinfo AF_UNSPEC use one address family where the new upstream no-aaaa option was provided as a supported solution to prevent IPv6 AAAA lookups.
Please note that changing the lookup behavior to disable IPv6 lookups despite of AF_UNSPEC breaks the Content from pubs.opengroup.org is not included.POSIX-2017.1 standard, specifically:
If hints is a null pointer, the behavior shall be as if it referred to a structure containing the value zero for the
ai_flags,ai_socktype, andai_protocolfields, andAF_UNSPECfor theai_familyfield.If the
ai_familyfield to which hints points has the valueAF_UNSPEC, addresses shall be returned for use with any address family that can be used with the specified nodename and/or servname.
In RHEL 7 and later, the A and AAAA lookups are done simultaneously. In RHEL 5 and earlier, the IPv6 lookup was done before the IPv4 lookup.
The order of responses presented to the application by the DNS resolver library can be configured as described in the /etc/gai.conf file as described on man gai.conf.
Diagnostic Steps
Perform packet capture and filter on DNS traffic, looking for DNS AAAA queries with delayed or no response.
Capture Java thread dumps and look for threads hanging in the call stack in the Issue section.
Capture ltrace (library trace) of applications to check if the third argument hints is NULL or not:
man 3 getaddrinfo
int getaddrinfo(const char *node, const char *service,
const struct addrinfo *hints,
struct addrinfo **res);
# ltrace -tt -e getaddrinfo -- getent ahosts example.com
HH:MM:SS.mmmuuu getent->getaddrinfo("example.com", nil, 0x7ffeca1e9260, 0x7ffeca1e9258) = 0
If hints is not NULL then the application must be debugged to inspect the contents of the hints structure.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.