HTTPS access to admin console in EAP6 is very slow when DNS is down

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP) 6.x

Issue

  • HTTPS access to admin console in EAP6 is very slow when DNS is down or slow
  • Thread dumps show SSL console connections stalled in DNS lookups like the following:
[Host Controller] "HttpManagementService-threads - 3" prio=10 tid=0x00007fd1c055e800 nid=0xf959 runnable [0x00007fd1d90f3000]
[Host Controller]    java.lang.Thread.State: RUNNABLE
[Host Controller]         at java.net.Inet4AddressImpl.getHostByAddr(Native Method)
[Host Controller]         at java.net.InetAddress$1.getHostByAddr(InetAddress.java:905)
[Host Controller]         at java.net.InetAddress.getHostFromNameService(InetAddress.java:590)
[Host Controller]         at java.net.InetAddress.getHostName(InetAddress.java:532)
[Host Controller]         at java.net.InetAddress.getHostName(InetAddress.java:504)
[Host Controller]         at java.net.InetSocketAddress$InetSocketAddressHolder.getHostName(InetSocketAddress.java:82)
[Host Controller]         at java.net.InetSocketAddress$InetSocketAddressHolder.access$600(InetSocketAddress.java:56)
[Host Controller]         at java.net.InetSocketAddress.getHostName(InetSocketAddress.java:345)
[Host Controller]         at org.jboss.sun.net.httpserver.SSLStreams.<init>(SSLStreams.java:73)
[Host Controller]         at org.jboss.sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:554)
[Host Controller]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[Host Controller]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[Host Controller]         at java.lang.Thread.run(Thread.java:745)
[Host Controller]         at org.jboss.threads.JBossThread.run(JBossThread.java:122)

Resolution

  • Upgrade to EAP 6.4.0 when available

  • A workaround exists on the platform layer. The nscd (name service cache daemon) can cache hosts entry. Enabling nscd, however a first access takes 15 seconds to DNS-lookup and timeout, once the lookup gets successful after DNS failover, the entry is cached in the local machine, and it does not access any remote DNS to lookup during the cache entry lifetime. To install and run nscd,

      1. yum install nscd
  2. /etc/init.d/nscd start

    By default, hosts entries caching is enabled, and the cache lifetime is 3200 seconds. Please see if it fits your requirement.

Root Cause

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.