Java threads stalled in InetAddress.getLocalHost or Inet4AddressImpl.getHostByAddr

Environment

• Red Hat Enterprise Linux (RHEL)
• JBoss Enterprise Application Platform
• Java

Issue

• We have java/JBoss threads getting stalled in the following call:

"ajp-127.0.0.1-8009-1" daemon prio=10 tid=0x00002aaab468b000 nid=0x144f runnable [0x0000000046cda000]
   java.lang.Thread.State: RUNNABLE
	at java.net.Inet4AddressImpl.getLocalHostName(Native Method)
	at java.net.InetAddress.getLocalHost(InetAddress.java:1342)

• Or stalled in:

"ajp-127.0.0.1-8009-1" daemon prio=10 tid=0x00007fc6e828a800 nid=0x5892 runnable [0x00007fc67b4c6000]
   java.lang.Thread.State: RUNNABLE
	at java.net.Inet4AddressImpl.getHostByAddr(Native Method)
	at java.net.InetAddress$1.getHostByAddr(InetAddress.java:880)
	at java.net.InetAddress.getHostFromNameService(InetAddress.java:559)
	at java.net.InetAddress.getCanonicalHostName(InetAddress.java:530)
	at org.rhq.core.pc.PluginContainerConfiguration.getContainerName(PluginContainerConfiguration.java:700)
	at org.rhq.core.pc.inventory.InventoryManager.discoverPlatform(InventoryManager.java:1651)
	at org.rhq.core.pc.inventory.InventoryManager.executePlatformScan(InventoryManager.java:445)
	at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.pluginDiscovery(AutoDiscoveryExecutor.java:154)
	at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.call(AutoDiscoveryExecutor.java:104)
	at org.rhq.core.pc.inventory.AutoDiscoveryExecutor.call(AutoDiscoveryExecutor.java:70)

Resolution

• Add the hostname being looked up in these stalled threads to the /etc/hosts file
• If using the IBM JDK, you may try adding the-Dcom.ibm.cacheLocalHost=trueproperty to your JVM options to improve performance from caching

Root Cause

Native/OS host resolution calls are slow

Diagnostic Steps

• Troubleshoot using thread dumps and additional steps mentioned in Java application unresponsive
• A heap dump can be captured during the stall to inspect the Inet object and see host/addresses being looked up.
• Capture TCP dumps from JBoss and the DNS server to check the network activity

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.