Fuse or ActiveMQ resource discovery reports "Failed to extract JMX service URL for process with PID [...]" due to failure in getProcCredName

Solution Unverified - Updated

Environment

  • Red Hat JBoss Operations Network (ON)
    • 3.3.0
    • 3.3.1
    • 3.3.2
    • 3.3.3
    • 3.3.4
    • 3.3.5
    • 3.3.6
    • 3.3.7
    • 3.3.8
    • 3.3.9
  • Red Hat JBoss Fuse or A-MQ/AMQ
    • JBoss A-MQ 6.2.1
    • JBoss Fuse 6.2.1
  • Linux or UNIX based operating system
  • JBoss ON agent's native support has not been disabled
  • JBoss ON agent has access to Java JDK/SDK which includes the Java Attach API libraries
  • The JBoss ON agent is not running as root

Issue

  • Agent is not detecting or discovering ActiveMQ

  • agent.log contains discovery warning:

      WARN  [ResourceDiscoveryComponent.invoker.daemon-2] (org.jbosson.plugins.fuse.JBossFuseContainerDiscoveryComponent)- Error when trying to discover Fuse Server process [ProcessScanResult: scan=[ProcessScan: query=[process|basename|match=^java.*,arg|-Dkaraf.home|match=.*,arg|-Dkaraf.base|match=.*,arg|*|match=org\.apache\.karaf\.(main|shell\.wrapper)\.Main], name=[null]], info=[process: pid=[1234], name=[java], ppid=[123]]] (enable DEBUG for stack trace): java.lang.RuntimeException: Failed to extract JMX service URL for process with PID [1234].

  • Agent's debug log shows following discovery failure:

      java.lang.RuntimeException: Failed to extract JMX service URL for process with PID [1234].
      at org.rhq.plugins.jmx.util.JvmUtility.extractJMXServiceURL(JvmUtility.java:100)
  Caused by: java.lang.NullPointerException
      at org.rhq.plugins.jmx.util.JvmUtility.attachToVirtualMachine(JvmUtility.java:117)

  • Unable to discover JBoss Fuse Container, ActiveMQ Server, Fabric Container, or Servicemix4

Resolution

This issue has been resolved in the latest JBoss ON 3.3 update.

If you are not able to apply the update, you can manually import the Fuse or ActiveMQ resource as explained in the JBoss ON Users Guide.

Root Cause

The Generic JMX plug-in used by ActiveMQ and some of the other Fuse components will attempt to utilize the Java Attach API to connect directly the the managed server's JVM process. This operation requires that the JBoss ON agent and the managed server's JVM process are owned by the same user. To determine which connection method to use, the Generic JMX plug-in will utilize a native library to determine the user information for the target process.

The native library will invoke various system calls to gather user and primary group information for the process that is to be discovered. These calls utilize a buffer with a hard-coded buffer size of 1024 bytes. If either the user or group entry returned by the operating system exceeds this buffer size, an ERANGE error, reported as Numerical result out of range, is returned and the native library call fails to return any user information.

The Generic JMX plug-in then attempts to access the user information which has not been initialized resulting in a NullPointerException when executing JvmUtility.attachToVirtualMachine.

This issue was originally captured in This content is not included.Red Hat Bugzilla 1382041 and it was thought that the issue was resolved in JBoss ON 3.3 Update 08, however, this may not be the case.

A new bug report was captured in This content is not included.Red Hat Bugzilla 1430697 which tracks an upstream issue identified in Content from jira.hyperic.com is not included.SIGAR-231. It is important to note that the resolution identified in SIGAR-231 may still be insufficient in many cases as the fix provided there only increases the hard-coded buffer size to 2048 bytes. This is still too small for many enterprise deployments that utilize LDAP or other user or group database implementations.

Diagnostic Steps

  • Test to see if credname is missing when invoking the Sigar's pinfo command as the same user who runs the agent and passing it the process id of the Fuse or ActiveMQ JVM.

      cd "${RHQ_AGENT_HOME}"/lib
  java -jar sigar-*.jar pinfo <PID>

    <PID> will need to be the actual process id of the Fuse or ActiveMQ JVM, not its start script or parent process or wrapper. If the output does not reflect state={Name=java... you probably have the wrong process id.

    This command should yield output similar to the following:

      pid=26418
  state={Name=java, Threads=108, Tty=34819, State=S, Ppid=26368, Processor=7, Priority=20, Nice=0}
  mem={Resident=372408320, PageFaults=187036, MajorFaults=0, Share=23486464, MinorFaults=187036, Size=5299863552}
  cpu={User=108750, LastTime=1475220798994, Percent=0.0, StartTime=1475159585000, Total=150960, Sys=42210}
  cred={Euid=1000, Gid=1000, Egid=1000, Uid=1000}
  credname={User=testUser, Group=testUser}

    If the last line that starts credname= is missing, this issue applies.

  • Enable debug logging on the agent. If you see DEBUG log messages similar to the following, this issue applies:

      DEBUG [InventoryManager.discovery-1] (org.rhq.core.system.ProcessInfo)- Unexpected error occurred while looking up info for [<unknown>] process with pid [26418] - call to getProcCredName failed. Did the process die? Cause: org.hyperic.sigar.SigarException: Numerical result out of range

    Please note that the pid value in the message should match that of the Fuse or ActiveMQ JVM you are attempting to discover.

SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.