JBoss EAP server high response times and high CPU with many HttpManagementService threads

Solution Unverified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform 6.x
  • Red Hat JBoss Enterprise Application Platform 7.x
  • Getting Hibernate Stats via JON

Issue

JBoss EAP high response times and high CPU. Many HttpManagementService-threads are running and all high CPU consumer threads are HttpManagementService-threads. An example of one of threads is like this:

"HttpManagementService-threads - 4223" prio=10 tid=0x00007f1d040f8800 nid=0x7596 runnable [0x00007f1ca5b03000]
   java.lang.Thread.State: RUNNABLE
	at java.util.regex.Matcher.appendReplacement(Matcher.java:756)
	at java.util.regex.Matcher.replaceAll(Matcher.java:906)
	at java.lang.String.replace(String.java:2180)
	at org.jboss.as.jpa.hibernate4.management.QueryName.displayable(QueryName.java:187)
	at org.jboss.as.jpa.hibernate4.management.QueryName.<init>(QueryName.java:120)
	at org.jboss.as.jpa.hibernate4.management.QueryName.queryName(QueryName.java:110)
	at org.jboss.as.jpa.hibernate4.management.HibernateStatisticsResource.getQueryNames(HibernateStatisticsResource.java:345)
	at org.jboss.as.jpa.hibernate4.management.HibernateStatisticsResource.hasQuery(HibernateStatisticsResource.java:332)
	at org.jboss.as.jpa.hibernate4.management.HibernateStatisticsResource.hasChild(HibernateStatisticsResource.java:81)
	at org.jboss.as.controller.OperationContextImpl.getAuthorizationResource(OperationContextImpl.java:1168)
	at org.jboss.as.controller.OperationContextImpl.getBasicAuthorizationResponse(OperationContextImpl.java:1126)
	at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1049)
	at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:845)
	at org.jboss.as.controller.operations.global.ReadResourceHandler.doExecuteInternal(ReadResourceHandler.java:265)
	at org.jboss.as.controller.operations.global.ReadResourceHandler.doExecute(ReadResourceHandler.java:142)
	at org.jboss.as.controller.operations.global.GlobalOperationHandlers$AbstractMultiTargetHandler.execute(GlobalOperationHandlers.java:249)
	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:601)
	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:479)
	at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:283)
	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:278)
	at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:231)
	at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:137)
	at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:437)
	at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:427)
	at org.jboss.as.domain.http.server.DomainApiHandler.processRequest(DomainApiHandler.java:325)
	at org.jboss.as.domain.http.server.DomainApiHandler.doHandle(DomainApiHandler.java:229)
	at org.jboss.as.domain.http.server.DomainApiHandler.handle(DomainApiHandler.java:236)
	at org.jboss.as.domain.http.server.security.SubjectAssociationHandler$1.run(SubjectAssociationHandler.java:69)
	at org.jboss.as.domain.http.server.security.SubjectAssociationHandler$1.run(SubjectAssociationHandler.java:65)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94)
	at org.jboss.as.domain.http.server.security.SubjectAssociationHandler.handleRequest(SubjectAssociationHandler.java:65)
	at org.jboss.as.domain.http.server.security.SubjectAssociationHandler.handle(SubjectAssociationHandler.java:59)
	at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:78)
	at org.jboss.sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:69)
	at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:81)
	at org.jboss.sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:710)
	at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:78)
	at org.jboss.as.domain.http.server.RealmReadinessFilter.doFilter(RealmReadinessFilter.java:47)
	at org.jboss.as.domain.http.server.DmrFailureReadinessFilter.doFilter(DmrFailureReadinessFilter.java:45)
	at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:81)
	at org.jboss.sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:682)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

Resolution

The internal implementation of our management API which is related to Hibernate/JPA statistic is causing this high cpu issue. This is a bug reported in the following BZ/JIRA which is fixed in EAP 6.3 CP01 (6.3.1) or later of EAP 6:

For EAP 7, unfortunately there is a regression of this issue. It will be addressed by the following JIRAs:

As workaround, please disable Hibernate/JPA statistics gathering from JON. If it's required for you, try increase interval of monitoring.

Diagnostic Steps

  • Capture high CPU data using the information in this article for linux, and this for windows.

  • In order to help us reproduce a similar issue we'd like to know:

    • How large is the query you are using in Hibernate/JPA?
    • Did they do any change in their application using Hibernate/JPA recently?
    • Did they do any change in their JON monitoring setting regarding Hibernate/JPA recently?
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.