High CPU in org.jboss.sun.net.httpserver.AuthFilter.consumeInput and BoundaryDelimitedInputStream.fullRead

Solution Verified - Updated 2 Aug 2024

Environment

JBoss Enterprise Application Platform (EAP) 6.4.4

Issue

After experiencing some network issues or timeout trying to upload an application to the EAP console over HTTPS, we see high CPU in threads like the following:

"HttpManagementService-threads - 506" #6733 prio=5 os_prio=0 tid=0x00007fc2441cf800 nid=0x4736 runnable [0x00007fc1f24c1000]
   java.lang.Thread.State: RUNNABLE
	at java.io.FilterInputStream.read(FilterInputStream.java:107)
	at org.jboss.sun.net.httpserver.AuthFilter.consumeInput(AuthFilter.java:54)
	at org.jboss.sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:72)
	at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:81)
	at org.jboss.sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:710)
	at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:78)
	at org.jboss.as.domain.http.server.RealmReadinessFilter.doFilter(RealmReadinessFilter.java:48)
	at org.jboss.as.domain.http.server.DmrFailureReadinessFilter.doFilter(DmrFailureReadinessFilter.java:45)
	at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:81)
	at org.jboss.sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:680)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

"HttpManagementService-threads - 408" #6622 prio=5 os_prio=0 tid=0x00007fc2441c7000 nid=0x3070 runnable [0x00007fc1f5e2b000]
   java.lang.Thread.State: RUNNABLE
	at org.jboss.as.domain.http.server.multipart.BoundaryDelimitedInputStream.fullRead(BoundaryDelimitedInputStream.java:129)
	at org.jboss.as.domain.http.server.multipart.BoundaryDelimitedInputStream.read(BoundaryDelimitedInputStream.java:171)
	at org.jboss.as.domain.http.server.multipart.BoundaryDelimitedInputStream.read(BoundaryDelimitedInputStream.java:269)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
	- locked <0x000000061dc745b0> (a java.io.BufferedInputStream)
	at java.io.FilterInputStream.read(FilterInputStream.java:107)
	at org.jboss.as.repository.ContentRepository$Factory$ContentRepositoryImpl.addContent(ContentRepository.java:204)
	- locked <0x000000061307dfc8> (a java.security.MessageDigest$Delegate)
	at org.jboss.as.server.deployment.AbstractDeploymentUploadHandler.execute(AbstractDeploymentUploadHandler.java:76)
	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:702)
	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:537)
	at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338)
	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314)
	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1144)
	at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:331)
	at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:188)
	at org.jboss.as.domain.http.server.DomainApiHandler.processUploadRequest(DomainApiHandler.java:226)
	at org.jboss.as.domain.http.server.DomainApiHandler.doHandle(DomainApiHandler.java:161)
	at org.jboss.as.domain.http.server.DomainApiHandler.handle(DomainApiHandler.java:186)
	at org.jboss.as.domain.http.server.security.SubjectAssociationHandler$1.run(SubjectAssociationHandler.java:69)
	at org.jboss.as.domain.http.server.security.SubjectAssociationHandler$1.run(SubjectAssociationHandler.java:65)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94)
	at org.jboss.as.domain.http.server.security.SubjectAssociationHandler.handleRequest(SubjectAssociationHandler.java:65)
	at org.jboss.as.domain.http.server.security.SubjectAssociationHandler.handle(SubjectAssociationHandler.java:59)
	at org.jboss.as.domain.http.server.DomainApiCheckHandler.handle(DomainApiCheckHandler.java:45)
	at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:78)
	at org.jboss.sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:69)
	at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:81)
	at org.jboss.sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:710)
	at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:78)
	at org.jboss.as.domain.http.server.RealmReadinessFilter.doFilter(RealmReadinessFilter.java:48)
	at org.jboss.as.domain.http.server.DmrFailureReadinessFilter.doFilter(DmrFailureReadinessFilter.java:45)
	at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:81)
	at org.jboss.sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:680)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

Resolution

Apply JBoss EAP 6.4 Cumulative Patch (CP) 6 or later
Try uploading applications via CLI or HTTP as a workaround

Root Cause

This content is not included.BZ-1269979 - SSLStreams return 0 upon eof
The SSLStream would hit an eof and improperly return 0. Thus stream read loops continued indefinitely since the expected -1 was not returned to end the stream read.

SBR

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

jbossas

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.