JBoss fails to start and deploy our applications with the main thread hanging on a socketRead through XML validation

Solution Verified - Updated 7 Aug 2024

Environment

JBoss Enterprise Application Platform (EAP)

Issue

All of a sudden JBoss fails to start and deploy our applications. It can start fine without our applications, but afterwards still hangs on deployment of our applications. No changes have been made in any way.

The thread dumps show the main thread is hanging on a socketRead through XML validation:

"main" prio=10 tid=0x00002aabac05b000 nid=0x1d67 runnable [0x0000000041deb000]
   java.lang.Thread.State: RUNNABLE
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:129)
     at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
     at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
     at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
     - locked <0x00002aaabcd15a20> (a java.io.BufferedInputStream)
     at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
     at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
     at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1072)
     - locked <0x00002aaabcd0b290> (a sun.net.www.protocol.http.HttpURLConnection)
     at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)
     at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
     at org.apache.xerces.impl.xs.opti.SchemaParsingConfig.parse(Unknown Source)
     at org.apache.xerces.impl.xs.opti.SchemaParsingConfig.parse(Unknown Source)
     at org.apache.xerces.impl.xs.opti.SchemaDOMParser.parse(Unknown Source)
     at org.apache.xerces.impl.xs.traversers.XSDHandler.getSchemaDocument(Unknown Source)
     at org.apache.xerces.impl.xs.traversers.XSDHandler.parseSchema(Unknown Source)
     at org.apache.xerces.impl.xs.XMLSchemaLoader.loadSchema(Unknown Source)
     at org.apache.xerces.impl.xs.XMLSchemaValidator.findSchemaGrammar(Unknown Source)
     at org.apache.xerces.impl.xs.XMLSchemaValidator.handleStartElement(Unknown Source)
     at org.apache.xerces.impl.xs.XMLSchemaValidator.startElement(Unknown Source)
     at org.apache.xerces.xinclude.XIncludeHandler.startElement(Unknown Source)
     at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
     at org.apache.xerces.impl.XMLNSDocumentScannerImpl$NSContentDispatcher.scanRootElementHook(Unknown Source)
     at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
     at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
     at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
     at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
     at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
     at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
     at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
     at org.jboss.xb.binding.parser.sax.SaxJBossXBParser.parse(SaxJBossXBParser.java:199)
     at org.jboss.xb.binding.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:168)
     at org.jboss.xb.util.JBossXBHelper.parse(JBossXBHelper.java:189)
     at org.jboss.xb.util.JBossXBHelper.parse(JBossXBHelper.java:166)
     at org.jboss.deployers.vfs.spi.deployer.SchemaResolverDeployer.parse(SchemaResolverDeployer.java:137)
     at org.jboss.deployers.vfs.spi.deployer.SchemaResolverDeployer.parse(SchemaResolverDeployer.java:121)
     at org.jboss.deployers.vfs.spi.deployer.AbstractVFSParsingDeployer.parseAndInit(AbstractVFSParsingDeployer.java:304)
     at org.jboss.deployers.vfs.spi.deployer.AbstractVFSParsingDeployer.parseAndInit(AbstractVFSParsingDeployer.java:286)
     at org.jboss.deployers.vfs.spi.deployer.AbstractVFSParsingDeployer.parse(AbstractVFSParsingDeployer.java:203)
     at org.jboss.deployers.spi.deployer.helpers.AbstractParsingDeployerWithOutput.createMetaData(AbstractParsingDeployerWithOutput.java:348)
     at org.jboss.deployers.spi.deployer.helpers.AbstractParsingDeployerWithOutput.createMetaData(AbstractParsingDeployerWithOutput.java:322)
     at org.jboss.deployers.spi.deployer.helpers.AbstractParsingDeployerWithOutput.createMetaData(AbstractParsingDeployerWithOutput.java:294)
     at org.jboss.deployers.spi.deployer.helpers.AbstractParsingDeployerWithOutput.deploy(AbstractParsingDeployerWithOutput.java:234)
     at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179)
     at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1454)
     at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1172)
     at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1113)
     at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
     at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)
     at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)
     at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
     at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)
     at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)
     at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)
     at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:789)
     at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:699)
     at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)
     at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)
     at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)
     at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403)
     at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
     at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)
     at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)
     at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
     at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)
     at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:778)
     at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:543)
     at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308)
     at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:256)
     at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)
     at org.jboss.Main.boot(Main.java:221)
     at org.jboss.Main$1.run(Main.java:556)
     at java.lang.Thread.run(Thread.java:619)

Resolution

Locally host any additional xsds needed but not already provided by JBoss for XML validation.

Root Cause

The application being deployed references some remote resource/xsd for XML validation. This resource could be having its own issues causing it to be unreachable, thus causing the socket and this thread to hang during deployment.

Diagnostic Steps

Troubleshoot using thread dumps and additional steps mentioned in Java application unresponsive and JBoss is slow to start
Gather verbose deployment logging during the issue on the following categories:
- org.jboss.deployment
- org.jboss.deployers
- org.apache.xerces
Determine what application or applications hang like this during deployment. What resources do they use for xml validation? Does it refer to anything not provided by JBoss's jboss-metadata.jar that is remotely hosted?

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.