High CPU and concurrency issues running Seam hot deploy/debug mode in production

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP)
    • 4.x
    • 5.x
  • Seam 1.2.x
  • Seam 2.x

Issue

  • When load testing against our Seam app with 10 users the server gets cpu 100% utilized.
  • We see page actions which are not fired at random.

Resolution

  • Seam should not be run in debug mode in production due to concurrency and other issues. This doesn't mean it can't be done, it can if one is willing to assume the risk of the occasional unexplained glitch when Seam comes under load. See the Seam This content is not included.documentation for a list of limitations when running Seam in debug mode.
  • Set debug="false" in <core:init /> in components.xml.

Diagnostic Steps

  • Check to see if debug="true" in <core:init /> in components.xml.
  • Analyze the thead dumps obtained following the Java application high CPU document and check for thread contention with the stack trace showing references to the jboss.seam.debug package. For example: 
    "ajp-192.0.0.222-8009-27" daemon prio=1 tid=0x00002aabe164a170 nid=0x3c6f waiting for monitor entry [0x00000000450f3000..0x00000000450f5b90]
    at org.apache.xerces.impl.dv.DTDDVFactory.getInstance(Unknown Source)
    - waiting to lock <0x00002aaaab27b7b0> (a java.lang.Class)
    at org.apache.xerces.parsers.XML11Configuration. (Unknown Source)
    at org.apache.xerces.parsers.XIncludeAwareParserConfiguration. (Unknown Source)
    at org.apache.xerces.parsers.XIncludeAwareParserConfiguration. (Unknown Source)
    at sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
    at java.lang.Class.newInstance0(Class.java:350)
    at java.lang.Class.newInstance(Class.java:303)
    at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source)
    at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
    at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
    at org.apache.xerces.parsers.SAXParser. (Unknown Source)
    at org.apache.xerces.parsers.SAXParser. (Unknown Source)
    at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser. (Unknown Source)
    at org.apache.xerces.jaxp.SAXParserImpl. (Unknown Source)
    at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParser(Unknown Source)
    at org.dom4j.io.JAXPHelper.createXMLReader(JAXPHelper.java:50)
    at org.dom4j.io.SAXHelper.createXMLReaderViaJAXP(SAXHelper.java:125)
    at org.dom4j.io.SAXHelper.createXMLReader(SAXHelper.java:78)
    at org.dom4j.io.SAXReader.createXMLReader(SAXReader.java:894)
    at org.dom4j.io.SAXReader.getXMLReader(SAXReader.java:715)
    at org.dom4j.io.SAXReader.read(SAXReader.java:435)
    at org.dom4j.io.SAXReader.read(SAXReader.java:343)
    at org.jboss.seam.util.XML.getRootElement(XML.java:24)
    at org.jboss.seam.navigation.Pages.getDocumentRoot(Pages.java:1093)
    at org.jboss.seam.navigation.Pages.parse(Pages.java:1083)
    at org.jboss.seam.navigation.Pages.parsePages(Pages.java:176)
    at org.jboss.seam.navigation.Pages.initialize(Pages.java:148)
    at org.jboss.seam.init.Initialization.redeploy(Initialization.java:780)
    at org.jboss.seam.debug.hot.HotDeployFilter.doFilter(HotDeployFilter.java:44)
    at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
    at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:73)
    at org.jboss.seam.web.IdentityFilter.doFilter(IdentityFilter.java:38)
    at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
    at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:90)
    at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
    at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
    at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
    at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
    at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
    at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:177)
    at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:267)
    at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:380)
    at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:507)
    at org.jboss.seam.web.Ajax4jsfFilter.doFilter(Ajax4jsfFilter.java:56)
    at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
    at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58)
    at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
    at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
    at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
    at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
    at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
    at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:437)
    at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:381)
    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
    at java.lang.Thread.run(Thread.java:595)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.