Failed deployments leak application classloaders in JBoss SuspendController and UndertowContainerProvider
Environment
- JBoss Enterprise Application Platform (EAP) 7.x
Issue
- We were try to deploy applications that had some recurring deployment errors. This then led to an OutOfMemoryError and the heap dump shows leaked application classloaders accumulating through references in the JBoss SuspendController:
Class Name | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
--------------------------------------------------------------------------------------------------------------------------------------------------
value org.jboss.as.server.suspend.SuspendController @ 0x4c2130d20 | 550 | 48 | 48,400 | 3,496
'- activities java.util.ArrayList @ 0x4c2130d68 | 550 | 24 | 48,400 | 3,336
'- elementData java.lang.Object[823] @ 0x56eda0df8 | 550 | 3,312 | 48,400 | 3,312
|- [148] org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$2 @ 0x579164a68 | 1 | 24 | 88 | 24
| '- this$0 org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService @ 0x51276a370| 1 | 176 | 88 | 2,424
| '- module org.jboss.modules.Module @ 0x51276a728 | 1 | 56 | 88 | 208,704
| '- moduleClassLoader org.jboss.modules.ModuleClassLoader @ 0x575161fb0 | 1 | 88 | 88 | 1,224,776
- And they are also accumulated in references from the UndertowContainerProvider:
Class Name | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
----------------------------------------------------------------------------------------------------------------------------------------------
[19] class io.undertow.websockets.jsr.UndertowContainerProvider @ 0xe2c84668| 5 | 24 | 480 | 2,121,200
'- webSocketContainers java.util.concurrent.ConcurrentHashMap @ 0xe2c85048 | 5 | 64 | 480 | 2,120,840
'- table java.util.concurrent.ConcurrentHashMap$Node[16] @ 0xe2c85088 | 5 | 80 | 480 | 2,120,776
|- [14] java.util.concurrent.ConcurrentHashMap$Node @ 0xe2c850d8 | 2 | 32 | 192 | 848,032
| |- key org.jboss.modules.ModuleClassLoader @ 0xe2ac72f0 | 1 | 96 | 96 | 3,848
| |- next java.util.concurrent.ConcurrentHashMap$Node @ 0xe1c45918 | 1 | 32 | 96 | 424,200
| | '- key org.jboss.modules.ModuleClassLoader @ 0xe2dd4400 | 1 | 96 | 96 | 3,904
Resolution
- Address any recurring deployment failures or otherwise avoid repeated attempts of the same failed deployments. Issue a periodic restart between the failed deployments.
- Update to EAP 7.4.8+
Root Cause
- This content is not included.WFLY-16412
- This content is not included.JBEAP-23627
- These references depend on a ServletContextListener to handle cleanup in any contextDestroyed event. But the ServletContextListener.contextDestroyed method is not invoked on listeners in the event of a deployment failure so anything depending on that for clean up is missed and leaves leaks.
Components
Category
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.