JBoss EAP domain controller reaches OOME with large org.jboss.as.protocol.mgmt.ActiveOperationImpl overhead

Environment

• JBoss Enterprise Application Platform (EAP) 7.x

Issue

• Our JBoss EAP domain controller reached an OOME state. Inspecting the heap dump shows its heap was consumed by some org.jboss.as.protocol.mgmt.ActiveOperationImpl objects and other related controller objects that are similar in count:

Class Name                                                                          | Objects | Shallow Heap |  Retained Heap
------------------------------------------------------------------------------------------------------------------------------
org.jboss.as.protocol.mgmt.ActiveOperationImpl                                      |      70 |        4,128 | >= 735,671,128
org.jboss.as.controller.remote.TransactionalProtocolClientImpl$ExecuteRequestContext|      70 |        2,800 | >= 735,650,096
org.jboss.as.controller.remote.TransactionalProtocolClientImpl$OperationWrapper     |      70 |        1,680 | >= 735,574,808
org.jboss.as.controller.remote.TransactionalOperationImpl                           |      70 |        1,680 | >= 733,801,720
org.jboss.as.controller.OperationContextImpl                                        |      70 |       17,920 | >= 733,799,112
------------------------------------------------------------------------------------------------------------------------------

Resolution

• Identify why any client problem is overloading the controller with expensive operations and stop that problem client activity if possible

Root Cause

• Some client is sending several repetitive, expensive management operations, perhaps like the following to do repetitive reads of the same descriptions from all server instances:

/host=$HOSTNAME/server=*:read-resource-description

Diagnostic Steps

• From Eclipse MAT when reviewing the heap dump, use the following OQL to identify the client hostname/port that is sending expensive operations still in a WAITING or incomplete state:

SELECT channel.connection.connection.connection.conduit.socketChannel.localAddress.holder.addr.holder.hostName.value.toString(), channel.connection.connection.connection.conduit.socketChannel.localAddress.holder.addr.holder.port, channel.connection.connection.connection.conduit.socketChannel.remoteAddress.holder.addr.holder.hostName.value.toString(), channel.connection.connection.connection.conduit.socketChannel.remoteAddress.holder.addr.holder.port, status FROM org.jboss.as.protocol.mgmt.ActiveOperationImpl

• You may use the following OQL to identify the user principal behind the problem operations:

SELECT channel.connection.identity.principal.name.value.toString() FROM org.jboss.as.protocol.mgmt.ActiveOperationImpl 

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.