Managed server can stop working and become blocked by logger when started using a JBoss ON management plug-in

Environment

• Red Hat JBoss Operations Network (ON) 3.2, 3.3
• Red Hat JBoss Enterprise Application Platform (EAP) 6 management plug-in for JBoss ON
• Affected server resource was started using its management plug-in's Start operation
• Affected server resource is configured to log to the console using standard out (stdout) or standard error (stderr)

Issue

• Server instance or resource hangs in native FileOutputStream.writeBytes call writing to server.log file and becomes blocked and stops responding after a period of time
• Server resource started using its Start operation and logging to console will eventually stop working

Resolution

Disable console logging for resources that will be started from JBoss ON using their management plug-in's Start operation.

Root Cause

The resource start operation for many managed servers utilize the JBoss ON agent's utility class org.rhq.core.util.exec.ProcessExecutor. By default, console or process output is captured and redirected to a buffer to be returned with the operation's result. However, if a an unexpected error occurs while capturing the console output, the redirection stops but the file descriptor is not closed or released. This results in the output from the managed server buffering until the buffer is full, at which point the logger of the managed server is blocked. Any logging call in the managed server will result in the thread, in which the logging call was made, hanging.

The issue with the stream not being properly closed when the stream redirection thread terminates has been captured in This content is not included.Red Hat Bugzilla 1212933 and will be addressed in a future release of the JBoss ON agent.

The unexpected error that is causing the stream redirection thread to prematurely terminate is an OutOfMemoryError. This condition is a result of the console output continuing to be captured even after the operation has finished executing. This issue has been captured in This content is not included.Red Hat Bugzilla 1212950 and will be addressed in a future release of the JBoss EAP 6 management plug-in.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.