EAP 7 creates too many jaeger threads during start up
Environment
- JBoss Enterprise Application Platform (EAP) 7.3.x
Issue
- Trying to start JBoss, we reach OutOfMemoryError: unable to create new native thread because thousands of jaeger threads like below are created:
"jaeger.RemoteReporter-FlushTimer" #5976 daemon prio=5 os_prio=0 tid=0x000055cc87bad800 nid=0xa7e1 in Object.wait() [0x00007f85fc4fb000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000003552ba340> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
"jaeger.RemoteReporter-QueueProcessor" #5975 daemon prio=5 os_prio=0 tid=0x000055cc87bab800 nid=0xa7e0 waiting on condition [0x00007f85fc57c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000003552ba110> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at io.jaegertracing.internal.reporters.RemoteReporter$QueueProcessor.run(RemoteReporter.java:173)
at java.lang.Thread.run(Thread.java:748)
Resolution
- Remove these tech preview features from the JBoss standalone or domain configuration:
<extension module="org.wildfly.extension.microprofile.config-smallrye"/>
<extension module="org.wildfly.extension.microprofile.health-smallrye"/>
<extension module="org.wildfly.extension.microprofile.metrics-smallrye"/>
<extension module="org.wildfly.extension.microprofile.opentracing-smallrye"/>
...
<subsystem xmlns="urn:wildfly:microprofile-config-smallrye:1.0"/>
<subsystem xmlns="urn:wildfly:microprofile-health-smallrye:2.0" security-enabled="false" empty-liveness-checks-status="${env.MP_HEALTH_EMPTY_LIVENESS_CHECKS_STATUS:UP}" empty-readiness-checks-status="${env.MP_HEALTH_EMPTY_READINESS_CHECKS_STATUS:UP}"/>
<subsystem xmlns="urn:wildfly:microprofile-metrics-smallrye:2.0" security-enabled="false" exposed-subsystems="*" prefix="${wildfly.metrics.prefix:jboss}"/>
<subsystem xmlns="urn:wildfly:microprofile-opentracing-smallrye:1.0"/>
- Update to EAP 7.4.0+ where these tech previews are removed
Components
Category
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.