EAP 8 and 7 container image jmx-exporter agent

Environment

• Red Hat Enterprise Application Platform
  • 7.x
  • 8.x
• Red hat OpenShift Container Platform (OCP)
  • 4.x

Issue

• What is EAP 7's jmx-exporter how to use it and how to set it up?
• What is AB_PROMETHEUS_ENABLE and how to set it?

Resolution

JMX exporter is an agent that is enabled via AB_PROMETHEUS_ENABLE variable in OCP.
The agent is a jar file located in /usr/share/java/prometheus-jmx-exporter/ called jmx_prometheus_javaagent.jar in the EAP image.
The variable AB_PROMETHEUS_ENABLE results in:
If set to true, this variable activates the jmx-exporter java agent that exposes Prometheus format metrics and default is set to false.

See solution How to enable Prometheus and statistics in EAP for OpenShift? - for more information.

More information

This information is collected by the exporter and pushed to the prometheus instance: EAP 7 in OCP

customizing the jmx-exporter-config.yaml:

User can add more attributes to the http-listener of the jmx-exporter-config.yaml - as below:

- pattern: "jboss.as<subsystem=undertow, *server=(.+), *http-listener=(.+)><>(request_count|processing_time|max_processing_time): (.+)"

Instead of the default:

- pattern: "jboss.as<subsystem=undertow, *server=(.+), *http-listener=(.+)><>(request_count): (.+)"

Where processing_time would be total processing time of all requests on the listener with max_processing_time being the peak response time. User can also add error_count if you wish to track the number of error responses.

Usage in OCP 4

1. First create the deployment and set the environment variables as above. So the metrics will be available on localhost/metrics.

   $ oc set env dc/$DC_NAME AB_PROMETHEUS_ENABLE=true
   $ oc set env dc/$DC_NAME JAVA_OPTS_APPEND=-Dwildfly.statistics-enabled=true
   ### access:
   $ curl http://localhost:9799/metrics
   

   See solution How to enable Prometheus and statistics in EAP for OpenShift?.

2. To use Prometheus on the OCP console, besides exporting, create a Service Monitor similar to the one below:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: prometheus-monitor
  name: prometheus-monitor     <--- set the name
  namespace: eap-test          <--- set the namespace
spec:
  endpoints:
  - interval: 30s
    port: admin
    scheme: http
  selector:
    matchLabels:
      app: eap74-basic-s2i      <-- set the app's label to watch
...
$ oc create -f servicemonitor.yaml 
servicemonitor.monitoring.coreos.com/prometheus-example-monitor created

If users are not using the prometheus exported metrics, you can disable the agent without major incidents/implications to the EAP functionality itself, it is just an observability tool.
The guide above brings more details on the exposition port and default configuration for example and how to set it for outside EAP's container as well.

Root Cause

EAP 8 and EAP 7 both have an agent jmx_prometheus_javaagent.jar located in /usr/share/java/prometheus-jmx-exporter/. The configuration file jmx-exporter-config.yaml will be located in /opt/jboss/container/prometheus/etc/.

EAP 7 setting:

EAP 7 template/deploymentconfig has the following configuration:

EAP 8

EAP 7 has the Prometheus agent that exposed the metrics, however in EAP 8 exposes the metrics endpoint by itself. And therefore, there is no need for the AB_PROMETHEUS_ENABLE environment variable and associated ones.

EAP JMX

EAP JMX exporter is based on Content from github.com is not included.jmx_exporter, which is added to the jar execution - together with the configuration file passed as a yaml:

java -javaagent:./jmx_prometheus_javaagent.jar=9799:configuration.yaml jboss-modules.jar ...

Prometheus' Properties

The following properties can be set on the Prometheus yaml to set the agent's behavior:

Example from the default prometheus configuration on EAP 7 image:

- pattern: "jboss.as<subsystem=messaging-activemq, *server=(.+), *(.+)=(.+)><>(consumer_count|delivering_count|durable_message_count|durable_subscription_count|message_count|non_durable_message_count|non_durable_subscription_count|scheduled_count|subscription_count): (.+)"
  name: jboss_messaging_activemq_$4
  help: Messaging JMS Queue Metrics $4
  value: $5
  valueFactor: 1
  labels:
    server: $1
    $2: $3
  attrNameSnakeCase: true
  type: GAUGE

Example of EAP execution with the prometheus jar:

sun.java.command = /opt/eap/jboss-modules.jar -javaagent:/usr/share/java/jolokia-jvm-agent/jolokia-jvm.jar=config=/opt/jboss/container/jolokia/etc/jolokia.properties -javaagent:/usr/share/java/prometheus-jmx-exporter/jmx_prometheus_javaagent.jar=9799:/opt/jboss/container/prometheus/etc/jmx-exporter-config.yaml -mp /opt/eap/modules org.jboss.as.standalone -Djboss.home.dir=/opt/eap ...

Prometheus jars

There are two prometheus jars on the image:

• Inside m2 galleon: /opt/jboss/container/wildfly/s2i/galleon/galleon-m2-repository/org/yaml/snakeyaml/redhat-00001/snakeyaml-id.jar - this is what EAP uses, which not present on JWS images.

• /opt/jboss/container/prometheus/jmx_prometheus_javaagent.jar -> /usr/share/java/prometheus-jmx-exporter/jmx_prometheus_javaagent.jar - this is prometheus which is a shaded jar

In the matter of m2 jars, the installed server is a thin server, no jar are installed under the /modules dir, they are loaded from the ./m2 repository.

Diagnostic Steps

1. Verify the deployment/deploymentconfig for the variables above;
2. Verify the ServiceMonitor creation for Prometheus observability on the OCP monitor: $oc get servicemonitor
3. To access the metrics as a file inside a container: $ curl -o metrics.txt http://localhost:9799/metrics

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.