Scrape Prometheus metrics using TLS in ServiceMonitor configuration

In OpenShift, we can leverage the service-ca controller to generate service serving certificates.

This article assumes some familiarity with the ServiceMonitor custom resource. It also assumes user workload monitoring has been enabled

The ServiceMonitor spec is defined This page is not included, but the link has been rewritten to point to the nearest parent document.here and it should be noted, that for user workload monitoring, the following fields Content from github.com is not included.will be rejected by prometheus-operator.

• caFile
• certFile
• keyFile

Let's take our Content from github.com is not included.instrumented sample application and configure a TLS enabled proxy sidecar.

First we create the service with the required annotation to generate the server certificate secret.

We also create a ConfigMap which will have the CA bundle injected for usage in the client.

cat <<EOF | oc apply -f -
apiVersion: v1
kind: Service
metadata:
  namespace: ns1
  annotations:
    service.beta.openshift.io/serving-cert-secret-name: sample-app-tls
  labels:
    app: prometheus-example-app
  name: prometheus-example-app
spec:
  type: ClusterIP
  ports:
  - name: https
    port: 8443
    targetPort: 8443
  selector:
    app: prometheus-example-app
---
apiVersion: v1
data: {}
kind: ConfigMap
metadata:
  namespace: ns1
  annotations:
    service.beta.openshift.io/inject-cabundle: "true"
  name: serving-certs-ca-bundle
EOF

Create a ConfigMap for the sidecar proxy with the following content:

cat << EOF > nginx.conf
worker_processes  1;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    sendfile        on;
    keepalive_timeout  65;
    server {
      listen  0.0.0.0:8443 ssl;
      server_name  prometheus-example-app;
      ssl_certificate /app/cert/tls.crt;
      ssl_certificate_key  /app/cert/tls.key;
      ssl_protocols TLSv1.2;
      ssl_ciphers EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:!EECDH+3DES:!RSA+3DES:!MD5;
      ssl_prefer_server_ciphers on;
      location / {
        proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
        proxy_set_header Host \$http_host;
        proxy_pass http://localhost:9100/;
      }
    }
}
EOF
oc -n ns1 create cm nginx-sidecar-conf --from-file=nginx.conf=./nginx.conf

Create the Deployment mounting the required configuration as volumes:

cat <<EOF | oc apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: ns1
  labels:
    app: prometheus-example-app
  name: prometheus-example-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-example-app
  template:
    metadata:
      labels:
        app: prometheus-example-app
    spec:
      containers:
      - name: prometheus-example-app
        image: ghcr.io/rhobs/prometheus-example-app:0.3.0
        args: ["--bind=0.0.0.0:9100"]
        ports:
        - name: web
          containerPort: 9100
      - name: tls-sidecar
        ports:
        - name: https
          containerPort: 8443
        image: bitnami/nginx
        imagePullPolicy: IfNotPresent
        volumeMounts:
          - name: tls-config
            mountPath: /app/cert
          - name: config-volume
            mountPath: /opt/bitnami/nginx/conf/nginx.conf
            subPath: nginx.conf
      volumes:
      - name: tls-config
        secret:
          secretName: sample-app-tls
      - name: config-volume
        configMap:
          name: nginx-sidecar-conf
EOF

Create the ServiceMonitor to discover and scrape the application over TLS via the proxy.

Note in this example the namespace ns1 is defined in the serverName.

This must match the namespace the resources are created in, since Prometheus will check against the Service that it scrapes through.

cat <<EOF | oc apply -f -
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  namespace: ns1
  labels:
    app: prometheus-example-app
  name: prometheus-example-app
spec:
  endpoints:
  - interval: 30s
    port: https
    scheme: https
    tlsConfig:
      ca:
        configMap:
          name: serving-certs-ca-bundle
          key: service-ca.crt
      serverName: prometheus-example-app.ns1.svc
  selector:
    matchLabels:
        app: prometheus-example-app
EOF

Additional resources

• Setting up metrics collection for user defined-projects