Receiving telemetry data

Red Hat build of OpenTelemetry 3.9

Connecting instrumented applications and the Collector

Red Hat OpenShift Documentation Team

Abstract

Learn how to configure the OpenTelemetry Collector to receive telemetry from instrumented applications.

Chapter 1. Receiving telemetry

After setting up the OpenTelemetry Collector and instrumenting your application, you need to connect the instrumentation and OpenTelemetry Collector so that the OpenTelemetry Collector can receive telemetry from the instrumentation.

1.1. Receiving telemetry from multiple clusters

If you need the Collector to receive telemetry from multiple remote clusters, create one OpenTelemetry Collector instance in each one of the remote clusters, and then have all of their telemetry forwarded to a central OpenTelemetry Collector instance.

Prerequisites

The Red Hat build of OpenTelemetry Operator is installed.
The Tempo Operator is installed.
A TempoStack instance is deployed on the cluster.
The following mounted certificates: Issuer, self-signed certificate, CA issuer, client and server certificates. To create any of these certificates, see step 1.

Procedure

Mount the following certificates in the OpenTelemetry Collector instance, skipping already mounted certificates.

An Issuer to generate the certificates by using the cert-manager Operator for Red Hat OpenShift.
```
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}
```

A self-signed certificate.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ca
spec:
  isCA: true
  commonName: ca
  subject:
    organizations:
      - <your_organization_name>
    organizationalUnits:
      - Widgets
  secretName: ca-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-issuer
    kind: Issuer
    group: cert-manager.io

A CA issuer.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: test-ca-issuer
spec:
  ca:
    secretName: ca-secret

The client and server certificates.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: server
spec:
  secretName: server-tls
  isCA: false
  usages:
    - server auth
    - client auth
  dnsNames:
  - "otel.observability.svc.cluster.local" 1
  issuerRef:
    name: ca-issuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: client
spec:
  secretName: client-tls
  isCA: false
  usages:
    - server auth
    - client auth
  dnsNames:
  - "otel.observability.svc.cluster.local" 2
  issuerRef:
    name: ca-issuer

1: List of exact DNS names to be mapped to a solver in the server OpenTelemetry Collector instance.
2: List of exact DNS names to be mapped to a solver in the client OpenTelemetry Collector instance.

Create a service account for the OpenTelemetry Collector instance.
Example ServiceAccount
```
apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-collector-deployment
```

Create a cluster role for the service account.

Example ClusterRole

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector
rules:
  1
  2
- apiGroups: ["", "config.openshift.io"]
  resources: ["pods", "namespaces", "infrastructures", "infrastructures/status"]
  verbs: ["get", "watch", "list"]

1: The k8sattributesprocessor requires permissions for pods and namespace resources.
2: The resourcedetectionprocessor requires permissions for infrastructures and status.

Bind the cluster role to the service account.

Example ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-collector
subjects:
- kind: ServiceAccount
  name: otel-collector-deployment
  namespace: otel-collector-<example>
roleRef:
  kind: ClusterRole
  name: otel-collector
  apiGroup: rbac.authorization.k8s.io

Create the YAML file to define the OpenTelemetryCollector custom resource (CR) in the edge clusters.

Example OpenTelemetryCollector custom resource for the edge clusters

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel
  namespace: otel-collector-<example>
spec:
  mode: daemonset
  serviceAccount: otel-collector-deployment
  config:
    receivers:
      jaeger:
        protocols:
          grpc: {}
          thrift_binary: {}
          thrift_compact: {}
          thrift_http: {}
      opencensus:
      otlp:
        protocols:
          grpc: {}
          http: {}
      zipkin: {}
    processors:
      batch: {}
      k8sattributes: {}
      memory_limiter:
        check_interval: 1s
        limit_percentage: 50
        spike_limit_percentage: 30
      resourcedetection:
        detectors: [openshift]
    exporters:
      otlphttp:
        endpoint: https://observability-cluster.com:443 1
        tls:
          insecure: false
          cert_file: /certs/server.crt
          key_file: /certs/server.key
          ca_file: /certs/ca.crt
    service:
      pipelines:
        traces:
          receivers: [jaeger, opencensus, otlp, zipkin]
          processors: [memory_limiter, k8sattributes, resourcedetection, batch]
          exporters: [otlphttp]
  volumes:
    - name: otel-certs
      secret:
        name: otel-certs
  volumeMounts:
    - name: otel-certs
      mountPath: /certs

1: The Collector exporter is configured to export OTLP HTTP and points to the OpenTelemetry Collector from the central cluster.

Create the YAML file to define the OpenTelemetryCollector custom resource (CR) in the central cluster.

Example OpenTelemetryCollector custom resource for the central cluster

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otlp-receiver
  namespace: observability
spec:
  mode: "deployment"
  ingress:
    type: route
    route:
      termination: "passthrough"
  config:
    receivers:
      otlp:
        protocols:
          http:
            tls: 1
              cert_file: /certs/server.crt
              key_file: /certs/server.key
              client_ca_file: /certs/ca.crt
    exporters:
      otlp/traces:
        endpoint: "tempo-<simplest>-distributor:4317" 2
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [otlp/traces]
  volumes:
    - name: otel-certs
      secret:
        name: otel-certs
  volumeMounts:
    - name: otel-certs
      mountPath: /certs

1: The Collector receiver requires the certificates listed in the first step.
2: The Collector exporter is configured to export OTLP and points to the Tempo distributor endpoint, which in this example is "tempo-simplest-distributor:4317" and already created.

Legal Notice

Except as otherwise noted below, the text of and illustrations in this documentation are licensed by Red Hat under the Creative Commons Attribution–Share Alike 3.0 Unported license . If you distribute this document or an adaptation of it, you must provide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, the Red Hat logo, JBoss, Hibernate, and RHCE are trademarks or registered trademarks of Red Hat, LLC. or its subsidiaries in the United States and other countries.

Linux® is the registered trademark of Linus Torvalds in the United States and other countries.

XFS is a trademark or registered trademark of Hewlett Packard Enterprise Development LP or its subsidiaries in the United States and other countries.

The OpenStack® Word Mark and OpenStack logo are trademarks or registered trademarks of the Linux Foundation, used under license.

All other trademarks are the property of their respective owners.