Config Listener pod in DG 8 OCP 4

Environment

• Red hat OpenShift Container Platform (OCP)
  • 4.x
• Red Hat Data Grid (RHDG)
  • 8.x
  • Data Grid Operator: 8.3.4+ <--- ConfigListener introduction

Issue

What is the Config Listener pod?
How to interpret its logs?
How to specify the CPU and memory limits for the configuration listener?

Resolution

This setting creates a pod in each Infinispan cluster that allows Infinispan Operator to reconcile server-side modifications with Infinispan resources such as the Cache CR.
The listener pod consumes minimal resources and is enabled by default. Setting a value of false removes the listener pod and disables bi-directional reconciliation. One should do this only if the user do not need declarative Kubernetes representations of Infinispan resources created through the Infinispan Console, CLI, or client applications.

The configlistener log will show the CR operations and this can be useful to track down the type of cache being created/deleted even without the caches custom resources.

2022-08-14T04:02:19.790Z	DEBUG	Checking if Cache CR 'example-cache' is stale. ListenerCreated=true. CacheExists=false
2022-08-14T04:02:19.790Z	INFO	Marking stale Cache resource 'example-cache' for deletion
...
2022-08-14T04:02:23.271Z	DEBUG	No existing Cache CR found for Cache=ExampleCache, Cluster=rhdg-example-prod
2022-08-14T04:02:23.271Z	INFO	Creating Cache CR for 'ExampleCache'
distributedCache:
  backups:
    rhdg-a-backup:
      backup:
        stateTransfer:
          chunkSize: "512"
          maxRetries: "30"
          mode: AUTO
          timeout: "1200000"
          waitTime: "2000"
        strategy: ASYNC
  encoding:
    key:
      mediaType: application/x-java-serialized-object
    value:
      mediaType: application/x-java-serialized-object
  expiration:
    lifespan: "7200000"
  locking:
    acquireTimeout: "15000"
    concurrencyLevel: "1000"
    striping: "false"
  memory:
    storage: OFF_HEAP
  mode: SYNC
  owners: "2"
  remoteTimeout: "17500"
  stateTransfer:
    timeout: "60000"
  statistics: "true"

Troubleshooting

From the config listener pod logs:

1. no such host warnings are issued when the service cannot resolve. This occurs when no datagrid pods are ready, i.e. all pods are in the process of restarting, as there's nothing for the ConfigListener to connect to.
2. EOF warnings occur when an existing connection is lost, i.e. because the Pod the ConfigListener stream was connected to has been restarted.

Default

The default ConfigListener spec is:

spec:
  configListener:
    enabled: true
    logging:
      level: info

About the configlistner, the operator doesn't set resources limits, so kuberntes will deploy with default: a pod with one of one cpu and 1/4 of gb of memory. So the default is what ocp sets.

Interpreting config listener logs

2022-10-12T07:52:58.025Z    DEBUG    Checking if Cache CR 'example-cache' is stale. ListenerCreated=true. CacheExists=false
2022-10-12T07:52:58.025Z    INFO    Marking stale Cache resource 'example-cache' for deletion

The above can be interpreted as. It's just a check on startup to see if any outdated Cache CRs exist for the Infinispan cluster. For example:

1. A cache "example-cache" gets created on the server
2. ConfigListener creates the "example-cache" CR
3. ConfigListener is disabled for a while, or even the pod just restarts due to the k8s scheduler
4. "example-cache" CR is deleted on the server
5. If the "example-cache" cache is removed from Infinispan server via a user action (Console, HR, REST, user deletion)
6. ConfigListener is enabled/restarts
7. ConfigListener checks all CRs to see if they exist on the server. It sees "example-cache" doesn't, so it deletes it from k8s

This doesn't happen for Cache CRs that are created by the user, only those created by the ConfigListener.

Certificates

The ConfigListener deployment retrieves server authentication credentials from the -generated-operator-secret Secret created with the Infinispan CR, so there's no need for it to create any secrets of it's own.
There shouldn't be any secrets created for the ConfigListener deployment.

Known Issues - i.e. Bugs that can be hit by the ConfigListener

The bug This content is not included.JDG-5531 apply for the config-listener when defining a persistence tag on the cache. JDG-5531 can apply for yaml, xml, and json declaration - for the first one (yaml) it will break the configuration, whereas for the last two (xml and json) it will create a cache that doensn't get ready.

Root Cause

ConfigListener was introduced on DG Operator 8.3.4+ DG Operator 8.3.x - Release notes.
Enhanced bidirectional reconciliation was introduced on DG Operator 8.3.7.
Customization of Data Grid configlistener were introduced on DG Operator 8.4.x - see below table:

See Content from infinispan.org is not included.Infinispan Operator.
About gossip router pod, see solution Gossip Router pod in DG 8 OCP 4

Deployment order/Workflow

The deployment of the pods are done in the following order: DG manager Operator pods > Router pods (in case of Xsite) > Data Grid pods > Config listener.

The ConfigListener (CL) does the bi-directional reconciliation after the operator does the reconciliation, given it is only looking for changes so it starts after everything else, so it won't see irrelevant changes from the startup.

Adjusting cpu and memory in the conf listener

Currently (DG Operator 8.4) allows setting up ConfigListener CPU and memory settings, which was introduced by This content is not included.JDG-5908:

  configListener:
    cpu: '1'
    enabled: false
    logging:
      level: info
    memory: 1Gi

Supported values for logging are: debug, info, and error for the spec.configListener.logging.level.

The communication between the DG Operator and the DG cluster must always be kept clean, otherwise the ConfigLister will show:

$ oc logs example-infinispan-config-listener-cf7b4949b-4tqpt
2025-09-11T21:23:08.555Z INFO Starting Infinispan ConfigListener Version: RHDG-8.5.11.GA
2025-09-11T21:23:38.557Z FATAL failed to create client
main.main
 /opt/app-root/src/go/src/github.com/infinispan/infinispan-operator/main.go:55
runtime.main
 /usr/lib/golang/src/runtime/proc.go:272

Deleting the respective netpol that blocks the communication will make the ConfigListner (CL) come back after in a few minutes.

Diagnostic Steps

1. See config listener pod logs - Config Listener does not have configurable log level - this is being address on This content is not included.JDG-5794.
2. To troubleshoot - set the logging.level for debug:

spec:
  configListener:
    enabled: true
    logging:
      level: info

The supported values for logging are: debug, info, and error for the spec.configListener.logging.level.

In case this happens: Exception info cluster cannot be found, edit the configlistener deployment with: a- correct cluster name, b- the '-zap-log-level' line, c- the env variable INFINISPAN_OPERAND_VERSIONS correctly set. See the solution Configlistener not coming up after DG Operator upgrade.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.