DataGrid 8 Operator Webhooks in OCP 4

Solution Verified - Updated

Environment

  • Red hat OpenShift Container Platform (OCP)
    • 4.x
  • Red Hat Data Grid (RHDG)
    • 8.x

Issue

  • What are the webhooks on DataGrid Operator?
  • What are the validation webhooks on DataGrid Operator?
  • Data Grid Webhook has issues:
Error "failed calling webhook "minfinispan.kb.io": failed to call webhook: Post "https://infinispan-operator-controller-manager-service.mot-int-06.svc:443/mutate-infinispan-org-v1-infinispan?timeout=10s": context deadline exceeded" for field "undefined"

Resolution

Data Grid Operator 8.3.9 introduces the following validating and mutating webhooks:

$ oc get ValidatingWebhookConfiguration
NAME                                  WEBHOOKS   AGE
(...)
vbackup.kb.io-rcb5p                   1          69m  <------------------------ backup
vbatch.kb.io-rn25f                    1          69m  <------------------------ batch
vcache.kb.io-zg5kh                    1          69m  <------------------------ cache
vinfinispan.kb.io-kxk29               1          69m  <------------------------ infinispan
vrestore.kb.io-vxtt5                  1          69m  <------------------------ restore

Data Grid Operator 8.3.9 introduces:

$ oc get MutatingWebhookConfiguration
mbackup.kb.io-nm9dd       1          4s <------------------------ backup
mcache.kb.io-vzn6h        1          4s    <------------------------ cache
minfinispan.kb.io-x8gfk   1          4s   <------------------------ infinispan
mrestore.kb.io-mnbgl      1          4s   <------------------------ restore

Example

$ oc get MutatingWebhookConfiguration minfinispan.kb.io-x8gfk -o yaml
    service:
      name: infinispan-operator-controller-manager-service
      namespace: openshift-operators
      path: /mutate-infinispan-org-v1-infinispan
      port: 443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: minfinispan.kb.io
  namespaceSelector: {}
  objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - infinispan.org
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - infinispans
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10

The above is a mutating webhook called minfinispan where match CREATE or UPDATE requests to v1 for the infinispan resources with 10 seconds timeout. So when api gets a CREATE or UPDATE request from infinispans resources, it will run the webhook to mutate the request (i.e. modify) data based on the condition. The scope above is not namespace-bounded, but rather cluster-wide.

Root Cause

DG Operator 8.3.9 introduces the webhooks where the OLM/CO installs them.
The MutatingWebhookConfiguration and ValidatingWebhookConfiguration are defined as part of the Operator CSV manifest. The mutation webhooks are called m+api, and the validation ones are called v+api.

### Validating webhook:
$ oc api-resources | grep -i valid
validatingwebhookconfigurations                           admissionregistration.k8s.io/v1               false        ValidatingWebhookConfiguration
### Mutating webhook:
$ oc api-resources | grep -i mutat
mutatingwebhookconfigurations                             admissionregistration.k8s.io/v1               false        MutatingWebhookConfiguration

Difference between Mutator Webhok vs Validation Webhook:

  • ValidatingWebhook: validates a certain configuration
  • MutatingWebhook: modify the original request for a custom one

The workflow of webhooks is highlighted as below:

  1. Resource gets submitted to k8s api-server
  2. api-server calls webhook implementation endpoint
  3. api-server accepts/rejects resource based upon response from 2

So if the API-server cannot complete step 2, because network traffic can't flow as expected to/from the webhook endpoint, then it has no choice but to reject the request
The webhook definitions are cluster-scoped, however, the webhook implementations must reside in a namespace. In this case, the implementation of the webhook(s) is provided by the DG Operator pod, so communication between the API-server and the Operator pods has to be possible. Therefore, netpol can play a role in this matter given step 2. Usage of network policies must insure the ingress of the DG operator pod works.

So example communication:

mutating webhook to service --> svc/infinispan-operator-controller-manager-service --> to pod -->infinispan-operator-controller-manager-pod in openshift-operators

Communication

The DG Operator must reach the API server's webhook all the time.
Workflow:

mutating webhook to service --> svc/infinispan-operator-controller-manager-service --> to pod -->infinispan-operator-controller-manager in DG Operator NS installation
  1. From mutating webhook yaml we track the service
  2. The service is called infinispan-operator-controller-manager-service and path: /mutate-infinispan-org-v1-infinispan
  3. and that service points to the dg operator manager

In summary:
a- The namespace that has the DG Operator must allow ingress (not necessarily egress but ingress) so then the DG Operator's service can listen to the request
b- In case the DG Operator and the DG (as in Infinispan CR) is on the same namespace, you need to make sure the allow-from-same-namespace is used.
c- If you have any network policies, you are denied everything else. That's the definition of the network policy. The network policies allow rules, so if you do not allow you are denied on the spot. That's because kubernets does not have a deny rule.
d- DG Operator must allow ingress for webhook validation.

The corollary from the above is if the DG Operator is in the same namespace as the CR, AND you have one network policy so then you MUST have an allow-from-same-namespace.

Finally, if you have a cluster-wide DG Operator, for instance on openshift-operators namespace, you need to make sure that the namespace has an allow ingress communicatoin for other namespaces.

Diagnostic Steps

Example of mutate issue:

Error "failed calling webhook "minfinispan.kb.io": failed to call webhook: Post "https://infinispan-operator-controller-manager-service.svc:443/mutate-infinispan-org-v1-infinispan?timeout=10s": context deadline exceeded" for field "undefined"

$ oc get MutatingWebhookConfigurations/minfinispan -o yaml:

service: 
     name: infinispan-operator-controller-manager-service 
     namespace: openshift-operators 
     path: /mutate-infinispan-org-v1-infinispan 
     port: 443

$ oc get svc/infinispan-operator-controller-manager-service -o yaml -n openshift-operators

selector: 
   app.kubernetes.io/name: infinispan-operator 
   control-plane: controller-manager

$ oc get pods -n openshift-operators --show-labels | grep infinispan-operator 

infinispan-operator-controller-manager-7f5c84c5f9-vqcwq  1/1    Running  7 (25m ago)  84m  app.kubernetes.io/name=infinispa
n-operator,control-plane=controller-manager,pod-template-hash=7f5c84c5f9

Enabling debug logs on the Catalog Operator shows:

$ oc logs catalog-operator-6f8d5c4c9-x68bc | grep -i infinispan | grep -i webhook
...
"apiservicedefinitions\":{},\"webhookdefinitions\":[{\"generateName\":\"vinfinispan.kb.io\",\"type\":\"ValidatingAdmissionWebhook\",\"deploymentName\":\"infinispan-operator-controller-manager\",\"containerPort\":443,\"targetPort\":9443,\"rules\":[{\"operations\":[\"CREATE\",\"UPDATE\"],\"apiGroups\":[\"infinispan.org\"],\"apiVersions\":[\"v1\"],\"resources\":[\"infinispans\"]}],\"failurePolicy\":\"Fail\",\"sideEffects\":\"None\",\"admissionReviewVersions\":[\"v1\",\"v1beta1\"],\"webhookPath\":\"/validate-infinispan-org-v1-infinispan\"},{\"generateName\":\"vbackup.kb.io\",\"type\":\"ValidatingAdmissionWebhook\",\"deploymentName\":\"infinispan-operator-controller-manager\",\"containerPort\":443,\"targetPort\":9443,\"rules\":[{\"operations\":[\"CREATE\",\"UPDATE\"],\"apiGroups\":[\"infinispan.org\"],\"apiVersions\":[\"v2alpha1\"],\"resources\":[\"backups\"]}],\"failurePolicy\":\"Fail\",\"sideEffects\":\"None\",\"admissionReviewVersions\":[\"v1\",\"v1beta1\"],\"webhookPath\":\"/validate-infinispan-org-v2alpha1-backup\"},{\"generateName\":\"vbatch.kb.io\",\"type\":\"ValidatingAdmissionWebhook\",\"deploymentName\":\"infinispan-operator-controller-manager\",\"containerPort\":443,\"targetPort\":9443,\"rules\":[{\"operations\":[\"CREATE\",\"UPDATE\"],\"apiGroups\":[\"infinispan.org\"],\"apiVersions\":[\"v2alpha1\"],\"resources\":[\"batches\"]}],\"failurePolicy\":\"Fail\",\"sideEffects\":\"None\",\"admissionReviewVersions\":[\"v1\",\"v1beta1\"],\"webhookPath\":\"/validate-infinispan-org-v2alpha1-batch\"},{\"generateName\":\"vcache.kb.io\",\"type\":\"ValidatingAdmissionWebhook\",\"deploymentName\":\"infinispan-operator-controller-manager\",\"containerPort\":443,\"targetPort\":9443,\"rules\":[{\"operations\":[\"CREATE\",\"UPDATE\"],\"apiGroups\":[\"infinispan.org\"],\"apiVersions\":[\"v2alpha1\"],\"resources\":[\"caches\"]}],\"failurePolicy\":\"Fail\",\"sideEffects\":\"None\",\"admissionReviewVersions\":[\"v1\",\"v1beta1\"],\"webhookPath\":\"/validate-infinispan-org-v2alpha1-cache\"},{\"generateName\":\"vrestore.kb.io\",\"type\":\"ValidatingAdmissionWebhook\",\"deploymentName\":\"infinispan-operator-controller-manager\",\"containerPort\":443,\"targetPort\":9443,\"rules\":[{\"operations\":[\"CREATE\",\"UPDATE\"],\"apiGroups\":[\"infinispan.org\"],\"apiVersions\":[\"v2alpha1\"],\"resources\":[\"restores\"]}],\"failurePolicy\":\"Fail\",\"sideEffects\":\"None\",\"admissionReviewVersions\":[\"v1\",\"v1beta1\"],\"webhookPath\":\"/validate-infinispan-org-v2alpha1-restore\"},{\"generateName\":\"minfinispan.kb.io\",\"type\":\"MutatingAdmissionWebhook\",\"deploymentName\":\"infinispan-operator-controller-manager\",\"containerPort\":443,\"targetPort\":9443,\"rules\":[{\"operations\":[\"CREATE\",\"UPDATE\"],\"apiGroups\":[\"infinispan.org\"],\"apiVersions\":[\"v1\"],\"resources\":[\"infinispans\"]}],\"failurePolicy\":\"Fail\",\"sideEffects\":\"None\",\"admissionReviewVersions\":[\"v1\",\"v1beta1\"],\"webhookPath\":\"/mutate-infinispan-org-v1-infinispan\"},{\"generateName\":\"mbackup.kb.io\",\"type\":\"MutatingAdmissionWebhook\",\"deploymentName\":\"infinispan-operator-controller-manager\",\"containerPort\":443,\"targetPort\":9443,\"rules\":[{\"operations\":[\"CREATE\",\"UPDATE\"],\"apiGroups\":[\"infinispan.org\"],\"apiVersions\":[\"v2alpha1\"],\"resources\":[\"backups\"]}],\"failurePolicy\":\"Fail\",\"sideEffects\":\"None\",\"admissionReviewVersions\":[\"v1\",\"v1beta1\"],\"webhookPath\":\"/mutate-infinispan-org-v2alpha1-backup\"},{\"generateName\":\"mcache.kb.io\",\"type\":\"MutatingAdmissionWebhook\",\"deploymentName\":\"infinispan-operator-controller-manager\",\"containerPort\":443,\"targetPort\":9443,\"rules\":[{\"operations\":[\"CREATE\",\"UPDATE\"],\"apiGroups\":[\"infinispan.org\"],\"apiVersions\":[\"v2alpha1\"],\"resources\":[\"caches\"]}],\"failurePolicy\":\"Fail\",\"sideEffects\":\"None\",\"admissionReviewVersions\":[\"v1\",\"v1beta1\"],\"webhookPath\":\"/mutate-infinispan-org-v2alpha1-cache\"},{\"generateName\":\"mrestore.kb.io\",\"type\":\"MutatingAdmissionWebhook\",\"deploymentName\":\"infinispan-operator-controller-manager\",\"containerPort\":443,\"targetPort\":9443,\"rules\":[{\"operations\":[\"CREATE\",\"UPDATE\"],\"apiGroups\":[\"infinispan.org\"],\"apiVersions\":[\"v2alpha1\"],\"resources\":[\"restores\"]}],\"failurePolicy\":\"Fail\",\"sideEffects\":\"None\",\"admissionReviewVersions\":[\"v1\",\"v1beta1\"],\"webhookPath\":\"/mutate-infinispan-org-v2alpha1-restore\"}],\"minKubeVersion\":\"1.11.0\",\"displayName\":\"Data Grid\",\"description\":\"Red Hat Data Grid is a distributed, in-memory data store built from tried and trusted open-source technology.\\n\\n### Data Grid\\n* **Schemaless structure:** Store objects in key/value pairs.\
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.