EAP 7 deployment with Service Mesh

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 7.x
• Red hat OpenShift Container Platform (OCP)
  • 4.x
• Red Hat Service Mesh

Issue

Does EAP 7 deployment works with Service Mesh?
How to troubleshot clustered EAP 7 deployed with OSSM?

Resolution

Before troubleshooting verify if the PeerAuthentication is not set for strict - preventing the EAP cluster to form

Troubleshooting EAP 7 deployment with OSSM:

Get the pods and verify istio basic pod logs:

$ oc get pod
NAME                                   READY   STATUS      RESTARTS   AGE
eap-app-1-8ccds                        1/1     Running     0          169m
eap-app-1-deploy                       0/1     Completed   0          169m
eap-app-1-flvf4                        1/1     Running     0          16m
eap-app-1-hg8bl                        1/1     Running     0          11m
eap-app-2-build                        0/1     Completed   0          170m
eap-app-build-artifacts-1-build        0/1     Completed   0          171m
grafana-68446ccdb8-kfnlv               2/2     Running     0          14m
istio-egressgateway-7bbddfcb4-wshpg    1/1     Running     0          14m
istio-ingressgateway-c845d4c87-tnsqk   1/1     Running     0          14m
istiod-basic-7c45f4459-82wbl           1/1     Running     0          14m <----- verify those logs
jaeger-77b57f5b77-jkdbz                2/2     Running     0          13m
kiali-f6748475d-jmww9                  1/1     Running     0          11m
prometheus-74c96988b-6xd8q             2/2     Running     0          14m

On the example above there are the following pods:

The name basic in the istio pod comes from the SMCP and the name eap-app comes from the deploymentconfig file of the application. The pod is called istiod, which manages and injects the envoy proxies inside the microservices pods - on this case the EAP app pods. Also note that istioD is the Pilot, Citadel, and Galley components combined as a pod.

Get the service name:

$ oc get svc
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                    AGE
eap-app                     ClusterIP   172.30.36.244    <none>        8080/TCP                                                   172m
eap-app-ping                ClusterIP   None             <none>        8888/TCP                                                   172m

And after deploying OSSM below will show:

$ oc get svc
NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                    AGE
eap-app                     ClusterIP   172.30.17.179    <none>        8080/TCP                                                   41m
eap-app-ping                ClusterIP   None             <none>        8888/TCP                                                   41m
grafana                     ClusterIP   172.30.9.4       <none>        3000/TCP                                                   7m53s
istio-egressgateway         ClusterIP   172.30.202.187   <none>        80/TCP,443/TCP                                             7m53s
istio-ingressgateway        ClusterIP   172.30.182.15    <none>        15021/TCP,80/TCP,443/TCP                                   7m53s
istiod-basic                ClusterIP   172.30.209.209   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP,8188/TCP             8m23s
jaeger-agent                ClusterIP   None             <none>        5775/UDP,5778/TCP,6831/UDP,6832/UDP                        7m51s
jaeger-collector            ClusterIP   172.30.233.158   <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP,4317/TCP,4318/TCP   7m50s
jaeger-collector-headless   ClusterIP   None             <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP,4317/TCP,4318/TCP   7m51s
jaeger-query                ClusterIP   172.30.249.166   <none>        443/TCP,16685/TCP                                          7m51s
kiali                       ClusterIP   172.30.253.41    <none>        20001/TCP,9090/TCP                                         6m53s
prometheus                  ClusterIP   172.30.167.111   <none>        9090/TCP                                                   8m10s
zipkin                      ClusterIP   172.30.22.99     <none>        9411/TCP                                                   7m53s

Get the service listed on istio's pod (istiod-basic):

$ oc logs istiod-basic-7c45f4459-82wbl | grep eap-app-ping
2022-12-09T02:25:32.758259Z	info	ads	Full push, new service eap-galleon/eap-app-ping.eap-galleon.svc.cluster.local
2022-12-09T02:28:25.104031Z	info	ads	Push debounce stable[36] 6 for config ServiceEntry/eap-galleon/eap-app-ping.eap-galleon.svc.cluster.local and 1 more configs: 101.04607ms since last change, 193.73896ms since last push, full=true
2022-12-09T02:28:37.048186Z	info	ads	Push debounce stable[40] 3 for config ServiceEntry/eap-galleon/eap-app-ping.eap-galleon.svc.cluster.local and 1 more configs: 100.099319ms since last change, 100.340712ms since last push, full=true
2022-12-09T02:28:41.174113Z	info	ads	Push debounce stable[43] 3 for config ServiceEntry/eap-galleon/eap-app-ping.eap-galleon.svc.cluster.local and 1 more configs: 100.370253ms since last change, 100.374713ms since last push, full=true
2022-12-09T02:28:57.773043Z	info	ads	Push debounce stable[44] 3 for config ServiceEntry/eap-galleon/eap-app-ping.eap-galleon.svc.cluster.local and 1 more configs: 100.32519ms since last change, 104.659631ms since last push, full=true

The name basic in the istio pod comes from the SMCP:

oc get smcp
NAME    READY   STATUS            PROFILES      VERSION   AGE
basic   9/9     ComponentsReady   ["default"]   2.3.0     9m45s

Root Cause

At this point OSSM integration with EAP is not fully supported.

Service Mesh itself is a pattern, Istio is the implementation.
The solution EAP 7 clustering deployments boundaries in OCP cluster describes the DNS_PING troubleshooting and in regards to EAP 7's deployment requirement for the DNS_PING service - the service name is set in standalone-openshift.xml.
The solution EAP 7 image clustering in OCP 4 describes all the details about clustering in EAP 7 image.

OSSM works by adding a sidecar in the application pod and another one to control the deployment, on this side car are there to proxy the traffic, where istio sidecar is the so called envoy. It is there to proxy the traffic to the container. So when traffic goes out: container - sidecar --> outside cluster.

Diagnostic Steps

1. Issues that can happen:

 [thread-18,ee,an-app] TRACE [org.jgroups.protocols.dns.DefaultDNSResolver] resolving DNS query: eap-app-ping of a type: A
[thread-18,ee,an-app] TRACE [org.jgroups.protocols.dns.DefaultDNSResolver] failed to resolve query eap-app-ping, ex: eap-app-ping: Name or service not known
[thread-18,ee,an-app] DEBUG [org.jgroups.protocols.dns.DNS_PING] an-app: no entries collected from DNS (in 3 ms)

2. Collect the regular must gather and the OSSM must gather - for the mg and the service mesh mg:

oc adm must-gather \ --image-stream=openshift/must-gather \ --image=registry.redhat.io/openshift-service-mesh/istio-must-gather-rhel8

The query above ^ will get you both a regular must-gather and a service-mesh gather as a bundle, and to do a regular must-gather it's just oc adm must-gather in comparison.

From the mg service mesh we'll get the SMMR SMCP and pod objects, istio-namespace project gather, openshift-operators namespace, subscription data and csvs

Solutions:

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.