Troubleshooting OpenShift Container Platform 4.x: Middleware Containers

Updated

The solution below is only applicable for images that have: jmap, jcmd, and jps. There are images that do not have those tools, therefore use kill -3 instead

Enable script debugging output for an application pod

Sometimes it's helpful to debug the dynamic configuration scripts that are included and used in the Middleware Containers for Openshift. Enabling this script debugging will show the order in which scripts are executed as well as the environment variables and their values as they are processed by the scripts. Follow these steps to enable script debugging. NOTE: Enabling or disabling script debugging requires a pod deployment.

Add the following environment variables to your application's deploymentconfig. This can be done via the Openshift Web Console or via the oc command line utility.

The specific environment variables to use for script debugging are:

SCRIPT_DEBUG=true
PS4=+ $(basename $0):${LINENO} :

$ oc set env dc/the-deployment SCRIPT_DEBUG=true PS4='+ $(basename $0):${LINENO} :'

This may automatically redeploy the application pod. If not, be sure to redeploy the pod as this is necessary to get the debug output at deploy time.

Use this command to remove the environment variables from the application pod:

$ oc set env dc/the-deployment SCRIPT_DEBUG- PS4-

Remember to redeploy the pod if this does not occur automatically.

Gather a JBoss Diagnostics Report (JDR) from a running container

POD=your-pod-name; oc rsync $POD:$(oc exec $POD /opt/eap/bin/jdr.sh | grep "JDR location" | awk '{print $3}') .

Save pod logs to a file

  • from running container
    $ oc logs $POD > pod.log

  • from the previous instance of the container
    $ oc logs -p $POD > pod-p.log

Gather a HEAP dump from a running container

OpenJDK 8:

POD=your-pod-name; oc exec $POD -- jmap -J-d64 -dump:format=b,file='/opt/eap/standalone/tmp/heap.hprof' $(oc exec $POD jps |grep -iv jps | awk '{print $1}'); oc rsync $POD:/opt/eap/standalone/tmp/heap.hprof .

OpenJDK 11:

POD=your-pod-name; oc exec $POD -- jmap -dump:format=b,file='/opt/eap/standalone/tmp/heap.hprof' $(oc exec $POD jps |grep -iv jps | awk '{print $1}'); oc rsync $POD:/opt/eap/standalone/tmp/heap.hprof .

Gather thread dump(s) from a running container

POD=your-pod-name; PID=$(oc exec $POD  jps | grep -iv jps  | awk '{print $1}'); oc exec $POD -- bash -c "for x in {1..10}; do jstack -l $PID >> /opt/eap/standalone/tmp/jstack.out; sleep 2; done"; oc rsync $POD:/opt/eap/standalone/tmp/jstack.out .

How to identify high CPU utilization by Java threads on OCP 4

This script is the equivalent on OCP4 as the existing one on Linux: How do I identify high CPU utilization by Java threads on Linux/Solaris

It takes as input parameter <pod-name> which is the name of the pod that you diagnose, and provides as output the files high-cpu.out and high-cpu-tdump.out.

POD=<pod-name>;  PID=$(oc exec $POD  jps | grep -iv jps  | awk '{print $1}');  oc exec $POD -- bash -c "for x in {1..6}; do jstack -l $PID >> /opt/eap/standalone/tmp/high-cpu-tdump.out; top -b -n 1 -H -p $PID >> /opt/eap/standalone/tmp/high-cpu.out; sleep 20;  done"; oc rsync $POD:/opt/eap/standalone/tmp/high-cpu.out . ; oc rsync $POD:/opt/eap/standalone/tmp/high-cpu-tdump.out .

Note:
The command presented here works only if package props-ng is part of the OCP/RH-SSO image.
For more information, see JIRA This content is not included.RHSSO-2239 - OCP/RH-SSO props-ng package missing from OCP/RH-SSO 7.6 image

You can run top command following KCS article:

This later article does not allow to run jstack in this case.

Gather a GC log from a running container

POD=your-pod-name; oc rsync $POD:/some/path/to/gc.log .

Another example where an environment variable is in the path of the log name:

POD=your-pod-name; oc rsync $POD:/some/path/$(oc exec $POD printenv HOSTNAME)/gc.log .

Retrieve the standalone-openshift.xml file from a running container

Example for a script. Use either the OCP or Docker commands, but not both.

  • OCP - $POD is a pod name as seen from oc get pods
# The config xml fully qualified name
X_CONFIG_XML="$X_JBOSS_HOME"/standalone/configuration/standalone-openshift.xml
X_JBOSS_HOME=`oc exec "$POD" env | grep -m 1 "JBOSS_HOME=" | cut -d"=" -f2`
# Copy the JBoss EAP config file from a running container to the local directory
oc rsync $POD:$X_CONFIG_XML .
  • Docker - $POD is either the CONTAINER ID or NAME as seen from docker ps
X_CONFIG_XML="$X_JBOSS_HOME"/standalone/configuration/standalone-openshift.xml
X_JBOSS_HOME=`docker exec $POD env | grep -m 1 "JBOSS_HOME=" | cut -d"=" -f2`
# Copy the JBoss EAP config file from a running container to the local directory
docker cp $POD:$X_CONFIG_XML .

Retrieve the Dockerfiles from a running container pulled from registry.access.redhat.com

  • OCP - $POD is a pod name as seen from oc get pods
    • Creates a ./buildinfo directory on the local host which contains all the Red Hat provided Dockerfiles from the pod
oc rsync $POD:/root/buildinfo .
  • Docker
    • Creates a ./buildinfo directory on the local host which contains all the Red Hat provided Dockerfiles from the container

In one terminal session, start a container:

docker run --rm -it --name=test-eap-123 registry.access.redhat.com/jboss-eap-6/eap64-openshift:latest /bin/bash

In a second terminal session, copy the Dockerfiles:

docker cp test-eap-123:/root/buildinfo .

Exit out of the first terminal session to shutdown the container.

Test connectivity between two pods for clustering

In this example, issue a curl command from pod jdg7-trace-1-4d40l to pod jdg7-trace-1-hhw5t.

  • Get the pod names:
$ oc get pods
NAME                 READY     STATUS      RESTARTS   AGE
jdg7-trace-1-4d40l   1/1       Running     0          17h
jdg7-trace-1-hhw5t   1/1       Running     0          17h
  • Get the IP address of pod jdg7-trace-1-hhw5t to use in a curl command:
$ oc describe pod jdg7-trace-1-hhw5t | grep 'IP:'
IP:			10.129.0.38
  • Now use the curl command from pod jdg7-trace-1-4d40l to pod jdg7-trace-1-hhw5t using the telnet protocol to try to connect to the ping port:
$ oc exec jdg7-trace-1-4d40l -- curl -v telnet://10.129.0.38:8888
* About to connect() to 10.129.0.38 port 8888 (#0)
*   Trying 10.129.0.38...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 10.129.0.38 (10.129.0.38) port 8888 (#0)

Hit ctrl-c to disconnect. This shows the curl output of a successful connection between the pods.

Utility script incorporating some of the items mentioned here

SBR
Product(s)
Category
Components
Article Type