Troubleshooting OpenShift Container Platform 3.x: Middleware Containers

Updated

Note: This article is for Openshift 3.x. For OCP 4.x please check this solution

Enable script debugging output for an application pod

Sometimes it's helpful to debug the dynamic configuration scripts that are included and used in the Middleware Containers for Openshift. Enabling this script debugging will show the order in which scripts are executed as well as the environment variables and their values as they are processed by the scripts. Follow these steps to enable script debugging. NOTE: Enabling or disabling script debugging requires a pod deployment.

Add the following environment variables to your application's deploymentconfig. This can be done via the Openshift Web Console or via the oc command line utility.

The specific environment variables to use for script debugging are:

SCRIPT_DEBUG=true
PS4=+ $(basename $0):${LINENO} :

$ oc set env dc/the-deployment SCRIPT_DEBUG=true PS4='+ $(basename $0):${LINENO} :'

This may automatically redeploy your application pod. If not, be sure to redeploy the pod as this is necessary to get the debug output at deploy time.

Use this command to remove the environment variables from the application pod:

$ oc set env dc/the-deployment SCRIPT_DEBUG- PS4-

Remember to redeploy the pod if this does not occur automatically.

Gather a JBoss Diagnostics Report (JDR) from a running container

POD=your-pod-name; oc rsync $POD:$(oc exec $POD /opt/eap/bin/jdr.sh | grep "JDR location" | awk '{print $3}') .

Save pod logs from a running container to a file

$ oc logs $POD > pod.log

Gather a HEAP dump from a running container

POD=your-pod-name; oc exec $POD -- /usr/lib/jvm/java-1.8.0-openjdk/bin/jmap -J-d64 -dump:format=b,file='/opt/eap/standalone/tmp/heap.hprof' $(oc exec $POD ps aux | grep java | awk '{print $2}'); oc rsync $POD:/opt/eap/standalone/tmp/heap.hprof .

Gather thread dump(s) from a running container

POD=your-pod-name; PID=$(oc exec $POD ps aux | grep java | awk '{print $2}'); oc exec $POD -- bash -c "for x in {1..10}; do jstack -l $PID >> /opt/eap/standalone/tmp/jstack.out; sleep 2; done"; oc rsync $POD:/opt/eap/standalone/tmp/jstack.out .

Gather a GC log from a running container

POD=your-pod-name; oc rsync $POD:/some/path/to/gc.log .

Another example where an environment variable is in the path of the log name:

POD=your-pod-name; oc rsync $POD:/some/path/$(oc exec $POD printenv HOSTNAME)/gc.log .

Retrieve the standalone-openshift.xml file from a running container

Example for a script. Use either the OCP or Docker commands, but not both.

  • OCP - $POD is a pod name as seen from oc get pods
# The config xml fully qualified name
X_CONFIG_XML="$X_JBOSS_HOME"/standalone/configuration/standalone-openshift.xml
X_JBOSS_HOME=`oc exec "$POD" env | grep -m 1 "JBOSS_HOME=" | cut -d"=" -f2`
# Copy the JBoss EAP config file from a running container to the local directory
oc rsync $POD:$X_CONFIG_XML .
  • Docker - $POD is either the CONTAINER ID or NAME as seen from docker ps
X_CONFIG_XML="$X_JBOSS_HOME"/standalone/configuration/standalone-openshift.xml
X_JBOSS_HOME=`docker exec $POD env | grep -m 1 "JBOSS_HOME=" | cut -d"=" -f2`
# Copy the JBoss EAP config file from a running container to the local directory
docker cp $POD:$X_CONFIG_XML .

Retrieve the Dockerfiles from a running container pulled from registry.access.redhat.com

  • OCP - $POD is a pod name as seen from oc get pods
    • Creates a ./buildinfo directory on the local host which contains all the Red Hat provided Dockerfiles from the pod
oc rsync $POD:/root/buildinfo .
  • Docker
    • Creates a ./buildinfo directory on the local host which contains all the Red Hat provided Dockerfiles from the container

In one terminal session, start a container:

docker run --rm -it --name=test-eap-123 registry.access.redhat.com/jboss-eap-6/eap64-openshift:latest /bin/bash

In a second terminal session, copy the Dockerfiles:

docker cp test-eap-123:/root/buildinfo .

Exit out of the first terminal session to shutdown the container.

Test connectivity between two pods for clustering

In this example, issue a curl command from pod jdg7-trace-1-4d40l to pod jdg7-trace-1-hhw5t.

  • Get the pod names:
$ oc get pods
NAME                 READY     STATUS      RESTARTS   AGE
jdg7-trace-1-4d40l   1/1       Running     0          17h
jdg7-trace-1-hhw5t   1/1       Running     0          17h
  • Get the IP address of pod jdg7-trace-1-hhw5t to use in a curl command:
$ oc describe pod jdg7-trace-1-hhw5t | grep 'IP:'
IP:			10.129.0.38
  • Now use the curl command from pod jdg7-trace-1-4d40l to pod jdg7-trace-1-hhw5t using the telnet protocol to try to connect to the ping port:
$ oc exec jdg7-trace-1-4d40l -- curl -v telnet://10.129.0.38:8888
* About to connect() to 10.129.0.38 port 8888 (#0)
*   Trying 10.129.0.38...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 10.129.0.38 (10.129.0.38) port 8888 (#0)

Hit ctrl-c to disconnect. This shows the curl output of a successful connection between the pods.

Utility script incorporating some of the items mentioned here

Product(s)
Category
Tags
Article Type