How to use tcpdump or other system-level commands inside OpenShift v3 pod?

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (OCP) v3
  • tcpdump

Issue

  • Need to do packet capture (tcpdump) from inside an OpenShift Pod.
  • Can you use commands in a container via nsenter to capture information with tools that are not built into the pod?
  • How to run iperf and other Linux tools that are not in the container's built-in command list
  • How to run commands (tcpdump, ping, etc.) from inside an OpenShift pod's network namespace

Resolution

NOTE: For OpenShift 4, please see How to use tcpdump inside OpenShift v4 Pod

We can make use of nsenter on the node where the pods are hosted:

Steps for docker

  • Login to node where pod is hosted.

  • Identify Container ID (series of numbers and letters) of the pod sandbox container using docker command (replace ${PROJECT_NAME} and ${POD_NAME} with the project and pod name, respectively):

    # docker ps -f label=io.kubernetes.pod.name=${POD_NAME} -f label=io.kubernetes.pod.namespace=${PROJECT_NAME} -f label=io.kubernetes.docker.type=podsandbox -q

  • Use nsenter command with the above container ID to capture packets. Replace <command> with the command you want to run and <parameters> with its parameters (or just omit <command> and <parameters> to open a special shell): 

        # cid="<Container ID>"

    # nsenter -n -t $(docker inspect --format "{{ .State.Pid }}" "$cid") [<command> [<parameters>]]

  • The command (or the shell) started in the step above will run inside pod's network namespace, i.e. it will see pod's networking (instead of host's).

Steps for crio

  • Login to node where pod is hosted.

  • Identify the sandbox id of the pod (replace ${NAME} with the pod name and ${NAMESPACE} with the namespace of the pod.

    pod_id=$(crictl pods --namespace ${NAMESPACE} --name ${NAME} -q)

  • Get the pid of the sandbox from the id obtained above

    pid=$(runc list | awk "/${pod_id}/{print \$2}")

  • Use nsenter with the obtained pid

    nsenter -n -t ${pid} [<command> [<parameters>]]

  • The command (or the shell) started in the step above will run inside pod's network namespace, i.e. it will see pod's networking (instead of host's).

Example

  • Example steps to perform packet capture on a pod named ruby-ex-1-2bdp8:

        # oc get pods -o wide -n test
    NAME              READY     STATUS      RESTARTS   AGE       IP           NODE
    ruby-ex-1-2bdp8   1/1       Running     0          1d        10.130.0.8   node-0

    # export PROJECT_NAME=test
    #export POD_NAME=ruby-ex-1-2bdp8

    # docker ps -f label=io.kubernetes.pod.name=${POD_NAME} -f label=io.kubernetes.pod.namespace=${PROJECT_NAME} -f label=io.kubernetes.docker.type=podsandbox -q
    19025a64e5e7

    # cid=19025a64e5e7

    # nsenter -n -t $( docker inspect --format "{{ .State.Pid }}" "$cid") tcpdump -s 0 -n -i ethX -w /tmp/$(hostname)-$(date +"%Y-%m-%d-%H-%M-%S").pcap

Replace ethX with your respective interface name. Can run command ip a at the place of tcpdump to get the details of the interfaces.

When polling a pod interface you can run the following to discover the interface that it is using:

# nsenter -n -t $( docker inspect --format "{{ .State.Pid }}" "$cid") tcpdump -D
or
# nsenter -n -t $( docker inspect --format "{{ .State.Pid }}" "$cid") ip a

typically, pod interfaces are defaulted to eth0

  • From another terminal:

        # oc -n test rsh ruby-ex-1-2bdp8

    # curl http://localhost:8080    <-- This HTTP request will appear in the packet capture started above.

Root Cause

  • Container runtimes use many mechanisms to isolate processes running inside containers from the host and other containers. One of them is network namespaces, which make each container believe that the available network stack (interfaces, connections...) is different.

  • By applying this solution, the networking tool will be run inside the network namespace of the pod, so it will see the same networking stack as the pod, but it won't be isolated in any other way (it still sees the filesystem of the node instead of the one of the container, you are still root user with no additional SELinux or capabilities restrictions...).

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.