Running tcpdump from a RHEL CoreOS OpenShift node

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4

Issue

• In some situations, Red Hat support may ask to provide a tcpdump file taken from one or more OpenShift nodes running on Red Hat Enterprise Linux CoreOS (RHCOS).
• RHCOS does not provide the tcpdump tool natively.

Resolution

Even if RHCOS is based on RHEL components, various classical RHEL tools are not included into the system. Additionally is not recommended to connect to a RHCOS node via SSH. The following steps provide the instructions on how to get the tcpdump and upload to a support case.

Note: In case it is requested to grab a tcpdump from a specific pod instead of a node, please have a look at Running tcpdump inside an OpenShift pod to capture network traffic

1. Start a debug container:

   • If the master API is reachable, spawn a debug container on the node:

       $ oc get nodes
       $ oc debug -t node/${NODE_NAME}
     
      The `oc debug` command schedule a `pod` on the selected `node` where the main container mounts the host file system on the `/host` directory. After scheduling the `pod`, a shell into the main container is provided.
     

   • If the Master API is not reachable but you have SSH access to the node, ssh to the node and start a toolbox container instead:

       $ ssh core@${NODE_HOSTNAME}
       $ toolbox
     

   • In both cases, the debug container spawned is based on the image registry.redhat.io/rhel9/support-tools:latest

   Note for disconnected clusters: In case the node is running on a disconnected environment, the image must be mirrored on the local registry and the mirrored image must be used instead.

2. Execute the tcpdump, make sure to replace ${INTERFACE} with the relevant interface (e.g. eth0, lo, etc) and ${TCPDUMP_EXTRA_PARAMS} with additional tcpdump options needed (including the capture filter). The next commands will define the FILENAME variable to be used as file path where the dump will be saved.

    # FILENAME="/host/var/tmp/${HOSTNAME}_${INTERFACE}_$(date +%d_%m_%Y-%H_%M_%S-%Z).pcap"
    # tcpdump -nn -s 0 -i ${INTERFACE} -w ${FILENAME} ${TCPDUMP_EXTRA_PARAMS}
   

3. Upload the tcpdump on the related support case:

   • In case the node is part of a connected cluster the dump file can be uploaded directly from the node:

       # redhat-support-tool addattachment -c <casenumber> ${FILENAME}
     
      **Important:** `redhat-support-tool` needs to access `api.access.redhat.com` via HTTPS, if the OCP `node` is forced to use a proxy in order to access external hosts, the proxy must be configured with the command `redhat-support-tool config proxy_url <proxy-url>` as mentioned at [Red Hat Access: Red Hat Support Tool](https://access.redhat.com/articles/445443).
     
      **Important:** `redhat-support-tool` is only available in `RHEL`, or `RHEL` base products lile `CoreOS`, 6, 7 and 8. For more details, please refer to the article [Red Hat Support Tool (RHST) Deprecation Guide](https://access.redhat.com/articles/6960871).
     

   • In case the node is part of a disconnected cluster, the tpcdump file must be copied locally for the upload:

       $ scp core@<nodename>:/var/tmp/XXXXX.pcap .
     

   • Now, the tcpdump file can be uploaded with redhat-support-tool as described in the previous step.

4. Once the tcpdump file is uploaded, it can be removed from the node by running this from the debug container:

    $ rm /host/var/tmp/XXXXX.pcap
   

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.