Capturing network packets with tcpdump

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 3 and later

Issue

There are times when troubleshooting of networking issue requires behavioural analysis on the packet level. Success of packet analysis often depends upon acquiring proper packet data in the first place. This document addresses this problem while also providing answers to following questions:

  • How do I capture network packets with tcpdump?
  • How can I monitor Network Traffic on my system?
  • Please provide a recommended utility for network capture

Resolution

Installation

The tcpdump utility is provided in an RPM package tcpdump-<version>.<arch>.rpm.

Tcpdump example

# tcpdump -s 0 -i {INTERFACE} -w {FILEPATH} [filter expression]

Once completed, kill the tcpdump command with the ctrl+c keys.

Parameters

  • -s is an option for snaplen. It tells tcpdump the amount of information (bytes) it should capture for each packet. Tcpdump from RHEL 6 and newer, captures 65535 bytes of data from each packet by default. On previous versions (RHEL 5 and older) tcpdumpcaptures 68 bytes by default. 68 bytes is often not sufficient to perform any sort of troubleshooting or diagnosis. Please refer to the tcpdump(8) man page and search for "snaplen" if you are unsure what the default is on your system. If you want to capture all the data from network packets, run the tcpdump command with the "-s 0" option as shown above. In the situation where network traffic is huge or there is need for capturing traffic for a prolonged period of time, this might not produce the best result.

When capturing packets, performance is also important; if there is shortage of resources tcpdump may not capture all packets. To tune tcpdump, it is recommended to limit snaplen to the smallest number that will capture all the important protocol information required for later analysis.

The amount of bytes which are going to be needed is issue dependent. For instance if you are troubleshooting NFS v3, a snaplen of 256 is usually sufficient unless you're troubleshooting READDIR/READDIRPLUS issues, in which case you will probably want to use a snaplen of 0. For NFS v4, you will typically use a snaplen of 0.

  • -i substitute {INTERFACE} with name of networking interface through which investigated traffic is flowing. You can capture traffic on various types of interfaces (physical, vlan, bond, team, etc.), but in most cases you want to avoid using keyword any. Using any will capture packets from all interfaces and may cause packets to be captured more than once in the case you are using bonding, teaming, etc. This will complicate troubleshooting because various tools will flag the extra packets as retransmitted data. If you know the address of the remote end that you are communicating with, then you can easily determine the source interface through which it's being reached via ip route get.

      # ip route get 10.16.29.55
  10.16.29.55 via 10.16.47.254 dev eth0  src 10.16.46.34 
      cache

In the above example, the source interface is eth0. That is the interface name you would pass to the -i option in tcpdump.
In case there is requirement to run tcpdump on multiple interfaces at the same time, please run concurrent separate tcpdump for each interface (writing to different files).

  • -w FILEPATH specified here should point to a local file system. This will prevent tcpdump from capturing un-desired traffic. Also make sure you have enough free space. Using /tmp is suggested if mounted to local storage. In order to ensure the capture being saved in binary format (which is highly preferred) the file name should end with binary extension (.cap, .pcap, .etc).

Filter expression


You can filter packets while you capturing them with `expression`. Please see the man page of `pcap-filter` for the reference of `expression`. Also if you planning to capture on vlan interface while using filters please visit following solution from KCS database: ["Why won't tcpdump capture any traffic when specifying a port and interface that is on a VLAN? "](https://access.redhat.com/site/solutions/160153). Its advised to use filters only when absolutely necessary. Using filters puts more stress and decreases performance of the server executing`tcpdump`. It also often makes analysis fail due to insufficient details being captured. If you must use filters, please state what filters have been used and provide the reason why, when uploading the capture file for analysis.

For more information about tcpdump, see the man-page.

Capturing tips


Syntax of `tcpdump` command is relatively easy, however, in order to capture truly relevant network traffic which will be useful for later analysis it'd be beneficial to follow these simple rules:
  • Run packet capture simultaneously on both connection sides (client and server)
  • Make sure packets are being captured when the issue is present
  • Make sure the capture file is saved onto the local file system
  • Note the system timezone, where the capture has been done for later analysis and possible correlations with system logs
  • Use rolling capture for issues which are hard to reproduce or to prevent capture files from growing too big
  • Use tcpdump -s <bytes> to limit the amount of information which will be saved per packet
  • Don't use text output but binary (files with extension .cap, .pcap, .etc)
  • Avoid using tcpdump -i any and be sure you are capturing on the right interface
  • Avoid using capture filters if possible as this may capture additional clues visible for different traffic types. However this may not always be feasible if there is high bandwidth usage on the interface as it could generate a huge file. If filters are required please consult with the Red Hat case team first. When filtering for a specified host or port take care to not discard one direction of the network conversation.
  • There is an application available on the Red Hat Customer portal that can help with syntax generation if required. Please see: https://access.redhat.com/labs/nptcpdump/
  • Compress capture files (eg. using gzip)

How to upload captures

  1. Verify the traffic has been captured properly by running the tcpdump command with the '-r' option. The output from this command should show packets sent from both client and server if captured properly, similar to the below example of NFS 'getattr' requests and responses:

     # tcpdump -r /tmp/client.cap
  ...
 09:39:06.272688 IP 192.168.155.74.3337237041 > 192.168.155.2.nfs: 216 getattr fh 0,0/22
 09:39:06.273342 IP 192.168.155.2.nfs > 192.168.155.74.3337237041: reply ok 88 getattr NON 2 ids 0/9 sz 0
 09:39:06.273365 IP 192.168.155.3354014257 > 192.168.155.2.nfs: 220 getattr fh 0,0/22
 09:39:06.273840 IP 192.168.155.2.nfs > 192.168.155.74.3354014257: reply ok 108 getattr NON 2 ids 0/9 sz 0

  2. If the trace files are large (hundreds of MB or even a few GB), compress them with gzip. Other compression methods, such as bzip2, can be used as well, but some tools such as tshark may not be able to read them, thus gzip is recommended.

     # gzip /tmp/client.cap
 # gzip /tmp/server.cap

  3. Attach the compressed trace(s) (the *.cap.gz files) from the client and the server system, to the support ticket or upload to Red Hat Global Support Services sFTP server.

Installation

  1. Make sure that the tcpdump RPM is installed on both the client and server systems.

    • In Red Hat Enterprise Linux 3 or 4, use the 'up2date' command:

        # up2date -i tcpdump

    • In Red Hat Enterprise Linux 5 or later, use the 'yum' command:

        # yum install tcpdump

Rolling Capture

Sometimes it is necessary for a capture to run for an extended period of time. In those case it is not a good idea to allow the capture file to grow indefinitely. Still check that enough disk space is available for the projected rolling capture. The arguments -w path -W n -C m will direct tcpdump to create "n" files of approximately "m" megabytes. The files will have the names path0 thru path(n-1). When the size of pathX exceeds m megabytes path(mod (X+1, n) is written or rewritten. Since tcpdump is by default run under the tcpdump user ID it is necessary that this user ID has the necessary permissions to create the pathX files. The "-Z userID" can be used to change the ID that runs the tcpdump command.

It is important that the combination of file size and the number of files versus the rate at which the files are cycled give you enough time to recognize that the event you are trying to capture has occurred and stop the capture before the file with the event is overwritten. The following script can help with that. It can be used to start tcpdump and then stop it when a trigger message is seen in a log file.

Example: ./pcapLogwatch.sh eth0 5 250 /tmp dumpfilename.pcap "" /var/log/messages "Time has been changed"

SCRIPT NAMEDEVICENUMBER-OF-FILESSIZE-OF-FILESDUMP-DIRECTORYDUMP-FILE-NAMEFILTERLOG-FILE-NAMETRIGGER-STRING
./pcapLogwatch.sheth05250/tmpdumpfilename.pcap""/var/log/messages"Time has been changed"

Where:

  • DEVICE = (eth0) The interface you are capturing from (#ip a)
  • NUMBER-OF-FILES = (5) number of files
  • SIZE-OF_FILES = (250) Where 250 is size in MB
  • DUMP-DIRECTORY = (/tmp) The directory to dump to
  • DUMP-FILE-NAME = (dumpfilename.pcap) The name that will be given to your capture file
  • FILTER = ("") the tcpdump filter, can be a protocol or IP
  • LOG-FILE-NAME = (/var/log/messages) The file that we are watching for our triggers
  • TRIGGER-STRING = (Time has been changed) The error message that stops the rolling capture in the watched file

pcapLogwatch.sh script:

#!/bin/sh
# pcaplogwatch.sh
# - capture packets until a given string is seen in a given logfile
#
# Usage:
#   pcapLogwatch.sh DEVICE NUMBER-OF-FILES SIZE-OF-FILES DUMP-DIRECTORY DUMP-FILE-NAME FILTER LOG-FILE-NAME TRIGGER-STRING
#
#
if [ $# -ne 8 ]
     then echo "Usage:"
          echo pcapLogwatch.sh DEVICE NUMBER-OF-FILES SIZE-OF-FILES \
               DUMP-DIRECTORY DUMP-FILE-NAME FILTER LOG-FILE-NAME \
               TRIGGER-STRING
          echo Where:
          echo "  DEVICE           name of the device used for capturing packets"
          echo "  NUMBER-OF-FILES number of files in the capture ring"
          echo "  SIZE-OF-FILES   size of each file (approximately)"
          echo "  DUMP-DIRECTORY  directory to put the files"
          echo "  DUMP-FILE-NAME  base name of the capture files"
          echo "  FILTER          tcpdump filter string, may be \"\""
          echo "  LOG-FILE-NAME   name of log file containing the trigger"
          echo " TRIGGER-STRING   when this string is seen tcpdump will be stopped"  
          exit
fi

# After setting the parameters below, this script will run
# a rolling packet capture until the string in $LOG_STRING
# is found in $LOG_FILE. Set the optional $DUMP_FILTER with
# whatever libpcap filter as needed.
     
# Interface to capture on:
DUMP_INTERFACE="$1"
     
# Number of capture files:
DUMP_NUM="$2"
     
# Size of capture files:
DUMP_SIZE="$3"
     
# Directory to save file to:
DUMP_DIR="$4"
     
# Filename to capture to:
DUMP_NAME="$5"
     
# libpcap Dump filter:
DUMP_FILTER="$6"
     
# Log file to monitor:
LOG_FILE="$7"
     
# String to match in LOG_FILE:
LOG_STRING="$8"
     
# Time to wait after LOG_STRING found
# until the dump is stopped (in seconds):
TIME_AFTER=1
     
mkdir -p "$DUMP_DIR"
chmod g+w "$DUMP_DIR"
chgrp tcpdump "$DUMP_DIR"
     
/usr/sbin/tcpdump -s 0 -w "$DUMP_DIR/$DUMP_NAME" -W "$DUMP_NUM" -C "$DUMP_SIZE" -i "$DUMP_INTERFACE" "$DUMP_FILTER" &DUMP_PID=$!

tail --follow=name --pid=$DUMP_PID -n 0 "$LOG_FILE" | awk "/$LOG_STRING/{system(\"sleep $TIME_AFTER; kill $DUMP_PID\")}" & 
disown -a -h
exit 0
# END pcapLogwatch.sh

In certain situations, the event is not triggered by an event but some time after. In this situation you can run the following command. Be sure to change the interface (eth0) to the relevant interface name on your server: 

# /usr/sbin/tcpdump -s 0 -w /tmp/reproduced_issue.pcap -W 4 -C 300 -i eth0
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.