ifconfig and ip commands report packet drops?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 9
  • Red Hat enterprise MRG realtime 2.3 (kernel-rt-3.8.13-rt14.25.el6rt)
  • Bonding

Issue

  • ifconfig reports rx drops for servers, physical or VMware guests, after upgrading to RHEL7
  • ip -s link and ifconfig report the rx_drop* counter increasing but no drops are seen with ethtool -S ethX
  • RHEL7 ifconfig show rx drops while RHEL 6.X guests running on the very same ESX host or cluster show no rx drops
  • Any NIC hardware type
  • netstat -i reports RX-ERR and RX-DRP
  • We are using bond interface on MRG 2.3. Suddenly, dropped packets occurs on slave's one of bond interfaces.
  • After installing the RT kernel we observed the rx_dropped counter increasing for ifconfig.

Resolution

This is not a bug. Previous kernels were not accounting for these dropped packets (unknown protocol for example), but they really were dropped.

ethtool -S should be used for packet drop detection, as described at:

Root Cause

Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Seeing high rx_drop_counters in RHEL 7 and later version from ifconfig|ip -s link is not a definitive conclusion of a network problem. To gain more traction of a network problem "ethtool -S" can be used for Network Card Statistics or "netstat -s" for Network Application statistics.

  • RHEL 7 differentiates packet loss from older o/s (RHEL 6 and RHEL 5), as rx drop counter can increment from the following reasons:
    • unrecognised protocol
    • unknown VLAN
    • unregistered multicast address.
    • Incremented for certain packets received by an inactive bond or team member.
  • From Content from git.kernel.org is not included.kernel git
net: add a core netdev->rx_dropped counter
In various situations, a device provides a packet to our stack and we
drop it before it enters protocol stack :
- softnet backlog full (accounted in /proc/net/softnet_stat)
- bad vlan tag (not accounted)
- unknown/unregistered protocol (not accounted)

We can handle a per-device counter of such dropped frames at core level,
and automatically adds it to the device provided stats (rx_dropped), so
that standard tools can be used (ifconfig, ip link, cat /proc/net/dev)

This is a generalization of commit 8990f468a (net: rx_dropped
accounting), thus reverting it.

Diagnostic Steps

  • ifconfig RX dropped packets increases steadily.

  • Just a sample of a frame with Ethernet Type 0 which causes rx drop:

# tshark -n -t ad -r /tmp/test.pcap -T fields -e frame.number -e eth.type 2>/dev/null
1	0x00000000
# tshark -n -t ad -r /tmp/test.pcap 2>/dev/null
  1 2017-07-28 12:06:40     3c.00.50 -> a8.7a.2f     FC 60 Link Ctl, ACK1
# tcpdump -r /tmp/test.pcap -ne 2>/dev/null
12:06:40.532597 00:00:00:00:00:00 > 00:00:00:00:00:01, 802.3, length 60: LLC, dsap Unknown (0x44) Group, ssap Null (0x00) Command, ctrl 0x2800: Information, send seq 0, rcv seq 20, Flags [Command], length 46

Note: While tcpdump is running, the above frame does not increase rx drop count.
LLDP frames will also cause this every 30 seconds:

$ tshark -r /tmp/lldp1.pcap
    1 06:50:01.100787   0.000000 00:00:00:00:00:00 → LLDP_Multicast LLDP 292 NoS = 00:00:00:00:00:00 TTL = 120 System Name = hostname.com System Description = Cisco Nexus Operating System (NX-OS) Software 7.0(3)I7(3)\nTAC 
    2 06:50:31.101609  30.000822 00:00:00:00:00:00 → LLDP_Multicast LLDP 292 NoS = 00:00:00:00:00:00 TTL = 120 System Name = hostname.com System Description = Cisco Nexus Operating System (NX-OS) Software 7.0(3)I7(3)\nTAC
SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.