RHEL network interface dropping packets

Environment

• Red Hat Enterprise Linux (All versions)
• Network interfaces
• Applications reporting loss of data (cluster fence, slow performance, timeouts, failed backups, etc)

Issue

• System dropping network packets
• High number of drop counters like drop, discard, err or error, fifo, buf or buffer, fail, miss, OOB, full counters in ethtool -S
• High number of dropped, error, overrun, or frame counters in ifconfig
• What is the first thing to try when I see my network interfaces dropping packets?
• High rx_out_of_buffer counter in ethtool -S output

Resolution

• RHEL 7, 8 & 9 drop statistics cover more reasons for discards of unwanted packets than previous version of RHEL. For more details, see document ifconfig and ip commands report packet drops?.

Packets can be dropped at other levels in the network stack. These will not be counted as discards/drops/etc shown by ethtool -S ethX.

An extremely small rate of dropped packets may be small enough to ignore. See Should I be concerned about a 0.05% packet drop rate? for more information.

If the packet drop rate is large enough:

• Increase the size of the ethernet device RX ring buffer. First, inspect the output of ethtool -g ethX for the maximum size:

[root@host]# ethtool -g eth0
 Ring parameters for eth0: 
 Pre-set maximums: 
 RX:             4080 
 RX Mini:        0 
 RX Jumbo:       16320 
 TX:             255 
 Current hardware settings: 
 RX:             255 
 RX Mini:        0 
 RX Jumbo:       0 
 TX:             255

• If the Pre-set maximums are more than the what is listed in the current hardware settings, it is recommend to increase this number as follows:

[root@host]# ethtool -G eth0 rx 4080

NOTE

• Changing the ring buffer may or may not cause a small interruption to network connectivity, depending on the driver in use, as change in any NIC property relies on an interaction between driver and firmware and the NIC may have to be reset to accept the new changes.

• Changes made with ethtool are not persistent across reboots. You can persist this setting across reboots by following How to make NIC ethtool settings persistent (apply automatically at boot)

• Different device drivers have different options relating to the ring buffer. For example the be2net driver cannot make changes to the ring buffer at all. The Cisco enic driver can not make changes with ethtool, these settings can be tuned in the UCS Manager interface on the Port Profile screen.

• If after increasing this buffer size to the maximum allowed on the card, you are still experiencing these drops, it may be beneficial to use a bonding mode that will spread the load of traffic across multiple network interfaces, or otherwise inspect options to carry out network performance tuning on the system.

Root Cause

• Counters like "discard" or "drop" in the output of ethtool -S ethX are caused by the exhaustion of the receive ring buffer.

• Each packet received by the NIC is stored in the ring buffer while an interrupt is sent to the kernel to fetch the packet into kernel memory.

• If the available buffer is filling up faster than the kernel can take the packets, there will be discarded packets.

Diagnostic Steps

• Verify packet loss with:

[root@host]# ip -s -s link

• Verify discards/drops/etc with:

[root@host]# ethtool -S ethX

• Verify RX ring buffer current and maximum settings with:

[root@host]# ethtool -g ethX

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.