How to make NIC ethtool settings persistent (apply automatically at boot)

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux
  • ethtool
  • initscripts
  • NetworkManager

Issue

  • Need to make a setting change with ethtool always apply at boot time or when the network interface is brought up.
  • Need to set NIC ring buffer size permanently.
  • Want to have a NIC offloading feature such as TSO, GSO, LRO, or GRO, etc, enabled or disabled all the time.

Resolution

RHEL 8.3 with NetworkManager 1.25.2-1 or higher

  • As of version 1.25.2-1.el8, NetworkManager has native support for setting the RX and TX ring buffer sizes, previously done with ethtool -G or ethtool --set-ring.

    # nmcli con mod <connection> ethtool.ring-rx 4096
# nmcli con mod <connection> ethtool.ring-tx 4096
# nmcli con up <connection>

    Other adjustments can be done by following the section below:

RHEL 7 and RHEL 8 with NetworkManager 1.14.0 or higher

  • As of version 1.14.0, NetworkManager has native support for controlling link auto-negotiation, speed, and duplex settings as well as hardware offloading settings (ethtool -k) and wake-on-lan settings.

    NOTE: If there is a need to set any other ethtool option (eg: changing ring buffers) then please see the RHEL 7 and 8 with NetworkManager version lower than 1.14.0 section below (for generic options) or the RHEL 8.3 with NetworkManager 1.25.2-1 or higher section above (if you only need to adjust ring buffers)

    • Auto-negotiation, speed, and duplex are controlled by the 802-3-ethernet.auto-negotiate, 802-3-ethernet.speed, and 802-3-ethernet.duplex settings, respectively.

    • If 802-3-ethernet.auto-negotiate is on then setting 802-3-ethernet.speed and 802-3-ethernet.duplex controls which single speed/duplex mode is advertised. Note that item a in both sections 28D.5 (1000Base-T) and 28D.6 (10GBASE-T) indicate that auto-negotiation is mandatory. Refer to Using the ethtool utility one cannot set autoneg off for ethernet cards.

    • If 802-3-ethernet.auto-negotiate is off then the 802-3-ethernet.speed and 802-3-ethernet.duplex properties must be set.

      # nmcli con mod <connection> 802-3-ethernet.auto-negotiate off 802-3-ethernet.speed 100 802-3-ethernet.duplex full

    • Possible hardware offloading features are controlled by the ethtool.* properties.

    • Documentation as per Configuring ethtool settings in NetworkManager connection profiles

    • Not all interfaces support all of these features or they may be fixed on.

              ethtool.feature-esp-hw-offload                ethtool.feature-tx-checksum-fcoe-crc
        ethtool.feature-esp-tx-csum-hw-offload        ethtool.feature-tx-checksum-ip-generic
        ethtool.feature-fcoe-mtu                      ethtool.feature-tx-checksum-ipv4
        ethtool.feature-gro                           ethtool.feature-tx-checksum-ipv6
        ethtool.feature-gso                           ethtool.feature-tx-checksum-sctp
        ethtool.feature-highdma                       ethtool.feature-tx-esp-segmentation
        ethtool.feature-hw-tc-offload                 ethtool.feature-tx-fcoe-segmentation
        ethtool.feature-l2-fwd-offload                ethtool.feature-tx-gre-csum-segmentation
        ethtool.feature-loopback                      ethtool.feature-tx-gre-segmentation
        ethtool.feature-lro                           ethtool.feature-tx-gso-partial
        ethtool.feature-ntuple                        ethtool.feature-tx-gso-robust
        ethtool.feature-rx                            ethtool.feature-tx-ipxip4-segmentation
        ethtool.feature-rx-all                        ethtool.feature-tx-ipxip6-segmentation
        ethtool.feature-rx-fcs                        ethtool.feature-tx-nocache-copy
        ethtool.feature-rx-gro-hw                     ethtool.feature-tx-scatter-gather
        ethtool.feature-rxhash                        ethtool.feature-tx-scatter-gather-fraglist
        ethtool.feature-rx-udp_tunnel-port-offload    ethtool.feature-tx-sctp-segmentation
        ethtool.feature-rxvlan                        ethtool.feature-tx-tcp6-segmentation
        ethtool.feature-rx-vlan-filter                ethtool.feature-tx-tcp-ecn-segmentation
        ethtool.feature-rx-vlan-stag-filter           ethtool.feature-tx-tcp-mangleid-segmentation
        ethtool.feature-rx-vlan-stag-hw-parse         ethtool.feature-tx-tcp-segmentation
        ethtool.feature-sg                            ethtool.feature-tx-udp-segmentation
        ethtool.feature-tls-hw-record                 ethtool.feature-tx-udp_tnl-csum-segmentation
        ethtool.feature-tls-hw-tx-offload             ethtool.feature-tx-udp_tnl-segmentation
        ethtool.feature-tso                           ethtool.feature-txvlan
        ethtool.feature-tx                            ethtool.feature-tx-vlan-stag-hw-insert

        ## Setting TSO and GSO off:
        # nmcli con mod <connection> ethtool.feature-tso off ethtool.feature-gso off

    • Wake-on-LAN is controlled by the 802-3-ethernet.wake-on-lan and 802-3-ethernet.wake-on-lan-password settings.

    • Please refer to the nm-settings man page for usage.

      802-3-ethernet.wake-on-lan:
The NMSettingWiredWakeOnLan options to enable. Not all devices support all options.
May be any combination of:
NM_SETTING_WIRED_WAKE_ON_LAN_PHY		(0x2)
NM_SETTING_WIRED_WAKE_ON_LAN_UNICAST	(0x4)
NM_SETTING_WIRED_WAKE_ON_LAN_MULTICAST	(0x8)
NM_SETTING_WIRED_WAKE_ON_LAN_BROADCAST	(0x10)
NM_SETTING_WIRED_WAKE_ON_LAN_ARP		(0x20)
NM_SETTING_WIRED_WAKE_ON_LAN_MAGIC		(0x40)
or the special values:
NM_SETTING_WIRED_WAKE_ON_LAN_DEFAULT	(0x1) (to use global settings)
and
NM_SETTING_WIRED_WAKE_ON_LAN_IGNORE		(0x8000) (to disable management of Wake-on-LAN in NetworkManager).

802-3-ethernet.wake-on-lan-password:
If specified, the password used with magic-packet-based Wake-on-LAN, represented as an Ethernet MAC address. If NULL, no password will be required.

    • Note that NetworkManager stores these settings in ifcfg files using the ETHTOOL_OPTS parameter with specific value formatting. Manually setting ETHTOOL_OPTS and specifying any ethtool feature other than those known to NetworkManager or with improper syntax will result in undefined behaviour. Therefore it is recommended to always use the nmcli tool to modify these settings.

RHEL 7 and 8 with NetworkManager version lower than 1.14.0

  • This is useful if there is a need to control an ethtool setting which is not currently known to NetworkManager such as ring buffer settings (ethtool -g).

  • Create a dispatcher script: How to write a NetworkManager dispatcher script to apply ethtool commands?

    • As an example, the following dispatcher script uses a custom ETHTOOL_CMD setting in an ifcfg file.

    Note: The following information has been provided by Red Hat, but is outside the scope of the posted Service Level Agreements and support procedures. The information is provided as-is, with any created script considered unsupported by Red Hat Global Support Services. The intent of this article is to provide information to accomplish the system's needs. Use of the information in this article at the user's own risk.

    • Set the ETHTOOL_CMD parameter in the interface's ifcfg file found in the /etc/sysconfig/network-scripts/ directory.

    • Create a dispatcher script for NetworkManager with the contents from the text block below.

    • Place the file under the /etc/NetworkManager/dispatcher.d/ directory and make it executable.

    • Note the ETHTOOL_OPTS parameter name is deliberately not used here as NetworkManager's ifcfg-rh plugin reserves this to store link speed, duplex, auto-negotiation, and offloading options. Any other options listed in this parameter (such as ring settings) will be ignored by NetworkManager and cause the parameter to be erased whenever the ifcfg file is rewritten.

      #!/bin/bash
# BEGIN 99-ethtool.sh
# This file must be made executable!
#
# If an ifcfg file has ETHTOOL_CMD set this
# dispatcher will pass the full ETHTOOL_CMD
# string to the ethtool program.

if [[ $2 == up ]]; then
    SCRIPT="$(basename "$0")"
    if [[ -e $CONNECTION_FILENAME ]]; then
        source $CONNECTION_FILENAME
        if [[ -n $ETHTOOL_CMD ]]; then
            ETHTOOL_CMD="/usr/sbin/ethtool $ETHTOOL_CMD"
            if $ETHTOOL_CMD; then 
                logger "$SCRIPT: success: $ETHTOOL_CMD"
            else
                logger "$SCRIPT: failed: $ETHTOOL_CMD"
            fi
        else
            logger "$SCRIPT: ETHTOOL_CMD not in $CONNECTION_FILENAME, skipping"
        fi
    else
        logger "$SCRIPT: $CONNECTION_FILENAME does not exist?"
    fi
fi

# END 99-ethtool.sh

RHEL 5, 6, 7, and 8 without NetworkManager

  • Set the ETHTOOL_OPTS parameter in the interface's ifcfg file found in the /etc/sysconfig/network-scripts/ directory.

    • The network service scripts will run the ethtool program every time the interface is brought up.

    • ethtool can be invoked multiple times by separating the values with a semicolon.

    • In RHEL 5, RHEL 6, and RHEL 7, all current versions of the initscripts package recognize the ETHTOOL_OPTS parameter in ifcfg files.

    • RHEL 8 requires the network-scripts package to be installed and network service enabled for network services to function without NetworkManager.

              ## Setting a single option (running ethtool once)
        ETHTOOL_OPTS="-G ${DEVICE} rx 4096"

        ## Setting a single option to set both rx and tx queues for ring buffer (running ethtool once to modify rx and tx)
        ETHTOOL_OPTS="-G ${DEVICE} rx 4096 tx 4096"
        
        ## Setting multiple options (running ethtool multiple times)
        ETHTOOL_OPTS="-G ${DEVICE} rx 4096; -A ${DEVICE} autoneg on"

All versions of RHEL, with or without NetworkManager

  • Create a udev rule file in the /etc/udev/rules.d/ directory which triggers on the presence of the expected network interface name and have it invoke ethtool. The following example will run /sbin/ethtool -G enp12s3 rx 4096 when the enp12s3 device is detected.

    • This method is not preferable, as it only runs once on interface detection. If something later changes the parameters then brings the interface up, the interface may not be configured as expected. It is recommended to use one of the other above methods which always runs on interface start.

      # cat /etc/udev/rules.d/99-ethtool.rules
ACTION=="add", SUBSYSTEM=="net", NAME=="enp12s3", RUN+="/sbin/ethtool -G enp12s3 rx 4096"

RHEL 4, 5, 6, and 7, without NetworkManager

Red Hat Virtualization 3.6 and later

Red Hat OpenShift Container Platform 4.x

Red Hat OpenStack Platform 11

  • RHOSP11 Director's os-net-config takes an option ethtool_opts inside a usual interface config, eg:

          "type": "interface",
      "name": "em2",
      "ethtool_opts": "speed 100 duplex full",

RHEL System Roles

  • RHEL 8.3 with NetworkManager 1.25.2-1 or higher

    ---
- name: Configure the NIC ring buffer size permanently with RHEL System Roles
  hosts: all
  become: true
  vars:
    network_connections:
      - name: eth1 
        state: up
        type: ethernet
        ip:
          dhcp4: false
          auto6: false
        ethtool:
          ring:
             rx: 4096
             tx: 4096
  roles:
    - rhel-system-roles.network
...

  • For more examples check the /usr/share/doc/rhel-system-roles/network/ folder

        # ls -al /usr/share/doc/rhel-system-roles/network/ | grep etht
    -rw-r--r--.  1 root root 1208 Aug 22 19:39 example-ethtool_coalesce-playbook.yml
    -rw-r--r--.  1 root root  409 Aug 22 19:39 example-ethtool_features_default-playbook.yml
    -rw-r--r--.  1 root root  555 Aug 22 19:39 example-ethtool_features-playbook.yml
    -rw-r--r--.  1 root root  550 Aug 22 19:39 example-ethtool_ring-playbook.yml

Root Cause

  • Any change made with the ethtool program does not persist across reboots or network module (driver) reloads.
  • NetworkManager earlier than version 1.14.0 does not recognize the ETHTOOL_OPTS ifcfg file parameter.
  • NetworkManager from 1.14.0 has native support for controlling link speed/duplex settings, hardware offloading settings, and Wake-on-LAN settings using the ETHTOOL_OPTS parameter. As of NetworkManager 1.24.0, any other ethtool setting requires using some other method.
  • NetworkManager does not invoke the /sbin/ifup-local script.

Diagnostic Steps

  • Please refer to the nm-settings man page for more detail.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.