How do I tune RHEL for better TCP performance over a specific network connection?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux
  • Network connection known performance characteristics, such as a WAN connection with high throughput and high latency, but does not necessarily have to be a WAN
  • TCP networking

Issue

  • We are transmitting a large amount of data over a WAN link and wish to tune the transfer to be as fast as possible
  • We are trying to backhaul traffic from one datacenter to another and appear to be running into small TCP window sizes that are impacting the transmission speed
  • We are looking for sysctl parameters which can be tuned to allow better performance over our network
  • How do I accurately tune TCP socket buffer sizes?

Resolution

Step 1 - Bandwidth Delay Product (BDP)

Calculate the BDP of the connection.

This value measures the amount of data which can be in flight at a given time.

Another way to think about it is the "holding capacity" of the wire.

The formula is:

connection speed in bytes * latency in seconds = BDP in bytes

For example, to calculate the BDP for a 10 Gbps connection that has a 17 ms latency:

(10 * 1000 * 1000 * 1000 / 8) * 0.017 = 21,250,000 bytes

An online calculator can be used, such as: <Content from www.speedguide.net is not included.https://www.speedguide.net/bdp.php>

Step 2 - Socket Buffer Tuning

Set socket buffer maximum values to a value larger than the BDP. Usually 2.5x is a good starting point.

21,250,000 x 2.5 = 53,125,000

In the /etc/sysctl.conf file, set the socket buffer maximum values:

net.core.rmem_max = 53125000
net.core.wmem_max = 53125000
net.ipv4.tcp_rmem = 8192 262144 53125000
net.ipv4.tcp_wmem = 8192 262144 53125000

These values can be refined through your own testing.

You might find it useful to increase the default value from 262144 up to 524288.

You might find it useful to increase or decrease the maximum value between 2x BDP and 3x BDP.

Step 3 - Other TCP Tuning

Ensure TCP Window Scaling is enabled. If Window Scaling is not enabled then TCP connections will be limited to putting only 64 KiB of data on the wire at any time, which will heavily restrict throughput:

net.ipv4.tcp_window_scaling = 1

Ensure TCP Timestamps are enabled, this allows TCP to more accurately measure latency so it can adjust the TCP Window for more accurate performance:

net.ipv4.tcp_timestamps = 1

Ensure TCP Selective Acknowledgement (SACK) is enabled to help mitigate delays due to packet loss:

net.ipv4.tcp_sack = 1

If the application is expected to pause for more than one latency round trip, then it will likely be useful to disable TCP "Slow Start" after that idle period:

net.ipv4.tcp_slow_start_after_idle = 0

Lastly, different TCP Congestion Control Algorithms can be explored:

Root Cause

These concepts are discussed further in the RHEL product documentation:

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.