Is it possible to configure bonding over bonded interface in Red Hat Enterprise Linux?

Environment

• Red Hat Enterprise Linux (all versions)
• bonding driver to aggregate network interfaces

Issue

• Is it possible to configure bonding over bonded interface in Red Hat Enterprise Linux?
• How to configure bonding or teaming between say bond0 and bond1 ?
• Can we stack bond adapters?
• Are hierarchical bonds supported?
• Are nested bonds supported?

For example:

              +----eth0
    +----bond0+----eth1
bond2
    +----bond1+----eth2
              +----eth3

Resolution

No, stacking or nesting of the bonding driver is not possible

However

Bonding Mode 4 (802.3ad aka LACP) can be used to achieve the same effective functionality, though it does require switch configuration and switch support of LACP aggregation.

Given the following configuration:

.-----------.   .-----------.
|  Switch1  |---|  Switch2  |
'-=-------=-'   '-=-------=-'
  |       |       |       |
  |       |       |       |
.-=----.--=---.---=--.----=-.

| eth0 | eth1 | eth2 | eth3 |
-|-|-|-
|           bond0           |
'---------------------------'

Where each switch has its two ports configured in a PortChannel, the Linux end with the LACP bond will negotiate two Aggregator IDs:

bond0 will see:

• Aggregator ID 1
  • eth0 and eth1
• Aggregator ID 2
  • eth2 and eth3

The two individual switches will have a view completely separate of each other.

Switch 1 will think it has:

• PortChannel 1
  • port X
  • port Y

Switch 2 will think it has

• PortChannel 1
  • port X
  • port Y

The key point is, from the Linux system with the bond, only one Aggregator will be used at a given time, so all traffic to and from a given Linux system will go via one switch.

Fail over between Aggregators can be configured depending on the ad_select bonding option.

Say Aggregator ID 1 is in use, and the cable to eth0 is disconnected, the default behaviour is to stay on Aggregator ID 1.

However, Aggregator ID 1 only has 1 cable, and there's a spare Aggregator ID 2 with 2 cables. It may be preferable to fail over to Aggregator ID 2.

If the ad_select=count bonding option is used, the active Aggregator ID fails over to an Aggregator with the most up ports available.

If the ad_select=bandwidth bonding option is used, the active Aggregator ID fails over to an Aggregator with the most bandwidth available, based on slave speed.

The LACP standard mandates an Aggregator's ports must all be the same speed and duplex. Configuring one Aggregator with 1Gbps ports and one Aggregator with 10Gbps ports is valid. When configured with ad_select=bandwidth, this would provide intelligent selection depending on whether there is 20Gbps, 10Gbps, 2Gbps, or 1Gbps of bandwidth available on an Aggregator.

Some switches can be configured to "logically join" an Aggregator across multiple physical switches, so then the two switches act as one Aggregator ID. This is commonly called Virtual Port Channel or Multi-Chassis Link Aggregation. This solution does not involve the use of VPC or similar MLAG technologies.

Root Cause

The network interface relationships in the Linux kernel are not generalized enough to allow for any type of logical interface to be embedded or enslaved into any other type. Most interesting relationships work fine because code has been added to account for them, such as VLAN-over-bond-over-physical-interfaces, but this also means that only the combinations that are explicitly coded are expected to work correctly.

The main code restriction is that the kernel rewrites the receiving device pointer skb->dev only once in the netif_receive_skb function, updating the skb to point to the bonded device rather than to the physical slave. Nested bonds would need the kernel code to loop until it found the top-most bond.

Another issue would happen with establishing who owns the answer to any given ARP probe request sent out, as it could be either the inner bond or the outer one.

LACP can be used to achieve the same effective functionality, with the added benefit of configurable failover.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.