Category Archives: CCIE Data Center

CCIE DC Written – 1.1.a Link Aggregation – LACP


Here are some Nexus facts to keep in mind:

  • With LACP, you can bundle up to 16 interfaces in a channel group. If the channel group has more than 8 interfaces, the remaining interfaces are in hot standby for the port channel associated with this channel group on the M-series modules.
  • From Cisco NX-OS Release 5.1, you can bundle up to 16 active links into a port channel on the F-series module.
  • When you delete the port channel, the software automatically deletes the associated channel group. All member interfaces revert to their original configuration.
  • You cannot disable LACP while any LACP configurations are present.
  • When you run static port channels with no aggregation protocol, the channel mode is always set to on.

Of course, you must globally enable LACP before you can use it on the Nexus device. There are two modes:

  • Passive – responds to negotiations, but does not initiate them – sounds like me at the High School dance
  • Active – initiates negotiations

Starting at 4.2(3) – Cisco introduced some LACP compatibility enhancements as follows:

  •  When a Cisco Nexus device is connected to a non-Nexus peer, its graceful failover defaults may delay the time taken for a disabled port to be brought down or cause traffic from the peer to be lost. To address these conditions, the lacp graceful-convergence command was added.
  • By default, LACP sets a port to the suspended state if it does not receive an LACP PDU from the peer. In some cases, although this feature helps in preventing loops created due to misconfigurations, it can cause servers to fail to boot up because they require LACP to logically bring up the port. You can put a port into an individual state by using the lacp suspend-individual command.

Starting with Release 5.1 Cisco introduced the Minimum Links feature as well as MaxBundle. The Minimum Links feature allows you to:

  • Configure the min number of links that must be in the bundle
  • Prevent low bandwidth LACP bundles from becoming available
  • Causes the port channel to go inactive if the required min bandwidth is not available

MaxBundle allows:

  • Upper limit on ports that are bundled
  • Allows the designation of ports as hot standby

Basic Configuration

  • Use feature lacp to enable the feature
  • Create the port channel interface with interface port-channel 10, use the switchport command in the interface
  • Add a Layer 2 interface to the port channel with switchport followed by channel-group 10 mode passive


CCIE DC Written – 1.1.a Link Aggregation – Load Balancing

Port Channels

The Cisco NX-OS software load balances traffic across all operational interfaces in a port channel by hashing the addresses in the frame to a numerical value that selects one of the links in the channel. Port channels provide load balancing by default. Port-channel load balancing uses MAC addresses, IP addresses, or Layer 4 port numbers to select the link. Port-channel load balancing uses either source or destination addresses or ports, or both source and destination addresses or ports.

You can configure the load-balancing mode to apply to all port channels that are configured on the entire device or on specified modules. The per-module configuration takes precedence over the load-balancing configuration for the entire device. You can configure one load-balancing mode for the entire device, a different mode for specified modules, and another mode for the other specified modules. You cannot configure the load-balancing method per port channel.

The default load-balancing mode for Layer 3 interfaces is the source and destination IP address (src-dst-ip), and the default load-balancing mode for non-IP traffic is the source and destination MAC address (src-dst-mac). Use the port-channel load-balance  command to set the load-balancing method among the interfaces in the channel-group bundle.

NOTE: F1-series modules do not support load balancing of non-IP traffic based on a MAC address. If ports on an F1-series module are used in a port channel and non-IP traffic is sent over the port channel, Layer 2 traffic might get out of order.

Here are the additional load balancing methods you can choose from:

  •  Destination MAC address
  • Source MAC address
  • Source and destination MAC address
  • Destination IP address
  • Source IP address
  • Source and destination IP address
  • Source TCP/UDP port number
  • Destination TCP/UDP port number
  • Source and destination TCP/UDP port number

Symmetric Hashing and Random Load Balancing

Here are a couple of other related topics you should be aware of:

Symmetric Hashing – To effectively monitor traffic on a port channel, it is essential that each interface connected to a port channel receives both forward and reverse traffic flows.

Normally, there is no guarantee that the forward and reverse traffic flows will use the same physical interface. However, when you enable symmetric hashing on the port channel, bidirectional traffic is forced to use the same physical interface and each physical interface in the port channel is effectively mapped to a set of flows. When symmetric hashing is enabled, the parameters used for hashing, such as the source and destination IP address, are normalized before they are entered into the hashing algorithm. This process ensures that when the parameters are reversed (the source on the forward traffic becomes the destination on the reverse traffic), the hash output is the same. Therefore, the same interface is chosen.

Random Load Balancing – Random load balancing on port channels is a software solution that enables better port-link bandwidth utilization for GPRS Tunneling Protocol (GTP) over IP-UDP packets. The existing M1, M2, F1, F2 and F2e line card hardware does not have the capability to perform random load balancing and hence, this software solution helps in load balancing and optimizing the port channels bandwidth. Random load balancing is supported only on F3 series line cards. Random load balancing is applicable on all types of traffic and is effective on egress ports of Layer 3 traffic. The Cisco NX-OS software does random load balancing of all traffic across all interfaces in a port channel by using polynomial scheme.

CCIE DC Written – 1.1.a Link Aggregation – Overview

Port Channels

A port channel is an aggregation of multiple physical interfaces that create a logical interface. You can bundle up to 8 individual active links into a port channel to provide increased bandwidth and redundancy. If a member port within a port channel fails, the traffic previously carried over the failed link switches to the remaining member ports within the port channel.

On the Nexus 7Ks, port channeling also load-balances traffic on the M series module and across these physical interfaces The port channel stays operational as long as at least one physical interface within the port channel is operational.

Starting with the Cisco NX-OS Release 5.1, you can bundle up to 16 active links into a port channel on the F-series module with a Nexus 7K.

NOTE: You cannot configure a shared interface to be part of a port channel.

Like many later Cisco devices, the Nexus devices do not support PAgP, and instead, offer LACP for a dynamic protocol to assist with port channel creation.

Each port can be in only one port channel. All the ports in a port channel must be compatible; they must use the same speed and duplex mode.

You can create port channels directly by creating the port-channel interface, or you can create a channel group that acts to aggregate individual ports into a bundle. When you associate an interface with a channel group, the software creates a matching port channel automatically if the port channel does not already exist. In this instance, the port channel assumes the Layer 2 or Layer 3 configuration of the first interface.

You can also create the port channel first. In this instance, the Cisco NX-OS software creates an empty channel group with the same channel number as the port channel and takes the default Layer 2 or Layer 3 configuration, as well as the compatibility configuration.

Your port channel is operationally up when at least one of the member ports is up and that port’s status is channeling. The port channel is operationally down when all member ports are operationally down.

NOTE: You can create a Layer 2 port channel by bundling compatible Layer 2 interfaces, or you can create Layer 3 port channels by bundling compatible Layer 3 interfaces. After you create a Layer 3 port channel, you can add an IP address to the port-channel interface and create subinterfaces on the Layer 3 port channel. Of course, you cannot combine Layer 2 and Layer 3 interfaces in the same port channel. You can also change the port channel from Layer 3 to Layer 2.

Also as you would expect, all ports in the port channel must be in the same virtual device context (VDC).

Any configuration changes that you apply to the port channel are applied to each member interface of that port channel. For example, if you configure Spanning Tree Protocol (STP) parameters on the port channel, the Cisco NX-OS software applies those parameters to each interface in the port channel.

After a Layer 2 port becomes part of a port channel, all switchport configurations must be done on the port channel; you can no longer apply switchport configurations to individual port-channel members. You cannot apply Layer 3 configurations to an individual port-channel member either; you must apply the configuration to the entire port channel.

You can create subinterfaces on a Layer 3 port channel, even though a subinterface is part of the logical port-channel interface.

You can configure Layer 2 port channels in either access or trunk mode. Layer 3 port-channel interfaces have routed ports as channel members and might have subinterfaces.

From Cisco NX-OS Release 4.2(1), you can configure a Layer 3 port channel with a static MAC address. If you do not configure this value, the Layer 3 port channel uses the router MAC of the first channel member to come up.