Content
Perfect Load-Balancing: How Close Can You Get?
Have you ever considered how much we're missing in our lives just because we've always been trained (not to say indoctrinated) to respect the rules and guidelines? For example, how many solutions and useful implementations of available networking tools and technologies have we missed just because we strictly followed the manuals and were unable to think out-of-the box? In fact, what's the difference between a proper use of a tool or available technology from an improper one? What separates a creative engineer from an average implementer? Standards? Best practices? Actually, no-one can really say. But still, it's something that limits us in our thinking and creativity.
Do you need an example? Multi-protocol Label Switching (MPLS) is considered by most enterprise-focused network engineers to be a technology that is only useful in a Service Provider environment. No doubt that’s the main focus of MPLS. But did you know you can deploy it to solve some of the challenging problems of today's enterprise networks? In this article, you’ll see how you can deploy MPLS Traffic Engineering (MPLS-TE) in your enterprise network to achieve optimal load-balancing in a highly redundant setup.
Overview
Load-balancing is one of those topics that we’ve all mastered a long time ago … or so we like to think, until the users come to us with impossible requirements. A common cause for our headaches are designs where the customers expect perfect load-balancing between two Local Area Networks (LANs) connected in a fully redundant setup similar to one in Figure 1. The task would be simple if the end devices would be able to load-balance their traffic between the attached routers. Unfortunately, not many IP hosts support dynamic routing protocols (or have them properly configured), usually they send the outbound traffic to a default gateway (a single IP address).
LAN-to-LAN load-balancing in redundant setup
Prior to IOS release 12.2(15)T, Cisco recommended a kludge: configuring multiple HSRP groups (MHSRP) on the same LAN interface to achieve at least a minimum amount of load-balancing (provided the default gateways on the end hosts were configured correctly). With the implementation of Gateway Load-Balancing Protocol (GLBP), available in IOS release 12.2(15)T and 12.3, LAN-to-LAN load-balancing became more predictable and easier to implement. However, both MHSRP and GLBP implement load-balancing based on source IP hosts – all sessions from the same IP host (or server) use the same WAN link. While this approach provides good link utilization in environments with a large number of clients and servers, it fails completely in cases where a small group of IP hosts (or even a single session between two IP hosts) should utilize all the available WAN bandwidth (see Figure 2 for a sample server-to-server replication scenario).
Server-to-server replication
Routing Tricks
The first solution any good networking engineer would propose is routing tricks – if we can ensure that all IP hosts send their traffic to a designated LAN router (A1 and B1 in example from Figure 2), and this router has two equal-cost paths to the destination, the traffic would always be optimally load-balanced as displayed in Figure 3.
Load-balancing with a single LAN gateway
The only way to get there is to decrease the cost of the A2-B2 link (even EIGRP with non-equal-cost load-balancing would not work otherwise) to ensure that the cost of A1-A2-B2-B1 path is equal to the cost of the A1-B1 link (Figure 4)
Load-balancing with reduced WAN link cost
When using OSPF or IS-IS as your routing protocol, you would specify manual costs of the WAN links with the ip ospf cost or isis metric interface configuration command. EIGRP users would have to change the delay portion of the EIGRP metric (as the EIGRP composite metric by default considers minimum bandwidth and sum of delays). With the reduced cost of the A2-B2 link, the A1 router would find two equal-cost paths to the destination network, performing load-balancing between them (Listing 1). Of course, the A2 router would never load-balance the traffic in this scenario (Listing 2).
Load-balancing on A1
a1#show ip route 192.168.0.0
Routing entry for 192.168.0.0 255.255.255.0
Known via "ospf 1", distance 110, metric 51, type intra area
Last update from 10.0.0.6 on FastEthernet0/0, 00:00:03 ago
Routing Descriptor Blocks:
* 172.16.1.2, from 172.16.0.22, 00:00:03 ago, via Serial0/0/0.100
Route metric is 51, traffic share count is 1
10.0.0.6, from 172.16.0.22, 00:00:03 ago, via FastEthernet0/0
Route metric is 51, traffic share count is 1
A2 cannot load-balance
a2#show ip route 192.168.0.0
Routing entry for 192.168.0.0 255.255.255.0
Known via "ospf 1", distance 110, metric 50, type intra area
Last update from 172.16.1.6 on Serial0/0/0.101, 00:01:04 ago
Routing Descriptor Blocks:
* 172.16.1.6, from 172.16.0.22, 00:01:04 ago, via Serial0/0/0.101
Route metric is 50, traffic share count is 1
It’s obvious that the routing tricks can solve load-balancing problems only in a very tightly controlled environment (the example solution was used on a firewall-to-firewall connection between two corporations with the routing protocol not being extended beyond the four routers described in this scenario). Additional routers participating in the same routing protocol or more complex paths between the LANs make this solution extremely complex and thus highly unusable.
Load-Balancing Caveats
Before moving to the advanced scenario deploying MPLS traffic engineering, it’s worth revisiting the major load-balancing caveats that apply even to perfectly symmetrical scenario from the previous section:
Without Cisco Express Forwarding (CEF), load-balancing is performed based on destination prefix only (unless you can afford configuring CPU-intensive process switching with the no ip route-cache configuration command, where the load-balancing is done on a packet-by-packet basis). In scenario from Figure 2, destination-only load-balancing is useless, as there is a single destination prefix.
By default, CEF (enabled with ip cef global configuration command) performs source-destination pair based load-balancing. Source and destination IP addresses are hashed together in a 4-bit value that is used to select one of the potential outgoing paths. This guarantees that the all the traffic between a pair of IP hosts will flow over the same path.
You can configure per-packet CEF load-balancing with the ip load-sharing per-packet interface configuration command.
Note
Configuring per-packet load-balancing over high-speed WAN links will reduce the session throughput due to varying TCP transmission delay.
In all designs where the router might route the packet back to the incoming interface (in our scenario, A1 would send half the packets back on the Ethernet LAN toward A2), you have to disable IP redirects with no ip redirects interface configuration command.
Note
If you don’t disable IP redirects when using CEF, all packets routed to the same interface will be sent to a slower switching method (fast switching or process switching).
If you’re not using CEF switching, make sure all packets are fast-switched with the ip route-cache same-interface interface configuration command.
Load-Balancing with MPLS Traffic Engineering
One of the design goals of the Traffic Engineering module of the Multi-Protocol Label Switching (MPLS-TE) was to enable optimum network utilization beyond the equal-cost load-balancing permitted by the IP routing. The MPLS TE is thus the ideal candidate technology in optimal load-balancing designs. A TE tunnel between A1 and B1 going through A2 and B2 would give us a second direct path between A1 and B1. As the routing protocol cost of a TE tunnel is by default equal to the IP routing cost between the tunnel endpoints, the A1-A2-B2-B1 tunnel would be the second equal-cost path between A1 and B1 (Figure 5).
Note
Similar result could be achieved with a GRE tunnel between A1 and B1, but the routing setup needed to ensure the GRE tunnel goes through A2 and B2 is exceedingly complex and not worth exploring.
MPLS TE tunnel between A1 and B1
However, Cisco IOS does not permit load-balancing between an MPLS TE interface and a regular IP interface. It’s thus necessary to establish a second MPLS TE tunnel directly between A1 and B1 (Figure 6). With the two tunnels in place, A1 performs equal-cost load-balancing between the two tunnels (Listing 3)
Complete TE solution
Load-balancing over MPLS TE tunnels
a1#show ip route
… parts deleted …
O 192.168.0.0 255.255.255.0 [110/51] via 0.0.0.0, 00:00:31, Tunnel1
[110/51] via 0.0.0.0, 00:00:31, Tunnel0
a1#show ip cef 192.168.0.0
192.168.0.0/24, version 17, epoch 0, per-packet sharing
0 packets, 0 bytes
tag information set
local tag: tunnel-head
via 0.0.0.0, Tunnel1, 0 dependencies
traffic share 1, current path
next hop 0.0.0.0, Tunnel1
valid adjacency
tag rewrite with Tu1, point2point, tags imposed: {}
via 0.0.0.0, Tunnel0, 0 dependencies
traffic share 1
next hop 0.0.0.0, Tunnel0
valid adjacency
tag rewrite with Tu0, point2point, tags imposed: {17}
0 packets, 0 bytes switched through the prefix
tmstats: external 0 packets, 0 bytes
internal 0 packets, 0 bytes
The MPLS TE tunnels are unidirectional, so a corresponding set of tunnels has to be configured separately on B1 if you want to have optimal load-balancing in both directions.
Note
In most designs, you want to achieve optimized load-balancing in one direction only (from the server toward the clients).
If you want to load-share IP packets from the LAN to the two ingress routers for any reason or if you have further IP subnets attached to the LAN via additional routers, a set of tunnels has to be configured on A2 and B2 to reach a symmetrical bidirectional design where a packet is always optimally load-shared regardless of its direction or ingress router.
Configure MPLS TE tunnels
In this section, you’ll find the steps necessary to implement load-balancing design in an IP network running OSPF routing protocol. The section lists only the steps necessary to configure the routers; in-depth technology explanation and further reading can be found in the references at the end of the article.
Before you can configure the MPLS TE tunnels in our design, you have to prepare the underlying MPLS TE infrastructure with the following steps:
Enable CEF with the ip cef global configuration command. MPLS TE tunnels can only work in conjunction with CEF switching.
Enable MPLS TE on all the routers that a MPLS TE tunnel can cross with the mpls traffic-eng tunnels global configuration command.
Specify the interface bandwidth on all interface (or at least those interface that have no hardware default, like the WAN links) with the bandwidth interface configuration command.
Specify the amount of interface bandwidth the MPLS TE tunnels can use with the ip rsvp bandwidth bandwidth interface configuration command.
Enable MPLS TE on all interfaces that a MPLS TE tunnel can cross with the mpls traffic-eng tunnels interface configuration command.
Enable OSPF support for MPLS TE with the mpls traffic-eng router-id interface and mpls traffic-eng area area router configuration commands.
The complete configuration for the A1 router is displayed in Listing 4. Similar commands have to be entered on A2, B1 and B2.
MPLS TE infrastructure configuration on A1
hostname A1
!
ip cef
!
mpls traffic-eng tunnels
!
interface Loopback0
ip address 172.16.0.11 255.255.255.255
!
interface FastEthernet0/0
mpls traffic-eng tunnels
ip rsvp bandwidth 100000
!
interface Serial0/0/0.100 point-to-point
description *** Link to B1 ***
bandwidth 2000
mpls traffic-eng tunnels
ip rsvp bandwidth 2000
!
router ospf 1
mpls traffic-eng router-id Loopback0
mpls traffic-eng area 0
Once the MPLS TE infrastructure has been configured, you have to configure two MPLS TE tunnels on each router:
Create tunnel interfaces. Specify tunnel encapsulation with tunnel mode mpls traffic-eng command and tunnel endpoint with tunnel destination ip-address command.
Note
The IP address entered in the tunnel destination command must match the IP address entered in the mpls traffic-eng router-id command on the tunnel tail router.
Specify IP address on the tunnel interfaces (most commonly MPLS TE tunnels are unnumbered interfaces).
Allow the tunnels to be used as shortcuts by the dynamic routing protocols with the tunnel mpls traffic-eng autoroute announce command.
The direct tunnel between a pair of routers shall follow the optimum IP routing path, so you can use the tunnel mpls traffic-eng path-option 1 dynamic command. The path for the tunnel across the second parallel WAN link can be specified in three ways:
You could specify the hop-by-hop explicit path listing ingress interface IP addresses or router-ids of each router in the explicit path (awkward and error-prone);
You could use MPLS TE link attributes and tunnel affinity bits to ensure that the tunnels follow specific links (manageable, but complex);
The best configuration option is the use of IP explicit address exclusion feature introduced in IOS release 12.2(4)T where you specify which IP addresses the tunnel shall not use (in our design, tunnel A1-A2-B2-B1 shall not cross the WAN link A1-B1).
The tunnel configuration on A1 is summarized in Listing 5.
Tunnel configuration on A1
interface Tunnel0
ip unnumbered Loopback0
tunnel destination 172.16.0.21
tunnel mode mpls traffic-eng
tunnel mpls traffic-eng autoroute announce
tunnel mpls traffic-eng path-option 1 explicit name Through_A2
tunnel mpls traffic-eng path-option 2 dynamic
tunnel mpls traffic-eng record-route
!
interface Tunnel1
ip unnumbered Loopback0
tunnel destination 172.16.0.21
tunnel mode mpls traffic-eng
tunnel mpls traffic-eng autoroute announce
tunnel mpls traffic-eng path-option 1 dynamic
no routing dynamic
!
ip explicit-path name Through_A2 enable
exclude-address 172.16.1.2
Summary
In this article, you’ve seen how you can use MPLS Traffic Engineering technology to configure optimal load-balancing in a highly redundant environment. The configuration example focused on a simple scenario with two parallel links between sets of redundant routers, but you can easily use traffic engineering to load balance traffic between paths of unequal length or bandwidth. Even more, Cisco IOS allows you to specify balancing ratio between the parallel tunnels.
Supplementary Information:
Router configurations for the Routing tricks section.
Router configurations for MPLS TE solution with IP explicit address exclusion feature.
Router configurations using MPLS TE link attributes and tunnel affinity bits.
More In-depth Information:
Practice the MPLS TE configuration with NIL remote labs.
Gain baseline knowledge of MPLS and MPLS TE by attending the Implementing Cisco MPLS training at NIL’s training centers.
Focus on advanced concepts of MPLS TE technology in the Advanced MPLS Traffic Engineering class.
Cisco’s Documentation
Configure Cisco Express Forwarding (CEF)