Josh Gentry, jgentry@swcp.com
Last Updated March 14, 2005
Acknowledgments
Disclaimer
This document carries no explicit or implied warranty. Nor is there any guarantee that the information contained in this document is accurate. It is offered in the hopes of helping others, but you use it at your own risk. The author will not be liable for any damages that occur as a result of using this document.
Conventions
Important terms and concepts, when they are introduced, may be displayed in bold. Commands included in the body of the text will be displayed in this font. Annoying Facts will be displayed in red, and will indicate pitfalls the reader should be aware of before taking any action based on this document.
This document is for those facing the common problem of using Cisco routers to load balance IP traffic across 2 or more parallel T1's. Although the problem is common, acquiring the knowledge to address the issue can be frustratingly difficult. Hopefully this will help.
What you want if you can get it, and if it will not break any of your applications, is per-packet load balancing. Load balancing on a router happens at layer 3, meaning it is a function of IP routing. Per-packet load balancing means the Cisco will route outbound IP packets in round-robin fashion, simply taking turns which interface it routes each IP packet out. Per-packet load balancing gives the most even distribution across outbound connections.
A router can load balance outbound traffic only. In order to get load balancing in both directions, the T1's on Router A must both terminate on the same Router B. Load balancing must be configured on both Router A and Router B. Easy enough if Router A and Router B are both in your control. If they are not, you will have to coordinate with the authority that controls the other router. Another unfortunate result is that per-packet load balancing on T1's to multiple ISP's is not feasible.
This method of per-packet load balancing can mean that packets in a particular connection or flow arrive at their destination out of sequence. This doesn't cause a problem for most applications, but it can cause problems for the increasingly popular streaming media, both video and audio. This includes voice over IP.
If out of order packets will cause you a problem, Cisco recommends using Multilink PPP to do your load balancing. I do not personally have experience with MLPPP between Cisco routers.
Cisco Express Forwarding (CEF) is one method that IOS can use to switch packets through the router. It is the method Cisco recommends, and seems to be the right thing most of the time. It is required for the per-packet load balancing this document discusses. You can tell if CEF is enabled by issuing the command:
show ip cef
If it is enabled you get some interesting output. If it is not enabled, IOS will tell you that it is not running. To enable CEF, in global configuration mode:
ip cef
To get load balancing to happen, you need equal cost routes pointing to the interfaces involved in the load balancing. For example, on Router A load balancing 2 T1's, the show ip route output might include something like this:
S* 0.0.0.0/0 is directly connected, Serial0/1
is directly connected, Serial0/0
And on Router B the show ip route output might include
something like this:
ip route 192.168.8.0 255.255.255.0 Serial1/3 10 ip route 192.168.8.0 255.255.255.0 Serial1/0 10
The magic IOS command you have been looking for but couldn't find on Cisco's website is:
ip load-sharing per-packet
You apply this to each interface you want to participate in per-packet load balancing. Remember, you want to do this on the routers at both ends of the T1's.
My experience is that this works like a charm. Graphs I look at everyday of traffic on T1's configured this way are indistinguishable from each other. The balancing is near perfect.