Load Balancing over Multiple Providers with Debian GNU/Linux

This article describes my experiences with using iproute2 to load balance traffic over 2 ISPs with Debian GNU/Linux. Feedback welcome at mailto:raju(at)linux-delhi(dot)org.

Author and copyright

Copyright 2008 by Raj Mathur. This document may be reproduced under the terms of the Creative Commons Share Alike License.

Background

I've been using MTNL as my Internet Service Provider (ISP) for a couple of years now. A month or so ago Airtel started providing Internet services in my area, so I decided to sign up with them too (don't ask me why). After much headache, Airtel finally installed a new phone line and DSL modem yesterday, so now I have Internet access through 2 different ISPs.

Naturally, the first thing I wanted to try was load balancing traffic over these two links, so that I could double my effective bandwidth. This document is the result of my experiments.

My Setup

MTNL

I'm using the MTNL DSL modem/router (a D-Link 502-T) in Bridge mode. This means that I run PPP on my computer and the D-Link just works as a modem to connect my computer to the MTNL Access Concentrator. Details on how to achieve this are given in my post to ILUGD at http://www.mail-archive.com/ilugd@lists.linux-delhi.org/msg13566.html .

Note that with this setup I get a device named ppp0 on my computer.

The speed of the connection is 256 Kb/s.

Airtel

Airtel provided me with a Beetel-branded 220Bx1 ADSL modem. I've not bothered to set it up in bridge mode yet, so my computer just uses it as a router.

The speed of the connection is 256 Kb/s.

Computer

I'm running Debian GNU/Linux testing (i386) on an AMD64 desktop.

Objective

To set up the computer to use both links automatically. This does not mean that a speed of any single connection is doubled. It does mean that multiple connections will be spread over the two links, so if you're downloading multiple files, you should see double throughput.

IP and Routing Setup

I currently have the following IPs setup on the computer and the routers (IP addresses have been obscured to protect the innocent):

There's no remote address for the Airtel link since the Airtel modem (192.168.1.1) is being used as a router.

Load Balancing Setup

I used the excellent guides at http://lartc.org/ to help me setup load balancing over the two links. The rest of this document assumes that you have the iproute2 package installed on your computer -- install it now if you haven't already! On Debian (and presumably other deb-based distributions) the command is:

  aptitude install iproute

Once you have iproute2 in place, you need to:

1. Create routing tables
2. Ensure that the right source address gets chosen for each interface
3. Set up the routing rules
4. Set up load balancing

Create routing tables

I created two tables named mtnl and airtel. To do this, edit /etc/iproute2/rt_tables and add the following lines at the end:

  1    mtnl
  2    airtel

You can choose whatever names you want instead of mtnl and airtel. Assign them numbers (1 and 2 here) that do not conflict with any other entry in the file.

Set up routing for the tables

Now tell the Linux kernel which routes belong to which table. For MTNL (see the IP list above!):

  ip route add 10.3.4.1 dev ppp0 src 10.1.2.50 table mtnl
  ip route add default via 10.3.4.1 dev ppp0 table mtnl

and for Airtel:

  ip route add 192.168.1.0/24 dev eth0 src 192.168.1.2 table airtel
  ip route add default via 192.168.1.1 dev eth0 table airtel

The first line on the above tables builds the route to the gateway and the second line sets the default route via the gateway.

If you're having a WTF? moment here, wondering how we're adding two default routes to a system, don't worry. The default routes are only in the tables, not in the main kernel routing table. We'll tie them up a bit later.

Ensure that the right source address gets chosen for each interface

We don't want packets from 192.168.1.2 going to the ppp0 interface, or vice versa, so we tell the kernel which IP address to use when sending packets to an interface:

  ip route add 10.3.4.1 dev ppp0 src 10.1.2.50
  ip route add 192.168.1.0/24 dev eth0 src 192.168.1.2

This ensures that packets that are sent to ppp0 go with source IP 10.1.2.50; similarly, packets sent out on eth0 will carry source IP address 192.168.1.2.

Set up routing rules

Next we tell the kernel which routing table to consult to send packets depending on the source IP address the packets carry. This operation and the previous one seem to complement each other:

  ip rule add from 10.1.2.50 table mtnl
  ip rule add from 192.168.1.2 table airtel

Set up load balancing

Right, we're nearly done here. So far we've set up two different routing tables, and associated source IPs with each table. Now we must tell the kernel to distribute connections over these two routes

  ip route del default

  ip route add default scope global \
   nexthop via 10.3.4.1 dev ppp0 weight 1 \
   nexthop via 192.168.1.1 dev eth0 weight 1

(The second command is a single line -- I've broken it up into multiple lines for readability.)

This command tells the kernel to assign two default routes to the computer, one over the mtnl routing table and one over the airtel table. The weight parameter determines what preference the kernel will use when choosing one table over the other: since both my links are 256Kb/s, I assign the same weight to each link. If I had, e.g. a 512Kb/s MTNL link and a 256Kb/s Airtel link, I would use weight 2 for the mtnl table and weight 1 for the airtel table. Then the kernel would establish twice as many connections over the MTNL link as it would over the Airtel link.

That's it, if all has gone well so far your computer will be load balancing over the two links now.

Seeing your routes

You can use the following commands to see your routes and rules:

View the main kernel routing table:

  ip route show

View the mtnl and airtel tables:

  ip route show mtnl
  ip route show airtel

View rules:

  ip rule show

Testing

If everything works as expected, testing is not easy. You can do the following to ensure that load is getting distributed over the two links:

Use tcpdump

tcpdump is $deity's gift to network testers. On my machine, I establish multiple downloads (I use wget) from multiple sites, and run the following commands (in two separate windows) to view the traffic on the two links:

  tcpdump -ippp0 -q -t -n
  tcpdump -ieth0 -q -t -n host 192.168.1.2

If you try this, you should see tons of lines of data streaming by in both the windows. If only one seems to be active, then there is something wrong with your setup and only one link is being used.

Use a torrent client

You can use a Bittorrent client like ktorrent to download some popular file. If everything is working properly you should see an effective download speed higher than either of your link speeds. This would happen since Bittorrent breaks a file into chunks, and downloads multiple chunks simultaneously from multiple peers. For example, here's a screen shot of ktorrent downloading a Debian DVD ISO. Look at the Down Sp (eed) column to see load balancing in action over two 256Kb/s links.

Caveats

Some things are not going to play too well with a load balanced computer:

Masquerading and NAT

If you have other computers on your network that you're masquerading through this computer on which you have set up load balancing, you may need to do more stuff to get their connections load balanced too. Please see discussions on the LARTC mailing list (archives at http://mailman.ds9a.nl/pipermail/lartc/) for more information.

Incoming connections

You may not be able to load balance incoming connections, e.g. VoIP with Ekiga. For applications that need to allow incoming connections, I'd suggest choosing one interface and forcing the application to publish that interface's address for incoming connections. It may be possible to load balance such applications too, but I personally haven't tried it.

Multiple routers

If both your ISP modems are in router mode, you may have issues getting load balancing to work. This is because both the connections will be using the same interface ( eth0). (Note that I'm using two interfaces, ppp0 and eth0, so I don't face this problem).

One solution would be to use at least one of the modems in bridged mode and let your computer do the PPPoE (PPP over Ethernet) instead of the modem.

I hear that VLANs may let you solve this problem too, but that's something you'll need to look up yourself on the 'net.

Multiple connections to one host

If you're downloading multiple files from a single host you will not get the benefits of load balancing. The first time you hit a host the kernel decides which link to use. After that, the host is in the kernel's route cache, and subsequent connections to that host will always use the same link.

Conclusion

With a bit of effort (about an hour's work) I was able to get load balancing working fine over two ISPs on my computer. I haven't really used it to download anything particularly heavy yet, but it's tested and working to the best of my knowledge. If you have suggestions for improving this document I'd like to hear about them.

Resources

The primary resource for all Linux routing-related information is the Linux Advanced Routing and Traffic Control HOWTO site at http://lartc.org .

Change History

2008-01-18: Initial publication.