My Internet connection slows down when I download -- how can I speed it up?

published May 14, 2007, last modified Jun 26, 2013

You're browsing the Web. It's smooth, fast sailing, with pages loading in a snap. Then you start a download -- and what was formerly a fast Internet connection turns into a slower-than-dialup experience. What happened?

Why Web browsing slows down when downloading files

Believe it or not, this is common. If you're like me, you have a fast fiber, DSL or cable modem hookup to the Internet, directly hooked to your computer. You're probably using file sharing applications all the time, as well. This causes terrible Web browsing -- and you can easily figure that out, because as soon as you stop any file sharing applications, Web browsing speed returns back to normal.

The slowdown is caused by the queues at your ISP (or your modem). The ISP limits your Internet speed either at the modem, or at the router assigned to your modem. Since information can't come in as fast as possible, long queues of pending information build up in your router/modem.

The effect? After you click on a link in your Web browser, quite a lot of information (usually, your download) needs to be dispatched before you get to see the new Web page.

The solution: rate-limiting (odd, isn't it?)

Counterintuitive as it may sound, the solution to the queues is to limit the rate at which information reaches your computer, directly on your computer. By using a clever combination of bandwidth limiting and priorization of outbound packages, you can have almost-normal Web browsing speeds, combined with fast downloads.

In other words:

  • Anything your computer sends to the Internet will be given priorities, and important information will jump the queue. Sends of acknowledgements and interactive traffic (remote desktops, shells) will get priority.
  • If any computer attempts to send data to your computer too quickly, your computer will tell it to slow down. This will avoid the buildup in the queue at your ISP.

Basically, you'll be executing a tradeoff: decreasing latency at the expense of a bit (5% to 10%) of bandwidth, but the cost is certainly worth it.

Now, you may balk because this technique requires you to actually scale back (a bit) your connection speed. Do not -- if you follow through, your effective download speed will go up, because TCP acknowledgements arrive to their destination faster. If you're using BitTorrent (which requires uploading), you'll see a stable, maxed-out, download speed, instead of seeing dips in download speed when uploading.

As usual, the solution is a clever script that does everything for you, instead of you having to do it manually.

The script that solves the speed problem

Further below, you'll find instructions on how to use it.

#!/bin/bash
# Wonder Shaper
# please read the README before filling out these values
#
# Set the following values to somewhat less than your actual download
# and uplink speed. In kilobits. Also set the device that is to be shaped.

DOWNLINK=176
UPLINK=120
DEV=eth1

# low priority source netmasks
NOPRIOHOSTSRC=

# low priority destination netmasks
NOPRIOHOSTDST=

# low priority source ports
NOPRIOPORTSRC="6881 6882 13810 6346 4800"

# low priority destination ports
NOPRIOPORTDST=

# high priority source netmasks
HIPRIOHOSTSRC=

# high priority destination netmasks
HIPRIOHOSTDST=

# high priority source ports
HIPRIOPORTSRC="5900 5901"

# high priority destination ports
HIPRIOPORTDST=


if [ "$1" = "status" ]
then
        echo "Queueing disciplines:"
        tc -s qdisc ls dev $DEV

        rootclassoutput=`tc -s class ls dev $DEV | grep -A 5 "htb 1:1 "`
        allclassesoutput=`tc -s class ls dev $DEV`

        echo ""
        echo "Root class:"
        echo "$rootclassoutput"
        echo ""
        echo "Child classes:"
        echo "$allclassesoutput" | cut -d "
" -f 6,7,8,9,10,11 --complement
        exit
fi

# clean existing down- and uplink qdiscs, hide errors
tc qdisc del dev $DEV root    2> /dev/null > /dev/null
tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null

if [ "$1" = "stop" ]
then
        exit
fi


###### uplink

# we'll have four classes:
# 10: acknowledgements and traffic marked as interactive (Minimize-Delay)
# 20: high-priority outbound traffic
# 30: regular traffic
# 40: low-priority outbound traffic and traffic marked as bulk (Maximize-Throughput)

# install root HTB, point default traffic to 1:30:

tc qdisc add dev $DEV root handle 1: htb default 30

# shape everything at $UPLINK speed - prevents huge outbound queues:

tc class add dev $DEV parent 1:  classid 1:1  htb rate ${UPLINK}kbit burst 5k

tc class add dev $DEV parent 1:1 classid 1:10 htb  rate ${UPLINK}kbit        ceil ${UPLINK}kbit  burst 5k  prio 1
tc class add dev $DEV parent 1:1 classid 1:20 htb  rate ${UPLINK}kbit        ceil ${UPLINK}kbit  burst 5k  prio 2
tc class add dev $DEV parent 1:1 classid 1:30 htb  rate $[8*$UPLINK/10]kbit  ceil ${UPLINK}kbit  burst 5k  prio 3
tc class add dev $DEV parent 1:1 classid 1:40 htb  rate $[6*$UPLINK/10]kbit  ceil ${UPLINK}kbit  burst 5k  prio 4

# all get Stochastic Fairness:

tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev $DEV parent 1:30 handle 30: sfq perturb 10
tc qdisc add dev $DEV parent 1:40 handle 40: sfq perturb 10

# TOS Minimum Delay (ssh, NOT scp) in 1:10:
tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
    match ip tos 0x10 0xff \
    flowid 1:10

# To speed up downloads while an upload is going on, put ACK packets in 1:10:
tc filter add dev $DEV parent 1:0 protocol ip prio 11 u32 \
    match ip protocol 6 0xff \
    match u8 0x05 0x0f at 0 \
    match u16 0x0000 0xffc0 at 2 \
    match u8 0x10 0xff at 33 \
    flowid 1:10

# High-priority traffic in 1:20:
for a in $HIPRIOPORTDST ; do
        tc filter add dev $DEV parent 1: protocol ip prio 12 u32 match ip dport $a 0xffff flowid 1:20
done
for a in $HIPRIOPORTSRC ; do
        tc filter add dev $DEV parent 1: protocol ip prio 12 u32 match ip sport $a 0xffff flowid 1:20
done
for a in $HIPRIOHOSTSRC ; do
        tc filter add dev $DEV parent 1: protocol ip prio 12 u32 match ip src   $a        flowid 1:20
done
for a in $HIPRIOHOSTDST ; do
        tc filter add dev $DEV parent 1: protocol ip prio 12 u32 match ip dst   $a        flowid 1:20
done


# Low-priority traffic in 1:40:
for a in $NOPRIOPORTDST ; do
        tc filter add dev $DEV parent 1: protocol ip prio 13 u32 match ip dport $a 0xffff flowid 1:40
done
for a in $NOPRIOPORTSRC ; do
        tc filter add dev $DEV parent 1: protocol ip prio 13 u32 match ip sport $a 0xffff flowid 1:40
done
for a in $NOPRIOHOSTSRC ; do
        tc filter add dev $DEV parent 1: protocol ip prio 13 u32 match ip src   $a        flowid 1:40
done
for a in $NOPRIOHOSTDST ; do
        tc filter add dev $DEV parent 1: protocol ip prio 13 u32 match ip dst   $a        flowid 1:40
done

# Maximize throughput (scp, BitTorrent, uploads) in 1:40:
tc filter add dev $DEV parent 1:0 protocol ip prio 14 u32 \
    match ip tos 0x08 0xff \
    flowid 1:40

# rest is 'non-interactive' ie 'bulk' and ends up in 1:30
tc filter add dev $DEV parent 1: protocol ip prio 15 u32 \
    match ip dst 0.0.0.0/0 \
    flowid 1:30


########## downlink #############
# slow downloads down to somewhat less than the real speed  to prevent
# queuing at our ISP. Tune to see how high you can set it.
# ISPs tend to have *huge* queues to make sure big downloads are fast
#
# attach ingress policer:

tc qdisc add dev $DEV handle ffff: ingress

# filter *everything* to it (0.0.0.0/0), drop everything that's
# coming in too fast:

tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \
   0.0.0.0/0 police rate ${DOWNLINK}kbit burst 5k drop flowid :1

This is a heavily tuned version of the Wonder Shaper script that roams the Linux advanced routing and traffic control Web site.

Keep reading to find out how to install and use this script on your computer.

How to use the script

To use this script, you need to:

  1. Install it to your computer

    Copy the contents to your clipboard, then save it to a file named rc.wshaper in your /etc directory. You may need to do this task as the root user, because the /etc directory is off-limits for unprivileged users.

  2. Configure it. It's easy to set up.

    • The line that starts with DOWNLINK= should be followed by your download bandwidth in kilobits per second, minus 5% to 10%. For example: if you have a 1000 Kbps link, you should set it between 900 and 950.
    • The line that starts with UPLINK= should be followed by your upload speed minus 5% to 10%.
    • The line that starts with DEV= should contain the name of your network interface (typically eth0 -- check your computer's network configuration).

    • Optional: set low- and high-priority ports.

      The lines that start with NOPRIOPORTSRC= and HIPRIOPORTSRC= can contain a quoted list of low- and high-priority ports (whose traffic will take a back seat in the face of regular traffic). The script already ships configured with ports for VNC (as high-priority), BitTorrent, Direct Connect and Gnutella.

  3. Set it to run during startup.

    In the file named /etc/rc.local on your computer, add a line that calls this script:

    #!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.


/etc/rc.wshaper
exit 0

    Take note: in some Linux variants, the file is named /etc/rc.d/rc.local.

  4. Reboot. Or run the command directly, if you know how to.

How do I know if it's doing anything?

Simple. You call it (with administrative privileges) using the status argument on your favorite terminal program:

[~] > sudo /etc/rc.wshaper status
Password:

Queueing disciplines:
qdisc htb 1: r2q 10 default 20 direct_packets_stat 0
 Sent 72212097 bytes 206134 pkt (dropped 0, overlimits 231945 requeues 0)
 rate 0bit 0pps backlog 0b 66p requeues 0
qdisc sfq 10: parent 1:10 limit 128p quantum 1514b perturb 10sec
 Sent 5268314 bytes 97447 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
qdisc sfq 20: parent 1:20 limit 128p quantum 1514b perturb 10sec
 Sent 2002303 bytes 26155 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
qdisc sfq 30: parent 1:30 limit 128p quantum 1514b perturb 10sec
 Sent 64941480 bytes 82532 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 72656b 66p requeues 0
qdisc ingress ffff: ----------------
 Sent 113358865 bytes 167464 pkt (dropped 7565, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0

Root class:
class htb 1:1 root rate 120000bit ceil 120000bit burst 5Kb cburst 1659b
 Sent 72141381 bytes 206073 pkt (dropped 0, overlimits 0 requeues 0)
 rate 101536bit 37pps backlog 0b 0p requeues 0
 lended: 24985 borrowed: 0 giants: 0
 tokens: -9440 ctokens: -245643

Child classes:
class htb 1:10 parent 1:1 leaf 10: prio 1 rate 120000bit ceil 120000bit burst 5Kb cburst 1659b
 Sent 5268314 bytes 97447 pkt (dropped 0, overlimits 0 requeues 0)
 rate 8304bit 19pps backlog 0b 0p requeues 0
 lended: 97447 borrowed: 0 giants: 0
 tokens: 346248 ctokens: 110045

class htb 1:20 parent 1:1 leaf 20: prio 2 rate 96000bit ceil 120000bit burst 5Kb cburst 1659b
 Sent 2002303 bytes 26155 pkt (dropped 0, overlimits 0 requeues 0)
 rate 2096bit 3pps backlog 0b 0p requeues 0
 lended: 26155 borrowed: 0 giants: 0
 tokens: 432128 ctokens: 109499

class htb 1:30 parent 1:1 leaf 30: prio 3 rate 72000bit ceil 120000bit burst 5Kb cburst 1659b
 Sent 64941480 bytes 82532 pkt (dropped 0, overlimits 0 requeues 0)
 rate 92136bit 16pps backlog 0b 61p requeues 0
 lended: 57486 borrowed: 24985 giants: 0
 tokens: -216810 ctokens: -125876

class htb 1:40 parent 1:1 leaf 30: prio 4 rate 72000bit ceil 120000bit burst 5Kb cburst 1659b
 Sent 64941480 bytes 82532 pkt (dropped 0, overlimits 0 requeues 0)
 rate 92136bit 16pps backlog 0b 61p requeues 0
 lended: 57486 borrowed: 24985 giants: 0
 tokens: -216810 ctokens: -125876

As you can deduce from the output of the script, there are four classes:

  1. 1:10: acknowledgements and interactive traffic (type of service Minimize-Delay) go here
  2. 1:20: high-priority traffic get classified there
  3. 1:20: normal traffic goes here
  4. 1:30: low-priority traffic and bulk traffic (type of service Maximize-Throughput) goes here, and gets trumped by the first two classes

Lower classes trump higher classes.

You can see how many bytes have been sent in each class, and the speed (rate) at which each class is operating.

How good is it?

Very good.

Obviously, if you're downloading twenty files simultaneously (which is the case with BitTorrent download managers), it won't give you a huge speedup, but you'll notice quite the difference in Web browsing speeds.

Perhaps the biggest speedup will be felt when using remote control applications, such as SSH or VNC.

Flaws in the script?

Yeah :-(. There's no easy way to classify and give priorities to incoming traffic, so only outbound traffic (uploads and the like) is prioritized. If you have a solution to this problem, contact me immediately -- I want to send you a case of beer.

However, the inbound traffic rate is policed by the script. Thus, no queues build up at your ISP, which means that you'll enjoy faster interactivity (Web browsing comes ti mind ;-))

Keywords
Linux system administration traffic shaping TCP/IP HOWTO