Differences between version 2 and predecessor to the previous major change of HowToADSLBandwidthManagementHOWTO.
Other diffs: Previous Revision, Previous Author, or view the Annotated Edit History
| Newer page: | version 2 | Last edited on Thursday, October 21, 2004 5:04:19 pm | by AristotlePagaltzis | Revert |
| Older page: | version 1 | Last edited on Friday, June 7, 2002 1:05:25 am | by perry | Revert |
@@ -1,484 +1 @@
-ADSL Bandwidth Management HOWTO
-!!!ADSL Bandwidth Management HOWTO
-!Dan Singletary
-
-dvsing@sonicspike.net
-
-
-__Revision History__Revision .12001-08-06Revised by: ds
-
-
-
-
-
-This document describes how to configure a linux router
-to more effectively manage outbound traffic on an ADSL modem.
-Emphasis is placed on lowering the latency for interactive
-traffic even when the upstream bandwidth is fully saturated.
-
-
-
-
-
-----; __Table of Contents__; 1. Introduction: ; 1.1. New Versions of This Document; 1.2. Disclaimer; 1.3. Copyright and License; 1.4. Feedback and corrections; 2. Background: ; 2.1. Prerequisites; 2.2. Layout; 2.3. Packet Queues; 3. How it Works: ; 3.1. Throttling Bandwidth with Linux CBQ; 3.2. N-Band Priority Queuing with sch_fwprio; 3.3. Classifying Packets with ipchains; 4. Implementation: ; 4.1. Obtaining, Compiling and Installing sch_fwprio: ; 4.1.1. Using the sch_fwprio Module; 4.1.2. Using the fwprio.diff Kernel Patch; 4.2. Setting the Queue Length; 4.3. Setting up CBQ; 4.4. Setting up Packet Classification; 5. Testing the New Queue; 6. OK It Works!! Now What?
-!!!1. Introduction
-
-The purpose of this document is to suggest a way to manage outbound
-traffic on an ADSL (or cable modem) connection to the Internet. The problem
-is that many ADSL lines are limited in the neighborhood of 128kbps for upstream
-data transfer. Aggravating this problem is the packet queue in the ADSL modem
-which can take 2 to 3 seconds to empty when full. Together this means that when
-the upstream bandwidth is fully saturated it can take up to 3 seconds for
-any other packets to get out to the Internet. This can cripple interactive
-applications such as telnet and multiplayer games.
-
-----
-!!1.1. New Versions of This Document
-
-You can always view the latest version of this document on the World
-Wide Web at the
-URL http://www.linuxdoc.org.
-
-
-
-New versions of this document will also be uploaded to various Linux
-WWW and FTP sites, including the LDP home page at
-http://www.linuxdoc.org.
-
-----
-!!1.2. Disclaimer
-
-Neither the author nor the distributors, or any other contributor of
-this HOWTO are in any way responsible for physical, financial, moral or any
-other type of damage incurred by following the suggestions in this text.
-
-----
-!!1.3. Copyright and License
-
-This document is copyright 2001 by Dan Singletary, and is
-released under the terms of the GNU Free Documentation License,
-which is hereby incorporated by reference.
-
-----
-!!1.4. Feedback and corrections
-
-If you have questions or comments about this document, please feel free
-
-
-----
-!!!2. Background
-!!2.1. Prerequisites
-
-The method outlined in this document may work in other Linux configurations
-however it remains untested in any configuration but the following:
-
-
-
-
-
-
-*
-
-586 or higher x86 PC
-
-
-*
-*
-
-Redhat Linux 6.2
-
-
-*
-*
-
-2.2.19 Kernel with QoS Support fully enabled (modules OK)
-
-
-*
-*
-
-sch_fwprio module or 2.2.19 kernel patch. You can get
-these
here.
-
-
-*----
-!!2.2. Layout
-
-In order to keep things simple, all references to network devices and
-configuration in this document will be with respect to the following
-network layout diagram:
-
-
- `-- 128kbit/s -------------- `-- 10Mbit --b
-Internet `--------------------b | ADSL Modem | `--------------------
-1.5Mbit/s --b -------------- |
-| eth0
-V
------------------
-| |
-| Linux Router |
-| |
------------------
-| .. | eth1..ethN
-| |
-V V
-Local Network
-----
-!!2.3. Packet Queues
-
-Packet queues are buckets that hold data for a network device when it
-can't be immediately sent. Most packet queues use a FIFO (first in, first out)
-discipline unless they've been specially configured to do otherwise. What this
-means is that when the packet queue for a device is completely full, the packet
-most recently placed in the queue will be sent over the device only after all
-the other packets in the queue at that time are sent.
-
-
-
-With an ADSL modem, bandwidth is asymmetric with 1.5Mbit/s typical downstream
-and 128kbit/sec typical upstream. Although this is the line speed, the interface
-to the router is typically at or above 10Mbit/s. If the interface to the Local Network
-is also 10Mbit/s, there will typically be NO QUEUING at the router when packets are sent from
-the Local Network to the Internet. Packets are sent out eth0 as fast
-as they are received from the Local Network. Instead, packets are queued at the ADSL
-modem since they are arriving at 10Mbit/s and only being sent at 128kbit/s. Eventually
-the packet queue at the ADSL modem will become full and any more packets sent to it
-will be silently dropped. TCP is designed to handle this and will adjust it's transmit
-window size accordingly to take full advantage of the available bandwidth.
-
-
-
-While packet queues combined with TCP result in the most effective use of bandwidth,
-large FIFO queues can increase the latency for interactive traffic.
-
-
-
-Another type of queue that is somewhat like FIFO is an n-band priority queue. However,
-instead of having just one queue that packets line up in, the n-band priority queue has
-n FIFO queues which packets are placed in by their classification. Each queue has a priority
-and packets are always dequeued from the highest priority queue that contains packets.
-Using this discipline FTP packets can be placed in a lower priority queue than telnet
-packets so that even during an FTP upload, a single telnet packet will jump the queue and be
-sent immediately.
-
-----
-!!!3. How it Works
-
-There are two basic steps to optimize upstream bandwidth. First we have to find a way to
-prevent the ADSL modem from queuing packets since we have no control over how it handles
-the queue. In order to do this we will throttle the amount of data the router sends out eth0
-to be slightly less than the total upstream bandwidth of the ADSL modem. This will result
-in the router having to queue packets that arrive from the Local Network faster than it is allowed
-to send them.
-
-
-
-The second step is to set up priority queuing discipline on the router.
-We'll investigate a queue that can be configured to give priority to interactive
-traffic such as telnet and multiplayer games.
-
-
-
-The final step is to configure the firewall to prioritize packets by using fwmark.
-
-----
-!!3.1. Throttling Bandwidth with Linux CBQ
-
-Although the connection between the router and the modem is at 10Mbit/s, the modem
-is only able to send data at 128kbit/s. Any data sent in excess of that rate will be queued
-at the modem. Thus, a ping packet sent from the router may go to the modem immediately, but
-may take a few seconds to actually get sent out to the Internet if the queue in the modem
-has any packets in it. Unfortunately most ADSL modems provide no mechanism to specify how
-packets are dequeued or how large the queue is, so our first objective is to move the place
-where the outbound packets are queued to somewhere where we have more control over the queue.
-
-
-
-We'll do this by using a simple implementation of CBQ (class-based queuing) to limit
-the rate at which we send packets to the ADSL modem. Even though our upstream bandwidth may be
-128kbit/s we'll have to limit the rate at which we send packets to be slightly below that. If
-we want to lower the latency we have to be SURE that not a single packet is ever queued at the
-modem. Through experimentation I have found that limiting the oubound traffic to about 90kbit/s
-with CBQ gives me almost 95% of the bandwidth I could achieve without CBQ. With CBQ enabled at this
-rate, we've prevented the ADSL modem from queuing packets.
-
-----
-!!3.2. N-Band Priority Queuing with sch_fwprio
-
-At this point we still haven't realized any change in the performance. We've merely moved the
-FIFO queue from the ADSL modem to the router. In fact, with linux configured to a default queue
-size of 100 packets we've probably made our problem worse at this point! But not for long...
-
-
-
-Unfortunately, the priority queuing discipline included with linux traffic control was not
-meant to be a queuing discipline of CBQ. Although it can be used as a leaf queuing discipline,
-there is no way to classify packets into bands. I solved this problem by changing the way
-the existing queue classifies packets. The new queue classifies packets into queues based on their
-fwmark. The lowest fwmark (0x00) is the highest priority queue. Higher fwmark'ed packets will be
-placed into lower priority queues. Packets fwmark'ed out of range will be placed as if their mark
-was 0x00 (an incentive to not incorrectly mark packets!) By using the modified queue
-(sch_fwprio.o) you can classify packets using ipchains (2.2.x) or netfilter (2.4.x) to
-set the fwmark. The new module must be installed ''prior'' to using the prio queuing
-discipline with tc. This prevents the old priority queue from being loaded by the kernel.
-
-----
-!!3.3. Classifying Packets with ipchains
-
-The final step in configuring your router to give priority to interactive traffic is
-to set up the firewall to define how traffic should be classified. This is done by setting the
-packet's fwmark field.
-
-
-
-Without getting into too much detail, here is a simplified description of how outbound packets
-might be classified into a 4-band priority queue:
-
-
-
-
-
-
-#
-
-Mark ALL packets as 0x03. This places all packets, by default, into the lowest priority queue.
-
-
-#
-#
-
-Mark ICMP packets as 0x00. We want ping to show the latency for the highest priority packets.
-
-
-#
-#
-
-Mark all packets that have a destination port 1024 or less as 0x01. This gives priority to system
-services such as Telnet and SSH. FTP's control port will also fall into this range however FTP data transfer
-takes place on high ports and will remain in the 0x03 band.
-
-
-#
-#
-
-Mark all packets that have a destination port of 25 (SMTP) as 0x03. If someone sends an email with
-a large attachment we don't want it to swamp interactive traffic.
-
-
-#
-#
-
-Mark all packets that are going to a multiplayer game server as 0x02. This will give gamers low latency but
-will keep them from swamping out the the system applications that require low latency
-
-
-#
-
-Obviously, this can be customized to fit your needs.
-
-----
-!!!4. Implementation
-
-Now with all of the explanation out of the way it's time to implement upstream bandwidth management with Linux.
-
-----
-!!4.1. Obtaining, Compiling and Installing sch_fwprio
-
-If you haven't already, you'll need to obtain the sch_fwprio module or kernel patch. You can download either
-of these here.
-
-
-
-Note: It is not necessary to compile the sch_fwprio module AND apply the fwprio.diff patch! Both
-accomplish the same thing.
-
-----
-!4.1.1. Using the sch_fwprio Module
-
-After downloading the module, extract it into an appropriate directory:
-
-
-# cd /usr/local/src
-# tar -xvzf sch_fwprio.tgz
-
-Now compile the module:
-
-
-# cd /usr/local/src/fwprio
-# make
-
-You're done compiling sch_fwprio. Go on to Setting the Queue Length.
-
-----
-!4.1.2. Using the fwprio.diff Kernel Patch
-
-You must have the kernel source package installed to apply the patch. After downloading the
-patch, apply it to the kernel source tree:
-
-
-# cd /usr/src
-# patch -b -p0 fwprio.diff
-
-Now you'll need to re-compile and install the kernel modules:
-
-
-# cd /usr/src/linux
-# make dep
-# make clean
-# make modules
-# make modules_install
-
-Now you're ready to set the queue length...
-
-----
-!!4.2. Setting the Queue Length
-
-Even though we'll be placing traffic in different bands based on priority, we still want the
-queues we set up to empty in about two seconds. This suggests a queue size of 20 packets
-for our example:
-
-
-ip link set eth0 txqueuelen 20----
-!!4.3. Setting up CBQ
-
-CBQ is poorly documented and can get very complicated, so I will attempt to keep things simple
-and explain them as best I can. Once again, the reason that we are using CBQ is to limit the rate
-that we send data out eth0. This will prevent the ADSL modem from queuing packets. Also, we will
-attach the sch_fwprio queue as a leaf queuing discipline.
-
-
-
-All of the statements should be added to your rc.local file or another appropriate startup script.
-
-
-
-First we tell Linux that we want CBQ to be the root queuing discipline on eth0:
-
-
-tc qdisc add dev eth0 root handle 128: cbq bandwidth 10Mbit avpkt 700
-
-With this line we've told Linux to use CBQ on eth0 and that the handle for this class
-is 128:, and the average packet is 700 bytes. We've also specified the total bandwidth for
-eth0 as 10Mbit/sec because this is the maximum speed at which our ethernet card can transmit
-data. This is NOT your upstream data rate! We'll specify that in the next statement:
-
-
-tc class add dev eth0 parent 128:0 classid 128:1 cbq bandwidth 10Mbit \
-rate 90Kbit allot 1514 weight 9Kbit prio 5 maxburst 1 avpkt 700 \
-bounded
-
-This statement creates the class which will throttle outbound bandwidth on eth0 (in this
-case, to 90kbit/s). Since this document is not meant to explain the inner workings of CBQ, I'll
-skip a detailed explanation. The only important numbers up there are numbers following rate and weight.
-The number following rate should be slightly lower than your upstream data rate. I use 90kbit
-which works well for my 128kbit/s upstream, giving me almost full use of my available
-bandwidth. Whatever you set your rate to, the weight should be about 1/10 of that value. Also,
-the number following allot sould be set to the mtu for eth0 (1514 works fine for ethernet).
-Also important is the ''bounded'' keyword. If this keyword is not specified,
-then the class will attempt to borrow extra bandwidth from the parent class.
-
-
-
-Now if you've decided NOT to patch your kernel, and instead have compiled the sch_fwprio
-module, you'll need to load it BEFORE you use the prio queue. Otherwise the stock sch_prio will
-be loaded:
-
-
-insmod /usr/local/src/fwprio/sch_fwprio.o
-
-Now you have to attach the queue as a leaf discipline to the CBQ:
-
-
-tc qdisc add dev eth0 parent 128:1 prio bands 4 priomap 0 1 2 3 3 3 3 3 3 3 3 3 3 3 3 3
-tc filter add dev eth0 parent 128:0 protocol ip prio 5 u32 match ip src \
-1.2.3.4/32 flowid 128:1
-
-With these statements we've created a prio queue of 4 bands
-(which is really using our new sch_fwprio code)
-and then we've used a filter to mark all of the packets
-to be handled by the 128:1 classid of our CBQ, which the
-prio filter is attached to. The priomap keyword, although unused by sch_fwprio, is still
-necessary to initialize the individual bands. You MUST specify each band number at least
-once in the 16-field priomap (in the above example I've initialized 4 bands, ..3). You
-must start at zero and end with the number of bands minus one. You must specify 16 fields
-so it's okay to repeat the last band like I've done above. All of this garbage is to
-satisfy the requirements of the old sch_prio code.
-
-----
-!!4.4. Setting up Packet Classification
-
-Here's where you get to specify how packets are classified into different priority bands
-in the queue. This is completely up to you, but here is a good place to start:
-
-
-
-Create a new chain called qos-out. Create a new rule as the first rule in the output
-chain that sends all packets to the qos-out chain (we'll send them back at the end of the
-qos-out chain):
-
-
-ipchains -N qos-out
-ipchains -I output -i eth0 -j qos-out
-
-Now we'll set up a few packet classification rules (don't forget to RETURN packets back to
-the output chain if you have other rules there that need to be checked):
-
-
-ipchains -A qos-out -m 3
-ipchains -A qos-out -p icmp -m
-ipchains -A qos-out -p tcp -s .../0 :1024 -m 1
-ipchains -A qos-out -p tcp -d .../0 :1024 -m 1
-ipchains -A qos-out -p tcp -d .../0 25 -m 2
-ipchains -A qos-out -j RETURN
-
-The first rule here marks all packets into the lowest priority band by default. The next rule
-puts ICMP packets in the highest priority band, since we will be using ping to test latency we want
-it to return an accurate result. The next two rules place packets to or from system ports (-1024)
-in band 1. Although not as high as the ICMP band, this is the band we'll use for 'interactive' traffic
-such as web requests and telnet. Note that we place port 25 outbound (SMTP) into band 2. This is
-because we don't want someone sending a file attachment in an email to swamp a telnet session. The
-final rule sends packets back to the output chain now that we're done classifying them.
-
-
-
-Use this only as a starting point. Your qos-out chain will undoubtedly become quite complex
-as you decide how you want to prioritize traffic. You may find that you need more than 4 bands.
-You will probably find that giving DNS requests high priority works well, although I haven't
-included this in the example above.
-
-----
-!!!5. Testing the New Queue
-
-The easiest way to test your new setup is to saturate the upstream with low-priority traffic.
-This depends how you have your priorities set up. For the sake of example, let's say you've placed
-telnet traffic and ping traffic at a higher priority (lower fwmark) than other high ports (that are
-used for FTP transfers, etc). If you initiate an FTP upload to saturate upstream bandwidth, you
-should only notice your ping times to the gateway (on the other side of the DSL line) increasing by
-a small amount compared to what it would increase to with no priority queuing. Ping times under 100ms
-are typical depending on how you've got things set up. Ping times greater than one or two seconds
-probably mean that things aren't working right.
-
-----
-!!!6. OK It Works!! Now What?
-
-Now that you've successfully started to manage your upstream bandwidth, you should start thinking
-of ways to use it. After all, you're probably paying for it!
-
-
-
-
-
-
-*
-
-Use a Gnutella client and SHARE YOUR FILES without adversely
-affecting your network performance
-
-
-*
-*
-
-Run a web server without having web page hits slow you down in Quake
-
-
-*
+Describe [HowToADSLBandwidthManagementHOWTO]
here.
