Before you start, if you want an authoritive guide to the following material, read 
The Linux Advanced Routing and Traffic Control HOWTO instead.
This is a brief but concerted effort to minimise the effect of TCP traffic on interactive applications, using QoS under linux.
The major motivating factor behind this work was the effect the new queuing imposed before the bandwidth throttling at the RAN, on Telecom's network in early 2004.
I live in a flat where we pay 3rds of the cost of the internet bill, so between three people, they each should have an even crack at the bandwidth.
I devised two queuing structures that should give a decent balance between users, one for egress on the internal interface and one for egress on the external interface.
Egress on internal interface (eth0).
Root Class
Qdisc 1: PRIO
| Class 1:1: PRIO Queue priority 1 | | Qdisc 10: CBQ 128kbit | | ------------------------------------------ | | | Class 10:1(CBQ Class, 128kbit) | | | | Qdisc 101: SFQ | | | | -------------------------------------- | | | | | High Priority Per-User Queue | | | | | | For interactive applications | | | | | | and ICMP etc. | | | | | -------------------------------------- | | | | Class 10:2(CBQ Class, 128kbit) | | | | Qdisc 102: SFQ | | | | -------------------------------------- | | | | | High Priority Per-User Queue | | | | | | For interactive applications | | | | | | and ICMP etc. | | | | | -------------------------------------- | | | | Class 10:x(CBQ Class, 128kbit) | | | | Qdisc 10x: SFQ | | | | -------------------------------------- | | | | | High Priority Per-User Queue | | | | | | For interactive applications | | | | | | and ICMP etc. | | | | | -------------------------------------- | | | | Class 10:5(CBQ Class, 120kbit) | | | | Qdisc 105: TBQ (Rate 96kbit) | | | | -------------------------------------- | | | | | Low Priority Per-User Queue | | |
Routing -> | | | For TCP and other non-interactive | | | -> Network card
| | | applications. | | | | | -------------------------------------- | | | | Class 10:6(CBQ Class, 120kbit) | | | | Qdisc 106: TBQ (Rate 96kbit) | | | | -------------------------------------- | | | | | Low Priority Per-User Queue | | | | | | For TCP and other non-interactive | | | | | | applications. | | | | | -------------------------------------- | | | | Class 10:x(CBQ Class, 120kbit) | | | | Qdisc 10x: TBQ (Rate 96kbit) | | | | -------------------------------------- | | | | | Low Priority Per-User Queue | | | | | | For TCP and other non-interactive | | | | | | applications. | | | | | -------------------------------------- | | | ------------------------------------------ | | Class 1:2 (PRIO band priority 2) | | Qdisc 20: SFQ | | ------------------------------------------ | | | Bypass Queue for internal | | | | Traffic | | | ------------------------------------------ | | |
Egress on external interface (ppp0).
Root Class
Qdisc 10: CBQ 128kbit
| Class 10:1(CBQ Class, 128kbit) | | Qdisc 101: SFQ | | -------------------------------------- | | | High Priority Per-User Queue | | | | For interactive applications | | | | and ICMP etc. | | | -------------------------------------- | | Class 10:2(CBQ Class, 128kbit) | | Qdisc 102: SFQ | | -------------------------------------- | | | High Priority Per-User Queue | | | | For interactive applications | | | | and ICMP etc. | | | -------------------------------------- | | Class 10:x(CBQ Class, 128kbit) | | Qdisc 10x: SFQ | | -------------------------------------- | | | High Priority Per-User Queue | | | | For interactive applications | | | | and ICMP etc. | | | -------------------------------------- | | Class 10:5(CBQ Class, 120kbit) | | Qdisc 105: TBQ (Rate 96kbit) | | -------------------------------------- | | | Low Priority Per-User Queue | |
Routing -> | | For TCP and other non-interactive | | -> ADSL modem
| | applications. | | | -------------------------------------- | | Class 10:6(CBQ Class, 120kbit) | | Qdisc 106: TBQ (Rate 96kbit) | | -------------------------------------- | | | Low Priority Per-User Queue | | | | For TCP and other non-interactive | | | | applications. | | | -------------------------------------- | | Class 10:x(CBQ Class, 120kbit) | | Qdisc 10x: TBQ (Rate 96kbit) | | -------------------------------------- | | | Low Priority Per-User Queue | | | | For TCP and other non-interactive | | | | applications. | | | -------------------------------------- |
Linux QoS is based around a heirachical structure of queueing disciples (qdiscs), classes and filters.
There are two major types of qdiscs, those that are classful, or contain classes, and those that are classless, or leaf qdiscs.
Examples of Classless qdiscs
Examples of Classful qdiscs
Of these classful qdiscs, some such as the PRIO class automagically create classes under the qdisc. Others, such as CBQ require you to explicitly create the classes inside the qdisc if you wish to apply a qdisc to that class.
You can only apply a qdisc to a class, which is why classless qdiscs are therefore leaf nodes of the 'QoS tree' so to speak.
With this in mind, let us first create the QoS rules on the ppp0 interface.
First, lets delete everything and start on a clean slate.
root@sleepy:# tc qdisc del dev ppp0 root root@sleepy:# tc qdisc show dev ppp0
Ok, now we need to add the qdisc to the device root class, we're using CBQ
root@sleepy:# tc qdisc add dev ppp0 root handle 10: cbq bandwidth 6Mbit rate 127kbit avpkt 1000 root@sleepy:# tc qdisc show dev ppp0 qdisc cbq 10: rate 127Kbit (bounded,isolated) prio no-transmit root@sleepy:# tc class show dev ppp0 class cbq 10: root rate 127Kbit (bounded,isolated) prio no-transmit
Notice how this has created a root class, we could refer to it as 10:0 if we desired. Lets create the first of our High priority queues.
root@sleepy:# tc class add dev ppp0 parent 10: classid 10:1 cbq bandwidth 6Mbit rate 127kbit weight 20kbit prio 8 allot 1514 cell 8 maxburst 20 avpkt 200 bounded root@sleepy:# tc class show dev ppp0 class cbq 10: root rate 127Kbit (bounded,isolated) prio no-transmit class cbq 10:1 parent 10: rate 127Kbit (bounded) prio no-transmit
And apply the SFQ to traffic in that queue!
root@sleepy:# tc qdisc show dev ppp0 qdisc sfq 101: quantum 1500b qdisc cbq 10: rate 127Kbit (bounded,isolated) prio no-transmit
This is what the QoS looks like now.
Root Class
Qdisc 10: CBQ 127kbit
| Class 10:1(CBQ Class, 127kbit) | | Qdisc 101: SFQ | | -------------------------------------- | | | High Priority Per-User Queue | | | | For interactive applications | | | | and ICMP etc. | | | -------------------------------------- |
root@sleepy:# tc -d -s qdisc show dev ppp0 qdisc sfq 101: quantum 1500b limit 128p flows 128/1024
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
qdisc cbq 10: rate 127Kbit cell 8b (bounded,isolated) prio no-transmit/8 weight 127Kbit allot 1500b level 1 ewma 5 avpkt 1000b maxidle 1921us
Sent 7672003 bytes 14328 pkts (dropped 70, overlimits 0)
borrowed 0 overactions 0 avgidle 50393 undertime 0
root@sleepy:# tc -d -s class show dev ppp0 class cbq 10: root rate 127Kbit cell 8b (bounded,isolated) prio no-transmit/8 weight 127Kbit allot 1500b level 1 ewma 5 avpkt 1000b maxidle 1921us
Sent 7713158 bytes 14548 pkts (dropped 70, overlimits 0)
borrowed 0 overactions 0 avgidle 48819 undertime 0
class cbq 10:1 parent 10: leaf 101: rate 128Kbit cell 8b (bounded) prio no-transmit/8 weight 20Kbit allot 1514b level 0 ewma 5 avpkt 200b maxidle 10812us
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
borrowed 0 overactions 0 avgidle 283475 undertime 0
Well as we can see, things appear to be working, at least to a certian point.
All traffic is being queued by the root class, and not being passed to the SFQ qdisc inside class 10:1.
Now we need to use the third major building block of Linux QoS, the filter.
We need to apply a filter, to the root qdisc 10: to move certian traffic to the 10:1 class.
Linux iptables comes with a feature called fwmark, which allows packets to be tagged as they traverse firewall rules. This can be useful for tagging packets for QoS based on PROTO/IP/PORT information. The iptables MARK target, can be applied to the PREROUTING or POSTROUTING chains on any interface. Because however, I use IP MASQUERADING on my ppp0 interface, the internal IP of the source host sending data to the internet via ADSL has been lost in the ppp0 POSTROUTING chain.
Therefore, we will use the PREROUTING chain on the eth0 interface to mark the packets.
root@sleepy:# iptables -t mangle -I PREROUTING -j MARK --set-mark 100 -s 10.1.13.3/32 -i eth0 root@sleepy:# iptables -L -n -t mangle -x -v Chain PREROUTING (policy ACCEPT 367 packets, 125801 bytes)
pkts bytes target prot opt in out source destination
32 2000 MARK all -- eth0 * 10.1.13.3 0.0.0.0/0 MARK set 0x64
We are now marking packets with 100, or hex 0x64.
All that remains to get packets matching the SFQ, is to add the filter with tc:
And the results...
root@sleepy:# tc -s -d qdisc show dev ppp0 qdisc sfq 101: quantum 1500b limit 128p flows 128/1024
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
qdisc cbq 10: rate 127Kbit cell 8b (bounded,isolated) prio no-transmit/8 weight 127Kbit allot 1500b level 1 ewma 5 avpkt 1000b maxidle 1921us
Sent 22225127 bytes 39734 pkts (dropped 103, overlimits 0)
borrowed 0 overactions 0 avgidle 50393 undertime 0
root@sleepy:# tc -s -d class show dev ppp0 class cbq 10: root rate 127Kbit cell 8b (bounded,isolated) prio no-transmit/8 weight 127Kbit allot 1500b level 1 ewma 5 avpkt 1000b maxidle 1921us
Sent 22206812 bytes 39626 pkts (dropped 103, overlimits 0)
borrowed 0 overactions 0 avgidle 50393 undertime 0
class cbq 10:1 parent 10: leaf 101: rate 128Kbit cell 8b (bounded) prio no-transmit/8 weight 20Kbit allot 1514b level 0 ewma 5 avpkt 200b maxidle 10812us
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
borrowed 0 overactions 0 avgidle 283475 undertime 0
root@sleepy:# tc -s -d qdisc show dev ppp0 qdisc sfq 101: quantum 1500b limit 128p flows 128/1024
Sent 548 bytes 9 pkts (dropped 0, overlimits 0)
qdisc cbq 10: rate 127Kbit cell 8b (bounded,isolated) prio no-transmit/8 weight 127Kbit allot 1500b level 1 ewma 5 avpkt 1000b maxidle 1921us
Sent 22437536 bytes 40128 pkts (dropped 103, overlimits 0)
borrowed 0 overactions 0 avgidle 50393 undertime 0
root@sleepy:# tc -s -d class show dev ppp0 class cbq 10: root rate 127Kbit cell 8b (bounded,isolated) prio no-transmit/8 weight 127Kbit allot 1500b level 1 ewma 5 avpkt 1000b maxidle 1921us
Sent 22308840 bytes 39979 pkts (dropped 103, overlimits 0)
borrowed 0 overactions 0 avgidle 50393 undertime 0
class cbq 10:1 parent 10: leaf 101: rate 128Kbit cell 8b (bounded) prio no-transmit/8 weight 20Kbit allot 1514b level 0 ewma 5 avpkt 200b maxidle 10812us
Sent 244 bytes 4 pkts (dropped 0, overlimits 0)
borrowed 0 overactions 0 avgidle 283475 undertime 0
Alright, now we're getting some traffic in the SFQ now!
One page links to LinuxQualityOfService:
lib/BlockParser.php:505: Notice: Undefined property: _tight_top (...repeated 2 times)