Penguin
Recent Changes
Blame: TrafficShaping
EditPageHistoryDiffInfoLikePages
Annotated edit history of TrafficShaping version 20, including all changes. View license author blame.
Rev Author # Line
15 JohnMcPherson 1 This describes traffic shaping under linux. This uses the __tc__ (traffic
18 IanMcDonald 2 control) program from the IpRoute package.
15 JohnMcPherson 3
18 IanMcDonald 4 Make sure you have the correct [Kernel] support for "QoS". In your
15 JohnMcPherson 5 LinuxKernel .config file, you will probably need support for the
6 following:
7 # CONFIG_NET_SCHED
8 # CONFIG_NET_SCH_CBQ (support for the cbq class, used in this script)
9 # CONFIG_NET_SCH_HTB (htb class, not used in this script)
10 # CONFIG_NET_SCH_PRIO (prio class, used in this script)
11 # CONFIG_NET_SCH_SFQ Stochastic Fairness Queueing qdisc, not used
12 # CONFIG_NET_SCH_TBF Token Bucket Filter qdisc, not used in this script
13 # CONFIG_NET_CLS_U32 used for matching in filters?
14
20 GreigMcGill 15 ----
16
17 See RouteBasedTrafficShaping for a somewhat complicated example for shaping based on a BGP feed of domestic routes.
15 JohnMcPherson 18
19 ----
20
21 This is a script I use to throttle one machine down to half our ADSL rate. This machine is used for downloading large files (for example .iso's of the latest LinuxDistribution), but we don't want it impacting the rest of our machines. This example was stolen from the Advanced Router HOWTO, and cleaned up a bit by me.
22
23 You run this script on your gateway rate limiting data to your internal client machine(s).
24
17 JohnMcPherson 25 <pre>
15 JohnMcPherson 26 #!/bin/sh
27
28 # The device data is going *out* of. You can't stop a machine recieving data (short of firewalling)
29 # But you can limit how fast it *sends* data.
16 JohnMcPherson 30 # this is the device that goes to the clients on our LAN
15 JohnMcPherson 31 DEV=eth0
32
16 JohnMcPherson 33 # The IP you want to throttle. Any IP that matches this destination
34 # will have its traffic added to our (single) child class.
35 # If you want to rate-limit all your clients, set it to something like:
36 # IP=0.0.0.0/0
37 # Note that this script will also ratelimit traffic generated by this
38 # router to any matching IP addresses!
15 JohnMcPherson 39 IP=10.10.10.13
16 JohnMcPherson 40
15 JohnMcPherson 41
42 # How fast your internal network is. This is used to estimate the rates more accurately. If this
43 # is wrong then the rate will be out by a little bit or something.
44 LINERATE=100mbit
45
46 # How fast you want them to be able to d/l at. "kbps" means "kilobytes per second", don't get it
47 # confused with "kbits" which is approximately 10 times slower :)
48 THROTTLERATE=15kbps
49
50 # Where the tc executable is.
51 TC=/sbin/tc
52
17 JohnMcPherson 53 if ~[ ! -x $TC ]; then
15 JohnMcPherson 54 echo Cant find $TC, aborting
55 exit 1
56 fi
57
58 # Remove any previous queuing. This probably removes any other policies you have on this interface
59 # too. Oh well.
60 $TC qdisc del dev $DEV root 2>/dev/null
61
62 # Add a Queuing Discipline. A Queuing discipline is what manages the queues.
63 # we're using the "cbq" discipline, and we're saying that the average packet size is 1000 bytes
64 # (probably completely wrong :) Once again this is just a parameter to make it more accurate.
65 $TC qdisc add dev $DEV root handle 1: cbq avpkt 1000 bandwidth $LINERATE
66
67 # Create a child class, also a cbq, rate limited to $THROTTLERATE. "__allot__" is how much data the
68 # parent class gets from each child class in turn. This must be at least the [MTU] (since you
69 # can't send partial packets you must be able to send at least one entire packet).
70 # "__prio__" is used when there is more than 1 child class, and both have data in their queue.
71 # "__bounded__" means that it cannot exceed this rate, if this was left off, I think it means that
72 # when the link is saturated that this can't use more than $THROTTLERATE and everyone else can
73 # share the rest. An example use of this might be to set $THROTTLERATE to 0 and remove bounded
74 # meaning that everything else can use the link in preference. I think.
75 # "__isolated__" I'm not sure about, I think it means it doesn't interact with any other rules.
76 $TC class add dev $DEV parent 1: classid 1:1 cbq rate $THROTTLERATE \
77 allot 1500 prio 5 bounded isolated
78
79 # Add a filter to the parent to redirect traffic into the the children classes.
80 # (in this case, only a single child). Traffic that doesn't fall into the child class
81 # will not be rate limited (depending on the parent class's settings, I guess)
82 #
83 # This uses the "u32" filter which matches based on header fields in the IP packet.
84 # If you want to match on multiple rules you can use "match ip dest $IP src $IP" etc.
85 # I think if you want to do anything interesting with TC you probably want
86 # to use fwmark from iptables(8).
87 $TC filter add dev $DEV parent 1: protocol ip prio 16 u32 \
88 match ip dst $IP flowid 1:1
89
90 # Peturb the random hash every 10s
91 # Ok, what cbq does (I think) is put everything into buckets based on a hash function. However
92 # sometimes you end up with hash collisions meaning that data will be occasionally lumped together
93 # with other data and they will both be rate limited as if they were one connection. The way
94 # around this is to change the hash function frequently so that this effect is reduced. However
95 # doing this too often makes your rate limiting less accurate, doing it too rarely means that
96 # data is incorrectly classified as above, so we tell the kernel to change the hash every 10s.
97 $TC qdisc add dev $DEV parent 1:1 sfq perturb 10
17 JohnMcPherson 98 </pre>
15 JohnMcPherson 99
100 Hopefully this is enough to get people started, please, if you know anything more add it to this page. I found the advanced router howto very oblique in it's information.
16 JohnMcPherson 101 ----
17 JohnMcPherson 102 Similar script, for the case where your router is also a fileserver (and you don't want to rate-limit traffic from it to the lan clients, obviously).
16 JohnMcPherson 103
104 My server has <tt>eth1</tt> going to the ISP, and <tt>eth0</tt> going to the lan clients. This doesn't rate-limit upstream connections, but few users on our lan use much upstream bandwidth. This script rate-limits to just below our ADSL bandwidth, so that packets get dropped (and [TCP] adjusts its sending rate) rather than getting queued in [TelecomNZ]'s equipment.
105
106 <verbatim>
107 /bin/sh
108
109 DEV=eth0
110 # the IP address of the above device (so it isn't ratelimited)
111 SERVERIP=10.21.1.2
112 # lan clients to rate-limit
113 LIMITIPS=10.21.1.0/24
114
115 # remove any existing queue discipline: (might say no such file or dir)
116 tc qdisc del dev eth0 root 2> /dev/null
117
118 # quit this script immediately if any command returns error
119 set -e
120
121 # create a root queuing discipline for our interface
122 tc qdisc add dev $DEV root handle 1:0 cbq bandwidth 100Mbit avpkt 1000 cell 8
123
124 # create a class called 1:1
125 tc class add dev $DEV parent 1:0 classid 1:1 cbq bandwidth 100Mbit \
126 prio 8 allot 1514 cell 8 rate 100Mbit maxburst 20 avpkt 1000
127
128 # create a sub-class of 1:1 called 1:10 that is rate-limited to 105kbit
129 tc class add dev $DEV parent 1:1 classid 1:10 cbq bandwidth 100Mbit \
130 rate 105Kbit prio 1 allot 1514 cell 8 maxburst 20 \
131 avpkt 1000 bounded
132
133 # create a sub-class called 1:20 that isn't limited, for locally generated traffic
134 tc class add dev $DEV parent 1:1 classid 1:20 cbq allot 1514 avpkt 1000 \
135 rate 100Mbit bandwidth 100Mbit prio 2
136
137 # locally generated traffic should go to the appropriate sub-class
138 tc filter add dev $DEV parent 1:0 protocol ip prio 1 u32 \
139 match ip src $SERVERIP/32 flowid 1:20
140 # not sure if this is really needed... traffic from one interface to another?
141 tc filter add dev $DEV parent 1:0 protocol ip prio 1 u32 \
142 match ip dst $SERVERIP/32 flowid 1:20
143
144 # traffic to our lan (that didn't match earlier rules) should go to appropriate sub-class
145 tc filter add dev $DEV parent 1:0 protocol ip prio 2 u32 \
146 match ip dst $LIMITIPS flowid 1:10
147
148 </verbatim>
149
15 JohnMcPherson 150
151 ----
152 Some points to remember:
153 !!Outgoing interface
154 You the interface you use must be your *outgoing* interface, not your *incoming* interface. Getting this confused will cause this to be of no use.
155
156 !!Tag data in the right direction
157 I use "dst $IP" for 'traffic destined to $IP', if you want traffic *from* an IP use 'src $IP' instead.
158
159 ----
160
161 I made some adjustments to the above script to split up the ADSL upload in my flat network. This was to ensure that no one person can whore the upload, which makes everything laggy for everyone else. (I found the wondershaper disappointing)
162
163 Notes:
164 This will not work with masquerading
165 The network connection can still get easily saturated
16 JohnMcPherson 166 <verbatim>
15 JohnMcPherson 167 #!/bin/sh
168 # List of IPs to have upload throttled
169 IPS=`seq 114 117 | awk '{print "1.2.3."$1}'`
170
171 LINERATE=2mbit
172 THROTTLERATE=14kbps
173
174 tc qdisc del dev ppp0 root 2>/dev/null
175
176 tc qdisc add dev ppp0 root handle 1: cbq avpkt 1000 bandwidth $LINERATE
177
178 for IP in $IPS;
179 do
180 echo throttling $IP
181
182 LASTTHING=`echo $IP | cut -d . -f 4`
183
184 tc class add dev ppp0 parent 1: classid 1:$LASTTHING cbq rate \
185 $THROTTLERATE allot 1500 prio 5 bounded isolated
186
187 tc filter add dev ppp0 parent 1: protocol ip prio 16 u32 \
188 match ip src $IP flowid 1:$LASTTHING
189
190 tc qdisc add dev ppp0 parent 1:$LASTTHING sfq perturb 10
191 done;
16 JohnMcPherson 192 </verbatim>
15 JohnMcPherson 193 ----
194 After a bit of fiddling I've managed to get TrafficShaping working on a per protocol (read port) basis
195
196 as per below
197
198 I wanted to limit my personal machine at work to only use 5kbps of bandwidth but ran into the quandry that
199 my machine also runs nagios for monitoring.
200
201 I started with the above but found that when the 5kbps limit was reached, all the nagios ping tests
202 started to go critical because of the latency introduced, so we needed to differentiate between different
203 ports and protocols
204
205 so I ended up with this script
16 JohnMcPherson 206 <verbatim>
15 JohnMcPherson 207 #!/bin/sh
208
209 DEV=eth0
210
211 IP=203.97.10.61
212
213 LINERATE=2mbit
214
215 THROTTLERATE=5kbps
216 ICMPRATE=40kbps
217 HTTP=80
218
219 # Where the tc executable is.
220 TC=/sbin/tc
221
222 if ! test -x $TC; then
223 echo Cant find $TC, aborting
224 exit 1
225 fi
226
227 $TC qdisc del dev $DEV root
228
229 $TC qdisc add dev $DEV root handle 1: cbq avpkt 1000 bandwidth $LINERATE
230
231 $TC class add dev $DEV parent 1: classid 1:1 cbq rate $THROTTLERATE \
232 allot 1500 prio 5 bounded isolated
233 $TC class add dev $DEV parent 1: classid 1:2 cbq rate $ICMPRATE \
234 allot 1500 prio 5 bounded isolated
235
236 # Filter ICMP traffic to class 1:2
237 $TC filter add dev $DEV parent 1: protocol ip prio 16 u32 \
238 match ip src $IP match ip protocol 1 0xFF flowid 1:2
239 # Filter port 80 (tcp and udp) to class 1:1
240 $TC filter add dev $DEV parent 1: protocol ip prio 16 u32 \
241 match ip src $IP match ip sport $HTTP 0xFFFF flowid 1:1
242
243
244 $TC qdisc add dev $DEV parent 1:1 sfq perturb 60
245
246 #Display Traffic Shaping details
247 echo "---- qdisc parameters Ingress ----------"
248 $TC qdisc ls dev $DEV
249 echo "---- Class parameters Ingress ----------"
250 $TC class ls dev $DEV
251 echo "---- filter parameters Ingress ----------"
252 $TC filter ls dev $DEV
16 JohnMcPherson 253 </verbatim>
15 JohnMcPherson 254 ''note that sport and protocols require 2 operands, the port/protocol number and a mask''
255
256
257 ----
258
259 [Ingress] shaping
260
261 It is possible to perform ingress shaping using a similar process. Your version of tc has to have ingress support compiled in - it appears that some RedHat versions may not have this.
262
263 The following script will limit traffic from source port 80 (ie, the return-path from a web connection) to 100kbit. It applies these rules on ppp0, which is my external interface.
16 JohnMcPherson 264 <verbatim>
15 JohnMcPherson 265 #!/bin/sh
266
267 TC=/sbin/tc
268 IPTABLES=/sbin/iptables
269
270 DEV=ppp0
271
272 MAXRATE=128kbit
273 THROTTLE=100kbit
274 BURST=5000
275 MTU=1492
276
277 # Mark traffic with a source port of 80 with the mark 1
278 $IPTABLES -A PREROUTING -i $DEV -t mangle -p tcp --sport 80 -j MARK --set-mark 1
279
280 # Delete the old ingres rule
281 $TC qdisc del dev $DEV ingress
282
283 # then add the queuing discipline
284 $TC qdisc add dev $DEV handle FFFF: ingress
285 # apply the actual filter.
286 $TC filter add dev $DEV parent ffff: protocol ip prio 50 handle 1 fw \
287 police rate $THROTTLE burst $BURST mtu $MTU drop flowid :1
16 JohnMcPherson 288 </verbatim>
15 JohnMcPherson 289
290 If I look at the output of wget, its reasonably good at limiting a port 80 download to 10 - 12k/second, which is about right for what we asked. If i look at my ppp0 usage meter in gkrellm, it seems to be using more bandwidth than it should - spends a lot of time at 16 or 17 K/s incoming. Running iptraf on ppp0, in detailed statistics mode, shows that my incoming rate seems to be about 100kbit/sec, although it tends to be a bit higher than this normally. I also tested, and verified, that traffic not caught by the above script - eg, FTP traffic, still obtained full rate
291
16 JohnMcPherson 292 In comparison with a by-port filter such as the one prior to the ingress script, I see a high level of fluctuation in the download rate, in all three test cases. Whether this is to do with some misconfiguration on my part I dont know.
18 IanMcDonald 293 ----
19 IanMcDonald 294 See also NetEm for information on simulating loss and delay.<br>
295 See also LinuxQualityOfService