Initial braindump. Please feel free to improve or explain why I'm "doing it wrong" - there's not a lot of good documentation out there for doing this, so here's a start, it works for me.

Scenario: We are a "mini ISP", or more a bandwidth reseller for our clients. We onsell bandwidth from FX Networks? to clients connecting via Velocity?. We wish to ensure international bandwidth is fairly provisioned.

Our (anonymised) network looks like this:

 Client  LAN          Client Firewall                Firewall/Router       ITP Edge Router/Firewall
{10.7.5.0/24} ---|10.7.5.254 - 1.1.1.1|--velocity--|1.1.1.254 - 1.1.2.1|--|1.1.2.254 - 1.1.3.1|-- {inet}
                  eth0            eth1     vlan     eth0           eth1    eth0           eth1

Step 1: Get a BGP feed from your ISP containing only domestic routes (administrative step) and configure quagga running ONLY the bgpd process. Configs (as on Ubuntu Hardy Heron after apt-get install quagga) as follows:

/etc/quagga/daemons

zebra=no
bgpd=yes
ospfd=no
ospf6d=no
ripd=no
ripngd=no
isisd=no

/etc/quagga/debian.conf

vtysh_enable=yes
zebra_options=" --daemon -A 127.0.0.1"
bgpd_options="  --daemon -A 127.0.0.1"
ospfd_options=" --daemon -A 127.0.0.1"
ospf6d_options="--daemon -A ::1"
ripd_options="  --daemon -A 127.0.0.1"
ripngd_options="--daemon -A ::1"
isisd_options=" --daemon -A 127.0.0.1"

/etc/quagga/bgpd.conf

hostname firewall
password secretpass
enable password secretenablepass
log syslog
!
router bgp YourASNumber
 bgp router-id 1.1.3.1
 network 1.1.1.0/24 # a line for each network route you wish to announce - may be none
 redistribute kernel
 redistribute connected
 redistribute static
 neighbor 131.203.118.254 remote-as 9503 # specific to FX - your ISP will advise you.
 neighbor 131.203.118.254 description FX
!
line vty
!

Once this is working, you should get a large list of NZ domestic routes from the output of vtysh -c 'sh ip bgp'

Step 2: Get the routes on the box doing the shaping. In our scenario, we're shaping on a different box to the one we're running our bgpd on. You may not need this setup. Either way, you need to get those routes into a "realm" which you can then shape on. We have an entry in our authorised_keys file on the firewall allowing the client to ONLY run vtysh -c 'sh ip bgp' with no password, so a simple ssh to the firewall results in the list of routes being dumped. With that in mind, here is the script (we run it hourly):

/etc/cron.hourly/getroutes

#!/bin/bash
#
# This script grabs route prefixes from bgp on the firewall and dumps them into a local route table
# This table can then be used to shape, as anything NOT in the table is international traffic

ipcmd=/sbin/ip
table=99
gw=1.1.2.254
dev=eth1

# Remove the lookup rule
$ipcmd rule del table $table

# Flush the existing routing table
$ipcmd route flush table $table

for prefix in `ssh -q firewall | grep ^\*\> | awk '{print $2}' | sed -e 's/^M//g'` ; do
        if ! [ $prefix == */*?] ; then
                addr=$prefix/`ipcalc -c $prefix`
                $ipcmd route add table $table $addr via $gw dev $dev realm 99
        else
                $ipcmd route add table $table $prefix via $gw dev $dev realm 99
        fi
done

# Add local routes
$ipcmd route add table $table 10.7.5.0/24 via 1.1.1.1 dev eth0 realm 2
# You might need to add other local routes - as usual, don't EVER paste scripts blindly.
# I've removed mine for anonymisation purposes.

# add lookup rule
$ipcmd rule add table $table

exit 0

Step 3: Shape it! This script creates the queues and sends traffic in realm 2 to the local queue, realm 99 to the national queue, and all other traffic to the international (default) queue.

/usr/local/sbin/traffic

#!/bin/sh
#
# traffic - script that configures network traffic shaping

extif=eth1

# line speed of the network interface
ifrate=1000mbit

# maximum traffic rate
maxrate=100mbit

# shaped limits
localrate=100mbit
intrate=3mbit
natrate=80mbit

TC=/sbin/tc

start() {
        # clear existing rules
        $TC qdisc del dev $extif root 2>/dev/null

        # root qdisc 1:0 (default queue - rate limit to international)
        $TC qdisc add dev $extif root handle 1: htb default 12

        # root class 1:1
        $TC class add dev $extif parent 1:0 classid 1:1 htb rate $maxrate

        # class 1:10 -- local destinations
        $TC class add dev $extif parent 1:1 classid 1:10 htb rate $localrate

        # class 1:11 -- national destinations
        $TC class add dev $extif parent 1:1 classid 1:11 htb rate $natrate

        # class 1:12 -- international destinations
        $TC class add dev $extif parent 1:1 classid 1:12 htb rate $intrate

        # qdisc defs for classes
        $TC qdisc add dev $extif parent 1:10 handle 10: sfq quantum 1514b perturb 15
        $TC qdisc add dev $extif parent 1:11 handle 11: sfq quantum 1514b perturb 15
        $TC qdisc add dev $extif parent 1:12 handle 12: sfq quantum 1514b perturb 15

        # filter for 1:10 -- local destinations
        $TC filter add dev $extif parent 1:0 protocol ip pref 100 route to 2 flowid 1:10

        # filter for 1:11 -- national routes
        $TC filter add dev $extif parent 1:0 protocol ip pref 100 route to 99 flowid 1:11
}

stop() {
        # clear existing rules
        $TC qdisc del dev $extif root 2>/dev/null
}

status() {
        echo "qdisc:"
        $TC qdisc show dev $extif
        echo "filter:"
        $TC filter show dev $extif parent 1:
        echo "class:"
        $TC class show dev $extif
}

counts() {
        echo "qdisc:"
        $TC -s qdisc show dev $extif
        echo "class:"
        $TC -s class show dev $extif
}

case "$1" in
        start)
                start
                ;;

        stop)
                stop
                ;;

        status)
                status
                ;;
        counts)
                counts
                ;;
        restart)
                stop
                start
                ;;
        *)
                echo $"Usage: $0 {start|stop|restart|status|counts}"
                exit 1
esac

exit 0