| Rev | Author | # | Line |
|---|---|---|---|
| 8 | DanielLawson | 1 | !!! [Squid] Caching Proxy Server Notes |
| 14 | CraigBox | 2 | ----- |
| 8 | DanielLawson | 3 | !!Problem solving |
| 4 | !Resolving name problems | ||
| 5 | Having problems that ~http://brian/wherever/whatever doesn't resolve in Squid? This is caused by Squid running its own DNS resolver, instead of using gethostbyname(3). It pulls the IPs of the name servers out of resolv.conf(5). Add a line like this to your squid.conf: | ||
| 6 | |||
| 7 | append_domain .yourdomain.tla | ||
| 8 | |||
| 9 | Any domain without a dot in it will get that domain prepended to it; everything works nicely all of a sudden. | ||
| 10 | |||
| 11 | !Always get Connection Refused for any website | ||
| 12 | probably means that squid has run out of disk space... | ||
| 13 | IanMcDonald | 13 | |
| 14 | This is also the default configuration - all users are denied access. If you want to simply allow all users to use squid and are just installing to save bandwidth go into <tt>/etc/squid/squid.conf</tt> and change the line <tt>http_access deny all</tt> to <tt>http_access allow all</tt> and restart squid - typically by <tt>/etc/init.d/squid restart</tt> | ||
| 8 | DanielLawson | 15 | |
| 16 | !"Unable to load page" error | ||
| 17 | |||
| 18 | [Microsoft] InternetExplorer 6 SP 1 has a bug where if you are using "Basic" auth (eg, with squid), the first page afterwards will display an "Unable to load page" error. This is because MSIE tries to reuse an already closed [TCP] connection. See KB:331906. | ||
| 19 | |||
| 20 | ! Caching / Proxying [Microsoft] Windows Update | ||
| 21 | |||
| 22 | Windows Update caching works just fine, for the most part. If you have an authenticated proxy, you might want to add "*.microsoft.com", "*.windowsupdate.com" and "*.akamai.net" as an auth-bypass whitelist. | ||
| 23 | |||
| 24 | As of mid December 2004, Windows Update (under XP at least) changed the way it works. It ignores proxy settings, and attempts to make direct connections to a pool of servers. This is fairly annoying, as if you have no default route set on your workstations (a sensible security measure), you can no longer run windows update. | ||
| 25 | |||
| 26 | The subnets in question are: 207.46.0.0/16 and (I think) 64.2.21.0/24. It seems that the only solution is to allow these direct access via your firewall. | ||
| 11 | RichardCrimp | 27 | |
| 28 | To use windows update via a proxy you must configure it using proxycfg as below | ||
| 29 | |||
| 30 | proxycfg -u | ||
| 31 | |||
| 32 | will import your proxy server setting from Internet Explorer. | ||
| 17 | AlastairPorter | 33 | |
| 34 | ! Allowing [SVN] through Squid | ||
| 35 | |||
| 36 | To allow access to an apache based svn server, you should add this to your squid.conf: | ||
| 37 | <pre> | ||
| 38 | extension_methods REPORT MERGE MKACTIVITY CHECKOUT | ||
| 39 | </pre> | ||
| 8 | DanielLawson | 40 | |
| 14 | CraigBox | 41 | ----- |
| 8 | DanielLawson | 42 | !!Add-on utilities for Squid |
| 43 | |||
| 44 | ! Log Analysis (sarg) | ||
| 45 | sarg is a log file analyser for squid. It's partially useful. | ||
| 46 | |||
| 47 | [Sarg|http://web.onda.com.br/orso/index.html] is a reasonably nice tool for generating nice reports for your squid logs. I have only two problems with it currently. | ||
| 48 | * Dates on reports spanning weeks or months are often wrong - all the data is there but the title of the report says it only covers 2-5 days. | ||
| 49 | * Only shows reports of the percentage of traffic that was/was not served from the cache. Does not give an actual byte count. Sure it is easy to calculate it from the total but it would be even easier if it did it for me. | ||
| 50 | |||
| 51 | ! Log Analysis (srg) | ||
| 52 | SRG is a fast and flexible log analyser written in C/C++, it was written by MattBrown while working for [CRCnet] because none of the existing log analysation programs such as sarg were adequate. In particular SRG allows you to generate reports right down to the level of each file requested from a site, and reports can be generated in plain html or using PHP to allow you to easily integrate with your squid authentication system to restrict access to all or parts of the report. Another useful feature of SRG is the ability to generate an email every time a report is generated summarising the traffic used during the reporting period. | ||
| 53 | |||
| 54 | SRG is released under the GPL and is under active development. | ||
| 55 | |||
| 56 | Find out more about srg at http://www.crc.net.nz/software/srg.php | ||
| 57 | |||
| 58 | ! Graphing Squid data | ||
| 59 | [Here|http://www.serassio.it/SquidNT/MRTG.htm] are some other notes on [Squid], [SNMP] and [MRTG]. This shows sample [MRTG] config options for graphing some of the info. Note that you can get MRTG to talk directly to Squid's nonstard SNMP port. | ||
| 60 | |||
| 61 | ! Content Blocking | ||
| 62 | |||
| 63 | Investigate the following blacklists: | ||
| 64 | |||
| 65 | * DansGuardian (http://dansguardian.org) (GPL, but commercial users are asked to pay for DL) | ||
| 66 | * SquidGuard (http://www.squidguard.org) (GPL but you can't sell it) | ||
| 9 | MarcusKool | 67 | * [ufdbGuard] (http://www.urlfilterdb.com) (GPL but you can't sell it) |
| 8 | DanielLawson | 68 | |
| 69 | (Note from Daniel Barron, DG author: the SG clause is in violation of the GPL and thus is invalid. The DG license is fully 100% within the GPL. What is asked for is that commercial users pay to __download__ DG. I just thought I'd clarify the [FUD].) | ||
| 70 | |||
| 14 | CraigBox | 71 | ----- |
| 8 | DanielLawson | 72 | !!Useful configurations and tips |
| 73 | |||
| 74 | ! Proxy Auto Detection | ||
| 75 | |||
| 76 | To set things up so that your web browsers auto detect your proxy server, investigate [WPAD], the Web Proxy Auto Detection script. | ||
| 77 | |||
| 78 | ! Filtering - [ACL]s in squid | ||
| 79 | |||
| 80 | When specifying ACLs, dont set more than one type of acl on a single acl line. Squid ignores them. eg: | ||
| 81 | |||
| 82 | <pre> | ||
| 83 | acl lab proxy_auth labuser src 192.168.2.0/32 | ||
| 84 | acl denylab proxy_auth labuser | ||
| 85 | .... | ||
| 86 | http_access allow lab | ||
| 87 | http_access deny denylab | ||
| 88 | </pre> | ||
| 89 | |||
| 90 | doesn't work. instead: | ||
| 91 | <pre> | ||
| 92 | acl labuser proxy_auth labuser | ||
| 93 | acl labmachines proxy_auth 192.168.2.0/32 | ||
| 94 | .... | ||
| 95 | http_access allow labuser labmachines | ||
| 96 | http_access deny labuser | ||
| 97 | </pre> | ||
| 98 | |||
| 99 | will do the trick. | ||
| 100 | |||
| 101 | !URL Blocking | ||
| 102 | <pre> | ||
| 103 | acl restrictedmachine src ip.ad.dr.ess/255.255.255.255 | ||
| 104 | acl restrictedmachinesites dstdomain "/etc/squid/list-of-sites" | ||
| 105 | |||
| 106 | http_access allow restrictedmachine restrictedmachinesites | ||
| 107 | http_access deny restrictedmachine | ||
| 108 | </pre> | ||
| 109 | list-of-sites takes the form | ||
| 110 | <pre> | ||
| 111 | # banned sites list | ||
| 112 | host.domain.com | ||
| 113 | # or | ||
| 114 | .domain.com | ||
| 115 | # for everything in domain.com | ||
| 10 | MarcusKool | 116 | </pre> |
| 117 | |||
| 118 | Alternatively, an external redirector such as [ufdbGuard] is used to block URL categories. | ||
| 119 | <pre> | ||
| 120 | redirect_program /local/squid/bin/ufdbGuard -c /local/squid/etc/ufdbGuard.conf | ||
| 121 | redirect_children 2 | ||
| 8 | DanielLawson | 122 | </pre> |
| 14 | CraigBox | 123 | ----- |
| 8 | DanielLawson | 124 | |
| 14 | CraigBox | 125 | !!Authentication and transparent proxying |
| 126 | |||
| 127 | !Proxy Auth with [NTLM] | ||
| 8 | DanielLawson | 128 | |
| 129 | A full working example on having a Squid proxy pick up user information from NTLM and a MicrosoftWindows ActiveDirectory. This will allow anyone in the AD Group "Internet" to have full access to the internet, and anyone in "Domain Users" (and not in "Internet") to access sites in the "/etc/squid-allowedsites" file only. | ||
| 130 | |||
| 131 | If you are using InternetExplorer or newer [Mozilla] browsers (on MicrosoftWindows), this will work transparently using NTLM Authentication. If you're using another browser (or are running [Linux]), you'll be prompted for a username and password. | ||
| 132 | |||
| 14 | CraigBox | 133 | The format for authentication helpers has changed as of Samba 3. This example works with Squid 2.5STABLE3 and Samba 3.0.10. |
| 8 | DanielLawson | 134 | |
| 14 | CraigBox | 135 | Initially we tried to use transparent proxying AND NTLM auth, as all indications were that this should work. In practice it does not - see below. |
| 8 | DanielLawson | 136 | |
| 14 | CraigBox | 137 | After installation of all packages and config files, samba must be joined to the domain with the command <tt>net join -U Administrator</tt> - this will prompt you for the admin password. Then, teach Winbind the domain credentials: <tt>wbinfo --set-auth-user Administrator%password</tt>. |
| 8 | DanielLawson | 138 | |
| 14 | CraigBox | 139 | At every boot, Winbind must be started. Packages do this for you automatically. |
| 8 | DanielLawson | 140 | |
| 141 | Config files: | ||
| 142 | |||
| 143 | !squid.conf | ||
| 144 | |||
| 145 | <verbatim> | ||
| 15 | CraigBox | 146 | # This configuration file is setup for NTLM authentication |
| 147 | # | ||
| 148 | # Set NTLM parameters | ||
| 149 | auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp | ||
| 8 | DanielLawson | 150 | auth_param ntlm children 5 |
| 151 | auth_param ntlm max_challenge_reuses 0 | ||
| 152 | auth_param ntlm max_challenge_lifetime 2 minutes | ||
| 153 | |||
| 15 | CraigBox | 154 | # Set basic parameters |
| 155 | auth_param basic program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-basic | ||
| 156 | auth_param basic children 5 | ||
| 157 | auth_param basic realm Squid proxy-caching web server | ||
| 158 | auth_param basic credentialsttl 2 hours | ||
| 8 | DanielLawson | 159 | |
| 15 | CraigBox | 160 | # Don't query neighbours for dynamic pages |
| 161 | hierarchy_stoplist cgi-bin ? | ||
| 162 | |||
| 163 | # Don't cache replies on dynamic pages | ||
| 8 | DanielLawson | 164 | acl QUERY urlpath_regex cgi-bin \? |
| 165 | no_cache deny QUERY | ||
| 166 | |||
| 15 | CraigBox | 167 | # Define ACLs |
| 8 | DanielLawson | 168 | acl all src 0.0.0.0/0.0.0.0 |
| 169 | acl allsites dst 0/0 | ||
| 170 | acl manager proto cache_object | ||
| 171 | acl localhost src 127.0.0.1/255.255.255.255 | ||
| 172 | acl localnet src 192.168.99.0/255.255.255.0 | ||
| 173 | |||
| 15 | CraigBox | 174 | acl allowedsites url_regex "/etc/squid/allowedsites" |
| 175 | external_acl_type ntgroup %LOGIN /usr/lib/squid/wbinfo_group.pl | ||
| 176 | acl fullusers external ntgroup "/etc/squid/fullusers" | ||
| 8 | DanielLawson | 177 | |
| 15 | CraigBox | 178 | http_access allow localhost |
| 179 | http_access allow localnet allowedsites | ||
| 180 | http_access allow fullusers | ||
| 181 | http_access deny all | ||
| 8 | DanielLawson | 182 | |
| 15 | CraigBox | 183 | # Allow ICP queries from all |
| 184 | icp_access allow all | ||
| 8 | DanielLawson | 185 | |
| 15 | CraigBox | 186 | # Hostname |
| 8 | DanielLawson | 187 | visible_hostname firewall.example.co.nz |
| 188 | </verbatim> | ||
| 189 | |||
| 190 | !/etc/squid-allowedsites | ||
| 191 | |||
| 192 | <verbatim> | ||
| 193 | .foo.bar | ||
| 194 | .foo.bar.baz | ||
| 195 | </verbatim> | ||
| 196 | |||
| 197 | !/etc/squid-fullusers | ||
| 198 | <verbatim> | ||
| 199 | Internet | ||
| 200 | </verbatim> | ||
| 201 | |||
| 15 | CraigBox | 202 | (These are checked against __groups only__). |
| 8 | DanielLawson | 203 | |
| 204 | !/etc/smb.conf | ||
| 205 | <verbatim> | ||
| 206 | [global] | ||
| 15 | CraigBox | 207 | # general options |
| 208 | workgroup = EXAMPLE | ||
| 209 | netbios name = FIREWALL | ||
| 8 | DanielLawson | 210 | |
| 15 | CraigBox | 211 | # winbindd configuration |
| 212 | # default winbind separator is \, which is good if you | ||
| 213 | # use mod_ntlm since that is the character it uses. | ||
| 214 | # users only need to know the one syntax | ||
| 215 | # winbind separator = \ | ||
| 8 | DanielLawson | 216 | |
| 15 | CraigBox | 217 | # idmap uid and idmap gid are aliases for |
| 218 | # winbind uid and winbid gid, respectively | ||
| 219 | idmap uid = 10000-20000 | ||
| 220 | idmap gid = 10000-20000 | ||
| 221 | winbind enum users = yes | ||
| 222 | winbind enum groups = yes | ||
| 223 | # makes wbinfo able to see groups | ||
| 224 | client schannel = no | ||
| 8 | DanielLawson | 225 | |
| 15 | CraigBox | 226 | security = ads |
| 227 | realm = example.co.nz | ||
| 228 | password server = 10.7.x.x | ||
| 8 | DanielLawson | 229 | </verbatim> |
| 230 | |||
| 231 | You will also need to allow the user ID Squid is running as to write to the <tt>/var/lib/samba/winbindd_privileged</tt> | ||
| 232 | directory or you will get authentication failures (with errors written to cache.log). | ||
| 14 | CraigBox | 233 | |
| 234 | ! Transparent proxy and authentication | ||
| 235 | |||
| 236 | This can't work. An [excellent post on the topic to the Squid users list|http://www.squid-cache.org/mail-archive/squid-users/200202/0147.html] summarises why: | ||
| 237 | |||
| 238 | > HTTP specifies two "authentication required" error codes. One for a HTTP server (401), the other for a HTTP proxy (407). When a browser connects to a server requiring authentication, the server examines the HTTP header supplied in the request. If it includes the correct authentication information (username and password) the request is honoured and the server sends back a return code of 200. If the authentication information is not present in the header, the server responds with a return code of 401. When the browser sees this it pops up the authentication window where you type your username and password. The browser then re-submits the original request this time containing the authentication information it just collected. All future requests to the server will contain the authentication information. | ||
| 239 | > | ||
| 16 | CraigBox | 240 | > Proxy authentication is handled in a similar manner. A browser that knows it's using a proxy (in tranparent proxying, this is NOT the case) makes a connection to the proxy and issues an HTTP request. That request can contain proxy authentication information. Note that this is in a different part of the HTTP request to the web server authentication information. If the proxy requires authentication and the proxy-auth HTTP header is empty, the proxy responds with a return code of 407. When the browser receives this it pops up a window asking for the proxy username and password. Once you've typed it in, the browser resubmits the original request this time containing the proxy authentication information. All further requests to the proxy will contain the authentication information. |
| 14 | CraigBox | 241 | > |
| 242 | > If a browser is not configured to use a proxy, it will quite rightly ignore any return code of 407. Why should it give away your proxy username and password to anyone who asks for it? | ||
| 243 | |||
| 244 | > In your case you have browser->transparent proxy->auth proxy. The auth proxy can certainly request authentication of the transparent proxy. The cache_peer config line supports this with the "login=user:password" option. However, all that does is authenticate the proxy with its parent. There is no way to make the transparent proxy authenticate individual users. Even if the 407 sent by the auth proxy, could be passed from transparent proxy to browser (it can't because the transparent proxy traps it) you cannot make the browser respond because as far as it knows, it isn't using a proxy. | ||
| 245 | |||
| 246 | > As has been stated many, many times on this list: | ||
| 247 | |||
| 248 | > transparency, authentication, pick one. |
lib/blame.php:177: Warning: Invalid argument supplied for foreach() (...repeated 3 times)