Squid + WCCP + TProxy

[Roth, Joe]

I followed the guide here to set up squid to do transparent cacheing using wccpv2, and it works quite well. So I took the next step to use tproxy.

I followed this page to introduce tproxy into the mix:

http://wiki.squid-cache.org/SquidFaq/InterceptionProxy

The kernel is compiled with tproxy, as is iptables and squid 2.7Stable7.

I have set up iptables, the ip rule and ip route according to the guide.

I'm running Ubuntu with kernel 2.6.28, iptables 1.4.3, squid 2.7.Stable7.

For some reason the traffic never makes it to port 3129. Do I need to leave the iptables nat config for 3128 even though I am using tproxy? Am I missing something here?

TIA,

--Joe

Squid port config:

# Squid normally listens to port 3128
http_port 128.226.100.61:3128 transparent
http_port 128.226.100.61:3129 tproxy

Output of iptables:

root@indianwells:~# iptables -t mangle -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DIVERT     tcp  --  anywhere             anywhere            socket
TPROXY     tcp  --  anywhere             anywhere            tcp dpt:www TPROXY redirect 128.226.100.61:3129 mark 0x1/0x1

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

Chain DIVERT (1 references)
target     prot opt source               destination
MARK       all  --  anywhere             anywhere            MARK xset 0x1/0xffffffff
ACCEPT     all  --  anywhere             anywhere
root@indianwells:~#

Joe Roth
Networking Group
Binghamton University
Ph. 607-777-7528
Fax [phone]

gNote: Information Technology Services (ITS) will never ask for personal or password information via email. If you have received an unsolicited email that appears to be from Binghamton University or Binghamton University ITS requesting password information, please DO NOT RESPOND.

[Amos Jeffries-2]

On Sun, 1 Nov 2009 08:09:52 -0500, "Roth, Joe" <jroth@...>
wrote:
> I followed the guide here to set up squid to do transparent cacheing
using

> wccpv2, and it works quite well. So I took the next step to use tproxy.
>
> I followed this page to introduce tproxy into the mix:
>
> http://wiki.squid-cache.org/SquidFaq/InterceptionProxy
>
> The kernel is compiled with tproxy, as is iptables and squid 2.7Stable7.
>
> I have set up iptables, the ip rule and ip route according to the guide.
>
> I'm running Ubuntu with kernel 2.6.28, iptables 1.4.3, squid

2.7.Stable7.
>
> For some reason the traffic never makes it to port 3129. Do I need to
> leave the iptables nat config for 3128 even though I am using tproxy? Am
I
> missing something here?

The native TPROXY that comes in kernel 2.6.28 is TPROXYv4.
Support for that version of TPROXY is only in Squid-3.1 and later.

Amos

[Roth, Joe]

Ahhhh... I will upgrade on Mon.
 
But will that also make a difference in the box itself seeing the connections coming in on 3129? i.e. when I do a "netstat -an | grep 3129", I see the box listening on 3129 but I do not see any connections on the port.
 
Thanks for the response!
 
--Joe

________________________________

From: Amos Jeffries [mailto:squid3@...]
Sent: Sun 11/1/2009 4:37 PM
To: Roth, Joe
Cc: squid-users@...
Subject: Re: [squid-users] Squid + WCCP + TProxy



On Sun, 1 Nov 2009 08:09:52 -0500, "Roth, Joe" <jroth@...>
wrote:
> I followed the guide here to set up squid to do transparent cacheing
using

> wccpv2, and it works quite well. So I took the next step to use tproxy.
>
> I followed this page to introduce tproxy into the mix:
>
> http://wiki.squid-cache.org/SquidFaq/InterceptionProxy
>
> The kernel is compiled with tproxy, as is iptables and squid 2.7Stable7.
>
> I have set up iptables, the ip rule and ip route according to the guide.
>
> I'm running Ubuntu with kernel 2.6.28, iptables 1.4.3, squid

2.7.Stable7.
>
> For some reason the traffic never makes it to port 3129. Do I need to
> leave the iptables nat config for 3128 even though I am using tproxy? Am
I
> missing something here?

The native TPROXY that comes in kernel 2.6.28 is TPROXYv4.
Support for that version of TPROXY is only in Squid-3.1 and later.

Amos

[Amos Jeffries-2]

On Sun, 1 Nov 2009 17:02:52 -0500, "Roth, Joe" <jroth@...>
wrote:
> Ahhhh... I will upgrade on Mon.
>  
> But will that also make a difference in the box itself seeing the
> connections coming in on 3129? i.e. when I do a "netstat -an | grep
3129",
> I see the box listening on 3129 but I do not see any connections on the
> port.

It makes a difference. The old TPROXYv2 used a kernel lookup similar to
the way NAT does to figure out what the  client and destination IPs were
and use them.  The new TPROXYv4 kernels send the IPs directly on the
accept()'d connection without being asked. The IPs arrive in reverse order
to usual ( local 'me' == real client IP, and remote 'client' == real client
destination IP) and can cause unsuspecting software to go badly.

The new Squid have to configure a special kernel option to indicate they
can handle this type of warped IP operations safely before the kernel will
permit connections to arrive.

Amos

>  
> Thanks for the response!
>  
> --Joe
>
> ________________________________
>
> From: Amos Jeffries [mailto:squid3@...]
> Sent: Sun 11/1/2009 4:37 PM
> To: Roth, Joe
> Cc: squid-users@...
> Subject: Re: [squid-users] Squid + WCCP + TProxy
>
>
>
> On Sun, 1 Nov 2009 08:09:52 -0500, "Roth, Joe" <jroth@...>
> wrote:
>> I followed the guide here to set up squid to do transparent cacheing
> using
>> wccpv2, and it works quite well. So I took the next step to use tproxy.
>>
>> I followed this page to introduce tproxy into the mix:
>>
>> http://wiki.squid-cache.org/SquidFaq/InterceptionProxy
>>
>> The kernel is compiled with tproxy, as is iptables and squid

2.7Stable7.
>>
>> I have set up iptables, the ip rule and ip route according to the
guide.
>>
>> I'm running Ubuntu with kernel 2.6.28, iptables 1.4.3, squid
> 2.7.Stable7.
>>
>> For some reason the traffic never makes it to port 3129. Do I need to
>> leave the iptables nat config for 3128 even though I am using tproxy?
Am
> I
>> missing something here?
>
> The native TPROXY that comes in kernel 2.6.28 is TPROXYv4.
> Support for that version of TPROXY is only in Squid-3.1 and later.
>
> Amos

[Roth, Joe]

I compiled 3.1.0.14 with the --enable-linux-netfilter option and
installed.

Everything starts and it is listening on 3129, however still nothing
arrives.

Is there any way for me to check that squid is properly enabling the
kernel option?

Thanks,

--Joe

-----Original Message-----
From: Amos Jeffries [mailto:squid3@...]
Sent: Sunday, November 01, 2009 5:33 PM
To: squid-users@...
Subject: RE: [squid-users] Squid + WCCP + TProxy

On Sun, 1 Nov 2009 17:02:52 -0500, "Roth, Joe" <jroth@...>
wrote:
> Ahhhh... I will upgrade on Mon.
>  
> But will that also make a difference in the box itself seeing the
> connections coming in on 3129? i.e. when I do a "netstat -an | grep
3129",
> I see the box listening on 3129 but I do not see any connections on
the
> port.

It makes a difference. The old TPROXYv2 used a kernel lookup similar to
the way NAT does to figure out what the  client and destination IPs were
and use them.  The new TPROXYv4 kernels send the IPs directly on the
accept()'d connection without being asked. The IPs arrive in reverse
order
to usual ( local 'me' == real client IP, and remote 'client' == real
client
destination IP) and can cause unsuspecting software to go badly.

The new Squid have to configure a special kernel option to indicate they
can handle this type of warped IP operations safely before the kernel
will
permit connections to arrive.

Amos

>  
> Thanks for the response!
>  
> --Joe
>
> ________________________________
>
> From: Amos Jeffries [mailto:squid3@...]
> Sent: Sun 11/1/2009 4:37 PM
> To: Roth, Joe
> Cc: squid-users@...
> Subject: Re: [squid-users] Squid + WCCP + TProxy
>
>
>
> On Sun, 1 Nov 2009 08:09:52 -0500, "Roth, Joe" <jroth@...>
> wrote:
>> I followed the guide here to set up squid to do transparent cacheing
> using
>> wccpv2, and it works quite well. So I took the next step to use

tproxy.
>>
>> I followed this page to introduce tproxy into the mix:
>>
>> http://wiki.squid-cache.org/SquidFaq/InterceptionProxy
>>
>> The kernel is compiled with tproxy, as is iptables and squid
2.7Stable7.
>>
>> I have set up iptables, the ip rule and ip route according to the
guide.
>>
>> I'm running Ubuntu with kernel 2.6.28, iptables 1.4.3, squid
> 2.7.Stable7.
>>
>> For some reason the traffic never makes it to port 3129. Do I need to
>> leave the iptables nat config for 3128 even though I am using tproxy?
Am
> I
>> missing something here?
>
> The native TPROXY that comes in kernel 2.6.28 is TPROXYv4.
> Support for that version of TPROXY is only in Squid-3.1 and later.
>
> Amos

[Henrik Nordstrom-5]

mån 2009-11-02 klockan 09:23 -0500 skrev Roth, Joe:
> I compiled 3.1.0.14 with the --enable-linux-netfilter option and
> installed.

> Is there any way for me to check that squid is properly enabling the
> kernel option?

The needed kernel option is enabled by iptables, not Squid.

The compile + http_port options just tells Squid to query the kernel a
little extra to get the actual address info. The actual intercept will
work even without any of that, just that the result may not be entirely
the expected..

REgards
Henrik

[Roth, Joe]

This is what I get when I put my tproxy iptables rules in:


-----Original Message-----
From: Henrik Nordstrom [mailto:henrik@...]
Sent: Monday, November 02, 2009 8:52 PM
To: Roth, Joe
Cc: Amos Jeffries; squid-users@...
Subject: RE: [squid-users] Squid + WCCP + TProxy

mån 2009-11-02 klockan 09:23 -0500 skrev Roth, Joe:
> I compiled 3.1.0.14 with the --enable-linux-netfilter option and
> installed.

> Is there any way for me to check that squid is properly enabling the
> kernel option?

The needed kernel option is enabled by iptables, not Squid.

The compile + http_port options just tells Squid to query the kernel a
little extra to get the actual address info. The actual intercept will
work even without any of that, just that the result may not be entirely
the expected..

REgards
Henrik

[Roth, Joe]

So I may have an iptables problem...

This is what I get in dmesg when I put in my iptables rules:

376.170216] ip_tables: (C) 2000-2006 Netfilter Core Team
[  376.272658] NF_TPROXY: Transparent proxy support initialized, version 4.1.0
[  376.272673] NF_TPROXY: Copyright (c) 2006-2007 BalaBit IT Ltd.

These are the iptables rules that I am using, copied straight from the wiki:

iptables -t mangle -N DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3129

I also do the following:

ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

echo 0 > /proc/sys/net/ipv4/conf/lo/rp_filter
echo 1 > /proc/sys/net/ipv4/ip_forward

Am I missing a step? I would suspect that after I do all of this I would at least see some packets hitting the box on 3129.

Thanks,

--Joe



-----Original Message-----
From: Henrik Nordstrom [mailto:henrik@...]
Sent: Monday, November 02, 2009 8:52 PM
To: Roth, Joe
Cc: Amos Jeffries; squid-users@...
Subject: RE: [squid-users] Squid + WCCP + TProxy

mån 2009-11-02 klockan 09:23 -0500 skrev Roth, Joe:
> I compiled 3.1.0.14 with the --enable-linux-netfilter option and
> installed.

> Is there any way for me to check that squid is properly enabling the
> kernel option?

The needed kernel option is enabled by iptables, not Squid.

The compile + http_port options just tells Squid to query the kernel a
little extra to get the actual address info. The actual intercept will
work even without any of that, just that the result may not be entirely
the expected..

REgards
Henrik

[Roth, Joe]

Does it appear that my iptables rules are in the correct order? I see packets matching them. Possible my problem is with the ip rule or ip route? I am using the standard ones from the wiki.

root@indianwells:~# iptables -t mangle -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DIVERT     tcp  --  anywhere             anywhere            socket
TPROXY     tcp  --  anywhere             anywhere            tcp dpt:www TPROXY redirect 128.226.100.61:3129 mark 0x1/0x1

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

Chain DIVERT (1 references)
target     prot opt source               destination
MARK       all  --  anywhere             anywhere            MARK xset 0x1/0xffffffff
ACCEPT     all  --  anywhere             anywhere

Thanks,

--Joe

-----Original Message-----
From: Henrik Nordstrom [mailto:henrik@...]
Sent: Monday, November 02, 2009 8:52 PM
To: Roth, Joe
Cc: Amos Jeffries; squid-users@...
Subject: RE: [squid-users] Squid + WCCP + TProxy

mån 2009-11-02 klockan 09:23 -0500 skrev Roth, Joe:
> I compiled 3.1.0.14 with the --enable-linux-netfilter option and
> installed.

> Is there any way for me to check that squid is properly enabling the
> kernel option?

The needed kernel option is enabled by iptables, not Squid.

The compile + http_port options just tells Squid to query the kernel a
little extra to get the actual address info. The actual intercept will
work even without any of that, just that the result may not be entirely
the expected..

REgards
Henrik