Icap Reqmod

[Robert Szabo-2]

From: Robert Szabo
Sent: Tuesday, November 03, 2009 10:46 PM
To: squid-users@...
Subject: Icap Reqmod


Hi all,

I am attempting to use the icap client interface to perform url redirects.  I have done this using my icap server connecting to several commercial gateway boxes without issue however, with squid it's not working as I had hoped.

A typical icap reqmod would appear something like:

REQMOD icap://localhost:8082/ ICAP/1.0
Host: localhost
Connection: close
Encapsulated: req-hdr=0, null-body=117
X-UserId: 55555555555

GET / HTTP/1.1
Host: fastclick.net
Accept: text/html, text/plain
Accept-Encoding: compress
Pragma: no-cache

And a blocking or redirect response would look like:

ICAP/1.0 200 OK
Encapsulated: req-hdr=0, req-body=148
ISTag: istag
Server: AMS-ICAP-Server-Software/1.0

GET www.nogo.com/redirect.pl?cat=3&a=b HTTP/1.1
Accept: text/html, text/plain
Accept-Encoding: compress
Pragma: no-cache
Content-Length: 2


You will notice the GET and Host:  portions in the reqmod, these are essential for redirection to work.

However with Squid 3.0 and Squid 3.1, the reqmod does not look right, it is missing the GET and instead has HEAD in that location and does not split out the query path from the host :

REQMOD icap://127.0.0.1:8082/reqmod ICAP/1.0
Host: 127.0.0.1:8082
Date: Tue, 03 Nov 2009 23:38:31 GMT
Encapsulated: req-hdr=0, null-body=135
Allow: 204

HEAD http://www.888.com/ HTTP/1.0
User-Agent: Wget/1.11.4 (Red Hat modified)
Accept: */*
Host: www.888.com
X-UserId: 5555555555

I have tried responding in the same vane as the message above replacing the url in the HEAD line, however, the request goes through fine without considering the redirect.

Can someone provide a sample reqmod and redirect response that would work?


Many Thanks,

Bob
****************************************************************************************
This email and any files transmitted with are confidential and intended solely for the
use of the individual or entity to whom they are addressed.  If you have received this
email in error then please delete it and notify the sender. Do not make a copy or forward
it to anyone.  This footnote also confirms that this email message has been swept for the
presence of computer viruses.

Adaptive Mobile Security Ltd, Dublin Technology Centre, Taylors Lane, Dublin 8,Ireland
Directors: B Collins, L Burke (UK), G. Maclachlan (UK), S. Tirtey (DE).
Registered in Ireland, Company No. 370343, VAT Reg.No.IE63903430
****************************************************************************************

[Amos Jeffries-2]

Robert Szabo wrote:

>
> From: Robert Szabo
> Sent: Tuesday, November 03, 2009 10:46 PM
> To: squid-users@...
> Subject: Icap Reqmod
>
>
> Hi all,
>
> I am attempting to use the icap client interface to perform url redirects.  I have done this using my icap server connecting to several commercial gateway boxes without issue however, with squid it's not working as I had hoped.
>
> A typical icap reqmod would appear something like:
>
> REQMOD icap://localhost:8082/ ICAP/1.0
> Host: localhost
> Connection: close
> Encapsulated: req-hdr=0, null-body=117
> X-UserId: 55555555555
>
> GET / HTTP/1.1
> Host: fastclick.net
> Accept: text/html, text/plain
> Accept-Encoding: compress
> Pragma: no-cache
>
> And a blocking or redirect response would look like:
>
> ICAP/1.0 200 OK
> Encapsulated: req-hdr=0, req-body=148
> ISTag: istag
> Server: AMS-ICAP-Server-Software/1.0
>
> GET www.nogo.com/redirect.pl?cat=3&a=b HTTP/1.1
> Accept: text/html, text/plain
> Accept-Encoding: compress
> Pragma: no-cache
> Content-Length: 2
>

Missing either protocol spec http:// or path-spec / at the start of the
URL, and a Host: header....

>
> You will notice the GET and Host:  portions in the reqmod, these are essential for redirection to work.
>
> However with Squid 3.0 and Squid 3.1, the reqmod does not look right, it is missing the GET and instead has HEAD in that location and does not split out the query path from the host :
>
> REQMOD icap://127.0.0.1:8082/reqmod ICAP/1.0
> Host: 127.0.0.1:8082
> Date: Tue, 03 Nov 2009 23:38:31 GMT
> Encapsulated: req-hdr=0, null-body=135
> Allow: 204
>
> HEAD http://www.888.com/ HTTP/1.0
> User-Agent: Wget/1.11.4 (Red Hat modified)
> Accept: */*
> Host: www.888.com
> X-UserId: 5555555555
>
> I have tried responding in the same vane as the message above replacing the url in the HEAD line, however, the request goes through fine without considering the redirect.
>
> Can someone provide a sample reqmod and redirect response that would work?
>
>
> Many Thanks,
>
> Bob

You seem to have generated that test request with wget --spider or -N
options and with wget configured to use a proxy.

That means it generates HEAD requests instead of GET and formats the
request URL suitable for a proxy to handle easily. Squid is simply
passing what it is given to the ICAP server.

What exact releases of squid did you test with?

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
   Current Beta Squid 3.1.0.14

[Robert Szabo-2]

Hi Amos,

Many thanks for the quick response.

I am currently using Current Beta Squid 3.1.0.14.

I removed the --spider option from the wget command and this is the
output

-- reqmod issues by squid --
REQMOD icap://127.0.0.1:8082/reqmod ICAP/1.0
Host: 127.0.0.1:8082
Date: Wed, 04 Nov 2009 13:45:18 GMT
Encapsulated: req-hdr=0, null-body=134
Allow: 204

GET http://www.888.com/ HTTP/1.0
User-Agent: Wget/1.11.4 (Red Hat modified)
Accept: */*
Host: www.888.com
X-UserId: 5555552222


-- my response --
ICAP/1.0 200 OK
Encapsulated: req-hdr=0, null-body=131
ISTag: istag
Server: AMS-ICAP-Server-Software/1.0
Content-Length: 131

GET www.nogo.com/redirect.pl?cat=19&a=b HTTP/1.0
Accept: */*
User-Agent: Wget/1.11.4 (Red Hat modified)
X-UserId: 5555552222


Could this be failing due to the missed Host header in the response or
the lack of http:// in the GET portion of the response?


If you have a trace of a working request/response performing url
redirection, I would truly welcome it.

Best regards,

Bob

-----Original Message-----
From: Amos Jeffries [mailto:squid3@...]
Sent: Wednesday, November 04, 2009 1:46 AM
Cc: squid-users@...
Subject: Re: [squid-users] Icap Reqmod

Robert Szabo wrote:

>
> From: Robert Szabo
> Sent: Tuesday, November 03, 2009 10:46 PM
> To: squid-users@...
> Subject: Icap Reqmod
>
>
> Hi all,
>
> I am attempting to use the icap client interface to perform url

redirects.  I have done this using my icap server connecting to several
commercial gateway boxes without issue however, with squid it's not
working as I had hoped.

>
> A typical icap reqmod would appear something like:
>
> REQMOD icap://localhost:8082/ ICAP/1.0
> Host: localhost
> Connection: close
> Encapsulated: req-hdr=0, null-body=117
> X-UserId: 55555555555
>
> GET / HTTP/1.1
> Host: fastclick.net
> Accept: text/html, text/plain
> Accept-Encoding: compress
> Pragma: no-cache
>
> And a blocking or redirect response would look like:
>
> ICAP/1.0 200 OK
> Encapsulated: req-hdr=0, req-body=148
> ISTag: istag
> Server: AMS-ICAP-Server-Software/1.0
>
> GET www.nogo.com/redirect.pl?cat=3&a=b HTTP/1.1
> Accept: text/html, text/plain
> Accept-Encoding: compress
> Pragma: no-cache
> Content-Length: 2
>

Missing either protocol spec http:// or path-spec / at the start of the
URL, and a Host: header....

>
> You will notice the GET and Host:  portions in the reqmod, these are
essential for redirection to work.
>
> However with Squid 3.0 and Squid 3.1, the reqmod does not look right,
it is missing the GET and instead has HEAD in that location and does not
split out the query path from the host :

>
> REQMOD icap://127.0.0.1:8082/reqmod ICAP/1.0
> Host: 127.0.0.1:8082
> Date: Tue, 03 Nov 2009 23:38:31 GMT
> Encapsulated: req-hdr=0, null-body=135
> Allow: 204
>
> HEAD http://www.888.com/ HTTP/1.0
> User-Agent: Wget/1.11.4 (Red Hat modified)
> Accept: */*
> Host: www.888.com
> X-UserId: 5555555555
>
> I have tried responding in the same vane as the message above

replacing the url in the HEAD line, however, the request goes through
fine without considering the redirect.
>
> Can someone provide a sample reqmod and redirect response that would
work?
>
>
> Many Thanks,
>
> Bob

You seem to have generated that test request with wget --spider or -N
options and with wget configured to use a proxy.

That means it generates HEAD requests instead of GET and formats the
request URL suitable for a proxy to handle easily. Squid is simply
passing what it is given to the ICAP server.

What exact releases of squid did you test with?

Amos
--
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
   Current Beta Squid 3.1.0.14
****************************************************************************************
This email and any files transmitted with are confidential and intended solely for the
use of the individual or entity to whom they are addressed.  If you have received this
email in error then please delete it and notify the sender. Do not make a copy or forward
it to anyone.  This footnote also confirms that this email message has been swept for the
presence of computer viruses.

Adaptive Mobile Security Ltd, Dublin Technology Centre, Taylors Lane, Dublin 8,Ireland
Directors: B Collins, L Burke (UK), G. Maclachlan (UK), S. Tirtey (DE).
Registered in Ireland, Company No. 370343, VAT Reg.No.IE63903430
****************************************************************************************

[Amos Jeffries-2]

Robert Szabo wrote:

> Hi Amos,
>
> Many thanks for the quick response.
>
> I am currently using Current Beta Squid 3.1.0.14.
>
> I removed the --spider option from the wget command and this is the
> output
>
> -- reqmod issues by squid --
> REQMOD icap://127.0.0.1:8082/reqmod ICAP/1.0
> Host: 127.0.0.1:8082
> Date: Wed, 04 Nov 2009 13:45:18 GMT
> Encapsulated: req-hdr=0, null-body=134
> Allow: 204
>
> GET http://www.888.com/ HTTP/1.0
> User-Agent: Wget/1.11.4 (Red Hat modified)
> Accept: */*
> Host: www.888.com
> X-UserId: 5555552222
>
>
> -- my response --
> ICAP/1.0 200 OK
> Encapsulated: req-hdr=0, null-body=131
> ISTag: istag
> Server: AMS-ICAP-Server-Software/1.0
> Content-Length: 131
>
> GET www.nogo.com/redirect.pl?cat=19&a=b HTTP/1.0
> Accept: */*
> User-Agent: Wget/1.11.4 (Red Hat modified)
> X-UserId: 5555552222
>
>
> Could this be failing due to the missed Host header in the response or
> the lack of http:// in the GET portion of the response?

Yes. Raw domain name like that is invalid URL.

Squid should be detecting it broken and abort the ICAP response
handling. Dropping back to pass-thru the original request.

>
> If you have a trace of a working request/response performing url
> redirection, I would truly welcome it.

To copy-n-paste your example and add the missing HTTP bits in I expect
this to work:

 > -- reqmod issues by squid --
 > REQMOD icap://127.0.0.1:8082/reqmod ICAP/1.0
 > Host: 127.0.0.1:8082
 > Date: Wed, 04 Nov 2009 13:45:18 GMT
 > Encapsulated: req-hdr=0, null-body=134
 > Allow: 204
 >
 > GET http://www.888.com/ HTTP/1.0
 > User-Agent: Wget/1.11.4 (Red Hat modified)
 > Accept: */*
 > Host: www.888.com
 > X-UserId: 5555552222
 >
 >
 > -- my response --
 > ICAP/1.0 200 OK
 > Encapsulated: req-hdr=0, null-body=131
 > ISTag: istag
 > Server: AMS-ICAP-Server-Software/1.0
 > Content-Length: 131
 >
   GET http://www.nogo.com/redirect.pl?cat=19&a=b HTTP/1.0
   Host: www.nogo.com
 > Accept: */*
 > User-Agent: Wget/1.11.4 (Red Hat modified)
 > X-UserId: 5555552222


Amos
--
Please be using
   Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20
   Current Beta Squid 3.1.0.14