ubuntu apt-get update 404
|
View:
New views
20 Messages
—
Rating Filter:
Alert me
|
 |
ubuntu apt-get update 404
by Matthew Morgan-3
::
Rate this Message:
Are there any known issues with squid 3.x and apt-get update on Ubuntu?
On 2.7 everything worked fine, but on 3.0-stable19 and 3.0-stable20, I get random 404 responses when doing apt-get update. I tried starting with a fresh cache, but no dice. Here's my squid.conf: visible_hostname proxy http_port 192.168.2.1:3128 transparent snmp_port 3401 acl all src all acl manager proto cache_object acl localhost src 127.0.0.1/32 acl to_localhost dst 127.0.0.0/8 acl snmppublic snmp_community public acl localnet src 192.168.1.0/24 # RFC1918 possible internal network acl localnet src 192.168.2.0/24 acl SSL_ports port 443 # https acl SSL_ports port 563 # snews acl SSL_ports port 873 # rsync acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 631 # cups acl Safe_ports port 873 # rsync acl Safe_ports port 901 # SWAT acl purge method PURGE acl CONNECT method CONNECT http_access allow manager localhost http_access deny manager http_access allow purge http_access allow purge localhost http_access deny purge http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localnet http_access allow localhost http_access deny all icp_access allow localnet icp_access deny all http_port 192.168.2.1:3128 hierarchy_stoplist cgi-bin ? cache_dir diskd /usr/local/squid/var/cache2 15000 16 256 maximum_object_size 819200 KB access_log /usr/local/squid/var/logs/access.log squid pid_filename /var/run/squid.pid refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern (Release|Package(.gz)*)$ 0 20% 2880 refresh_pattern . 0 20% 4320 cache_effective_user proxy cache_effective_group proxy client_persistent_connections off server_persistent_connections off snmp_port 3401 snmp_access allow snmppublic localnet snmp_access allow localnet snmp_access allow localhost snmp_incoming_address 0.0.0.0 snmp_outgoing_address 255.255.255.255 hosts_file /etc/hosts forwarded_for off coredump_dir /usr/local/squid/var/cache |
|
|
Re: ubuntu apt-get update 404
by Matthew Morgan-3
::
Rate this Message:
Matthew Morgan wrote:
> Are there any known issues with squid 3.x and apt-get update on > Ubuntu? On 2.7 everything worked fine, but on 3.0-stable19 and > 3.0-stable20, I get random 404 responses when doing apt-get update. I > tried starting with a fresh cache, but no dice. Here's my squid.conf: Quick note: the errors are not always 404's. Sometimes they are like: Err http://us.archive.ubuntu.com jaunty Release.gpg The HTTP server sent an invalid reply header [IP: 91.189.88.45 80] > > visible_hostname proxy > http_port 192.168.2.1:3128 transparent > snmp_port 3401 > acl all src all > acl manager proto cache_object > acl localhost src 127.0.0.1/32 > acl to_localhost dst 127.0.0.0/8 > acl snmppublic snmp_community public > acl localnet src 192.168.1.0/24 # RFC1918 possible internal network > acl localnet src 192.168.2.0/24 > acl SSL_ports port 443 # https > acl SSL_ports port 563 # snews > acl SSL_ports port 873 # rsync > acl Safe_ports port 80 # http > acl Safe_ports port 21 # ftp > acl Safe_ports port 443 # https > acl Safe_ports port 70 # gopher > acl Safe_ports port 210 # wais > acl Safe_ports port 1025-65535 # unregistered ports > acl Safe_ports port 280 # http-mgmt > acl Safe_ports port 488 # gss-http > acl Safe_ports port 591 # filemaker > acl Safe_ports port 777 # multiling http > acl Safe_ports port 631 # cups > acl Safe_ports port 873 # rsync > acl Safe_ports port 901 # SWAT > acl purge method PURGE > acl CONNECT method CONNECT > http_access allow manager localhost > http_access deny manager > http_access allow purge > http_access allow purge localhost > http_access deny purge > http_access deny !Safe_ports > http_access deny CONNECT !SSL_ports > http_access allow localnet > http_access allow localhost > http_access deny all > icp_access allow localnet > icp_access deny all > http_port 192.168.2.1:3128 > hierarchy_stoplist cgi-bin ? > cache_dir diskd /usr/local/squid/var/cache2 15000 16 256 > maximum_object_size 819200 KB > access_log /usr/local/squid/var/logs/access.log squid > pid_filename /var/run/squid.pid > refresh_pattern ^ftp: 1440 20% 10080 > refresh_pattern ^gopher: 1440 0% 1440 > refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 > refresh_pattern (Release|Package(.gz)*)$ 0 20% 2880 > refresh_pattern . 0 20% 4320 > cache_effective_user proxy > cache_effective_group proxy > client_persistent_connections off > server_persistent_connections off > snmp_port 3401 > snmp_access allow snmppublic localnet > snmp_access allow localnet > snmp_access allow localhost > snmp_incoming_address 0.0.0.0 > snmp_outgoing_address 255.255.255.255 > hosts_file /etc/hosts > forwarded_for off > coredump_dir /usr/local/squid/var/cache > |
|
|
Re: ubuntu apt-get update 404
by Amos Jeffries-2
::
Rate this Message:
Matthew Morgan wrote:
> Are there any known issues with squid 3.x and apt-get update on Ubuntu? > On 2.7 everything worked fine, but on 3.0-stable19 and 3.0-stable20, I > get random 404 responses when doing apt-get update. I tried starting > with a fresh cache, but no dice. Here's my squid.conf: > > visible_hostname proxy > http_port 192.168.2.1:3128 transparent > snmp_port 3401 > acl all src all If that is actually in your squid.conf remove it. > acl manager proto cache_object > acl localhost src 127.0.0.1/32 > acl to_localhost dst 127.0.0.0/8 > acl snmppublic snmp_community public > acl localnet src 192.168.1.0/24 # RFC1918 possible internal network > acl localnet src 192.168.2.0/24 > acl CONNECT method CONNECT > http_access allow manager localhost > http_access deny manager > http_access allow purge The above line makes the next two useless... > http_access allow purge localhost > http_access deny purge > http_access deny !Safe_ports > http_access deny CONNECT !SSL_ports > http_access allow localnet > http_access allow localhost > http_access deny all > icp_access allow localnet > icp_access deny all > http_port 192.168.2.1:3128 You have this http_port 192.168.2.1:3128 attempting to open twice. Once with and once without a NAT lookups required. > hierarchy_stoplist cgi-bin ? > cache_dir diskd /usr/local/squid/var/cache2 15000 16 256 Not exactly the fastest disk IO method. On Linux its better to use AUFS. > client_persistent_connections off > server_persistent_connections off Which means squid must open a new TCP link for every single request fetched. This will be slowing things down a fair amount. Amos -- Please be using Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20 Current Beta Squid 3.1.0.14 |
|
|
Re: Re: ubuntu apt-get update 404
by Amos Jeffries-2
::
Rate this Message:
Matthew Morgan wrote:
> Matthew Morgan wrote: >> Are there any known issues with squid 3.x and apt-get update on >> Ubuntu? On 2.7 everything worked fine, but on 3.0-stable19 and >> 3.0-stable20, I get random 404 responses when doing apt-get update. I >> tried starting with a fresh cache, but no dice. Here's my squid.conf: > Quick note: the errors are not always 404's. Sometimes they are like: > > Err http://us.archive.ubuntu.com jaunty Release.gpg > The HTTP server sent an invalid reply header [IP: 91.189.88.45 80] You may be encountering the remains of bug #7. Or some upstream provider with bug #2624 (fixed in 3.0.stable20). If you can track down what that invalid reply header is and whether its coming into Squid from the web server would be a great help. Amos -- Please be using Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20 Current Beta Squid 3.1.0.14 |
|
|
|
|
|
Re: Re: ubuntu apt-get update 404
by Matthew Morgan-3
::
Rate this Message:
On Fri, Nov 6, 2009 at 10:03 PM, Amos Jeffries <squid3@...> wrote:
> Matthew Morgan wrote: >> >> Matthew Morgan wrote: >>> >>> Are there any known issues with squid 3.x and apt-get update on Ubuntu? >>> On 2.7 everything worked fine, but on 3.0-stable19 and 3.0-stable20, I get >>> random 404 responses when doing apt-get update. I tried starting with a >>> fresh cache, but no dice. Here's my squid.conf: >> >> Quick note: the errors are not always 404's. Sometimes they are like: >> >> Err http://us.archive.ubuntu.com jaunty Release.gpg >> The HTTP server sent an invalid reply header [IP: 91.189.88.45 80] > > You may be encountering the remains of bug #7. Or some upstream provider > with bug #2624 (fixed in 3.0.stable20). > > If you can track down what that invalid reply header is and whether its > coming into Squid from the web server would be a great help. I'll fix the other problems with my config that you saw, and if this doesn't go away I'll do some tracking and let you know. Thanks! > > Amos > -- > Please be using > Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20 > Current Beta Squid 3.1.0.14 > |
|
|
Re: Re: ubuntu apt-get update 404
by Matthew Morgan-3
::
Rate this Message:
Amos Jeffries wrote:
> Matthew Morgan wrote: >> Matthew Morgan wrote: >>> Are there any known issues with squid 3.x and apt-get update on >>> Ubuntu? On 2.7 everything worked fine, but on 3.0-stable19 and >>> 3.0-stable20, I get random 404 responses when doing apt-get update. >>> I tried starting with a fresh cache, but no dice. Here's my >>> squid.conf: >> Quick note: the errors are not always 404's. Sometimes they are like: >> >> Err http://us.archive.ubuntu.com jaunty Release.gpg >> The HTTP server sent an invalid reply header [IP: 91.189.88.45 80] > > You may be encountering the remains of bug #7. Or some upstream > provider with bug #2624 (fixed in 3.0.stable20). > > If you can track down what that invalid reply header is and whether > its coming into Squid from the web server would be a great help. > > Amos using 3.0.stable20, only 19. I'm still working on the 404's. I must be doing something wrong somewhere. I have found a few more dumb things in my config (aka, multiple snmp_port lines) but the going is slow. We're really busy here at the shop, and my computer is the only one running ubuntu, so it's not a priority. I'll post back when I get this resolved. |
|
|
Re: Re: ubuntu apt-get update 404
by Matthew Morgan-3
::
Rate this Message:
Matthew Morgan wrote:
> On Fri, Nov 6, 2009 at 10:03 PM, Amos Jeffries <squid3@...> wrote: > >> Matthew Morgan wrote: >> >>> Matthew Morgan wrote: >>> >>>> Are there any known issues with squid 3.x and apt-get update on Ubuntu? >>>> On 2.7 everything worked fine, but on 3.0-stable19 and 3.0-stable20, I get >>>> random 404 responses when doing apt-get update. I tried starting with a >>>> fresh cache, but no dice. Here's my squid.conf: >>>> >>> Quick note: the errors are not always 404's. Sometimes they are like: >>> >>> Err http://us.archive.ubuntu.com jaunty Release.gpg >>> The HTTP server sent an invalid reply header [IP: 91.189.88.45 80] >>> >> You may be encountering the remains of bug #7. Or some upstream provider >> with bug #2624 (fixed in 3.0.stable20). >> >> If you can track down what that invalid reply header is and whether its >> coming into Squid from the web server would be a great help. >> > > I'll fix the other problems with my config that you saw, and if this > doesn't go away I'll do some tracking and let you know. Thanks! > > >> Amos >> -- >> Please be using >> Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20 >> Current Beta Squid 3.1.0.14 >> >> > > some of the urls. With squid-2.7 (which works), there are many requests like the following: http://us.archive.ubuntu.com/ubuntu/dists/jaunty-security/main/binary-i386/Packages.bz2 With 3.x, a few of them look like this: http://us.archive.ubuntu.com/ubuntu/dists/jaunty-security/main/binary-i386/Packages They're identical, but somewhere the file extension is getting ripped off...but only on some of them. Do you know of a way to find out where this is happening? I don't exactly grok squid yet, so I don't think I'm equipped to divine the answer from the source code. |
|
|
Re: Re: ubuntu apt-get update 404
by Matthew Morgan-3
::
Rate this Message:
Matthew Morgan wrote:
> Matthew Morgan wrote: >> On Fri, Nov 6, 2009 at 10:03 PM, Amos Jeffries <squid3@...> >> wrote: >> >>> Matthew Morgan wrote: >>> >>>> Matthew Morgan wrote: >>>> >>>>> Are there any known issues with squid 3.x and apt-get update on >>>>> Ubuntu? >>>>> On 2.7 everything worked fine, but on 3.0-stable19 and >>>>> 3.0-stable20, I get >>>>> random 404 responses when doing apt-get update. I tried starting >>>>> with a >>>>> fresh cache, but no dice. Here's my squid.conf: >>>>> >>>> Quick note: the errors are not always 404's. Sometimes they are like: >>>> >>>> Err http://us.archive.ubuntu.com jaunty Release.gpg >>>> The HTTP server sent an invalid reply header [IP: 91.189.88.45 80] >>>> >>> You may be encountering the remains of bug #7. Or some upstream >>> provider >>> with bug #2624 (fixed in 3.0.stable20). >>> >>> If you can track down what that invalid reply header is and whether its >>> coming into Squid from the web server would be a great help. >>> >> >> I'll fix the other problems with my config that you saw, and if this >> doesn't go away I'll do some tracking and let you know. Thanks! >> >> >>> Amos >>> -- >>> Please be using >>> Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20 >>> Current Beta Squid 3.1.0.14 >>> >>> >> >> > It looks like for some reason the .bz2 extensions is getting dropped > off some of the urls. With squid-2.7 (which works), there are many > requests like the following: > > http://us.archive.ubuntu.com/ubuntu/dists/jaunty-security/main/binary-i386/Packages.bz2 > > > With 3.x, a few of them look like this: > > http://us.archive.ubuntu.com/ubuntu/dists/jaunty-security/main/binary-i386/Packages > > > They're identical, but somewhere the file extension is getting ripped > off...but only on some of them. Do you know of a way to find out > where this is happening? I don't exactly grok squid yet, so I don't > think I'm equipped to divine the answer from the source code. > transparently, which is how our network is set up. If I manually specify http_proxy on my workstation to point to squid directly, I don't have any problems with apt-get update. Has anyone ever heard of this? Here's my updated squid config (this is 3.0-STABLE20, btw). visible_hostname proxy http_port 192.168.2.1:3128 transparent acl manager proto cache_object acl localhost src 127.0.0.1/32 acl to_localhost dst 127.0.0.0/8 acl snmppublic snmp_community public acl localnet src 192.168.1.0/24 # RFC1918 possible internal network acl localnet src 192.168.2.0/24 acl SSL_ports port 443 # https acl SSL_ports port 563 # snews acl SSL_ports port 873 # rsync acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl Safe_ports port 631 # cups acl Safe_ports port 873 # rsync acl Safe_ports port 901 # SWAT acl purge method PURGE acl CONNECT method CONNECT http_access allow manager localnet http_access allow manager localhost http_access deny manager http_access allow purge localnet http_access allow purge localhost http_access deny purge http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localnet http_access allow localhost http_access deny all icp_access allow localnet icp_access deny all hierarchy_stoplist cgi-bin ? cache_dir diskd /usr/local/squid/var/cache 15000 16 256 maximum_object_size 819200 KB access_log /usr/local/squid/var/logs/access.log squid pid_filename /var/run/squid.pid refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims override-expire refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims override-expire refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 reload-into-ims override-expire refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 reload-into-ims override-expire refresh_pattern updates.superantispyware.com/sas_processlist.* 1440 100% 1441 ignore-reload override-lastmod override-expire refresh_pattern http://mbam-cdn.malwarebytes.org/.* 1440 100% 1441 ignore-reload override-lastmod override-expire refresh_pattern http://download682.avast.com/.* 1440 100% 1441 ignore-reload override-lastmod override-expire refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 cache_effective_user proxy cache_effective_group proxy snmp_port 3401 snmp_access allow snmppublic localnet snmp_access allow localnet snmp_access allow localhost snmp_incoming_address 0.0.0.0 snmp_outgoing_address 255.255.255.255 hosts_file /etc/hosts coredump_dir /usr/local/squid/var/cache |
|
|
RE: Re: ubuntu apt-get update 404
by Michael Bowe
::
Rate this Message:
> -----Original Message-----
> From: Matthew Morgan [mailto:atcs.matthew@...] > Sent: Saturday, 14 November 2009 7:59 AM > To: Squid Users > Subject: Re: [squid-users] Re: ubuntu apt-get update 404 > > Apparently I only get the dropped .bz2 extensions when using squid > transparently, which is how our network is set up. If I manually > specify http_proxy on my workstation to point to squid directly, I > don't > have any problems with apt-get update. Has anyone ever heard of this? > Here's my updated squid config (this is 3.0-STABLE20, btw). I've been having perhaps related problems with Debian servers behind Squid 3.1.0.14 TPROXY I am not getting 404's but am intermittently seeing "invalid reply header" errors. eg : Failed to fetch http://backports.org/debian/dists/etch-backports/main/binary-amd64/Packages. gz The HTTP server sent an invalid reply header Err http://security.debian.org lenny/updates Release.gpg The HTTP server sent an invalid reply header [IP: 150.203.164.38 80] W: Failed to fetch http://security.debian.org/dists/lenny/updates/Release.gpg The HTTP server sent an invalid reply header [IP: 150.203.164.38 80] As you say, if I specify HTTP_PROXY= to go direct to the cache rather than transparent then all works fine Michael. |
|
|
Re: Re: ubuntu apt-get update 404
by Amos Jeffries-2
::
Rate this Message:
Michael Bowe wrote:
>> -----Original Message----- >> From: Matthew Morgan [mailto:atcs.matthew@...] >> Sent: Saturday, 14 November 2009 7:59 AM >> To: Squid Users >> Subject: Re: [squid-users] Re: ubuntu apt-get update 404 >> > >> Apparently I only get the dropped .bz2 extensions when using squid >> transparently, which is how our network is set up. If I manually >> specify http_proxy on my workstation to point to squid directly, I >> don't >> have any problems with apt-get update. Has anyone ever heard of this? >> Here's my updated squid config (this is 3.0-STABLE20, btw). > > I've been having perhaps related problems with Debian servers behind Squid > 3.1.0.14 TPROXY > > I am not getting 404's but am intermittently seeing "invalid reply header" > errors. eg : > > Failed to fetch > http://backports.org/debian/dists/etch-backports/main/binary-amd64/Packages. > gz The HTTP server sent an invalid reply header > > Err http://security.debian.org lenny/updates Release.gpg > The HTTP server sent an invalid reply header [IP: 150.203.164.38 80] > > W: Failed to fetch > http://security.debian.org/dists/lenny/updates/Release.gpg The HTTP server > sent an invalid reply header [IP: 150.203.164.38 80] > > As you say, if I specify HTTP_PROXY= to go direct to the cache rather than > transparent then all works fine > > Michael. > I wonder. Is that actually 3.1.0.14 direct to origin? or perhapse going through some older sub-cache? Are the two of you able to provide me with "tcpdump -s0" traces of the data between apt and squid please? particularly for the transparent mode problems. Amos -- Please be using Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20 Current Beta Squid 3.1.0.14 |
|
|
Re: Re: ubuntu apt-get update 404
by Matthew Morgan-3
::
Rate this Message:
Amos Jeffries wrote:
> Michael Bowe wrote: >>> -----Original Message----- >>> From: Matthew Morgan [mailto:atcs.matthew@...] >>> Sent: Saturday, 14 November 2009 7:59 AM >>> To: Squid Users >>> Subject: Re: [squid-users] Re: ubuntu apt-get update 404 >>> >> >>> Apparently I only get the dropped .bz2 extensions when using squid >>> transparently, which is how our network is set up. If I manually >>> specify http_proxy on my workstation to point to squid directly, I >>> don't >>> have any problems with apt-get update. Has anyone ever heard of this? >>> Here's my updated squid config (this is 3.0-STABLE20, btw). >> >> I've been having perhaps related problems with Debian servers behind >> Squid >> 3.1.0.14 TPROXY >> >> I am not getting 404's but am intermittently seeing "invalid reply >> header" >> errors. eg : >> >> Failed to fetch >> http://backports.org/debian/dists/etch-backports/main/binary-amd64/Packages. >> >> gz The HTTP server sent an invalid reply header >> >> Err http://security.debian.org lenny/updates Release.gpg >> The HTTP server sent an invalid reply header [IP: 150.203.164.38 80] >> >> W: Failed to fetch >> http://security.debian.org/dists/lenny/updates/Release.gpg The HTTP >> server >> sent an invalid reply header [IP: 150.203.164.38 80] >> >> As you say, if I specify HTTP_PROXY= to go direct to the cache rather >> than >> transparent then all works fine >> >> Michael. >> > > I wonder. Is that actually 3.1.0.14 direct to origin? or perhapse > going through some older sub-cache? >> >> Are the two of you able to provide me with "tcpdump -s0" traces of >> the data between apt and squid please? particularly for the >> transparent mode problems. >> >> Amos I will get you that as soon as I can steal a few minutes. Thanks so much for the help! >> I will get you that as soon as I can steal a few minutes. Thanks so >> much for the h |
|
|
Re: Re: ubuntu apt-get update 404
by Matthew Morgan-3
::
Rate this Message:
Amos Jeffries wrote:
> Michael Bowe wrote: >>> -----Original Message----- >>> From: Matthew Morgan [mailto:atcs.matthew@...] >>> Sent: Saturday, 14 November 2009 7:59 AM >>> To: Squid Users >>> Subject: Re: [squid-users] Re: ubuntu apt-get update 404 >>> >> >>> Apparently I only get the dropped .bz2 extensions when using squid >>> transparently, which is how our network is set up. If I manually >>> specify http_proxy on my workstation to point to squid directly, I >>> don't >>> have any problems with apt-get update. Has anyone ever heard of this? >>> Here's my updated squid config (this is 3.0-STABLE20, btw). >> >> I've been having perhaps related problems with Debian servers behind >> Squid >> 3.1.0.14 TPROXY >> >> I am not getting 404's but am intermittently seeing "invalid reply >> header" >> errors. eg : >> >> Failed to fetch >> http://backports.org/debian/dists/etch-backports/main/binary-amd64/Packages. >> >> gz The HTTP server sent an invalid reply header >> >> Err http://security.debian.org lenny/updates Release.gpg >> The HTTP server sent an invalid reply header [IP: 150.203.164.38 80] >> >> W: Failed to fetch >> http://security.debian.org/dists/lenny/updates/Release.gpg The HTTP >> server >> sent an invalid reply header [IP: 150.203.164.38 80] >> >> As you say, if I specify HTTP_PROXY= to go direct to the cache rather >> than >> transparent then all works fine >> >> Michael. >> > > I wonder. Is that actually 3.1.0.14 direct to origin? or perhapse > going through some older sub-cache? > > Are the two of you able to provide me with "tcpdump -s0" traces of the > data between apt and squid please? particularly for the transparent > mode problems. > > Amos I will get you that as soon as I can steal a few minutes. Thanks so much for the help! |
|
|
RE: Re: ubuntu apt-get update 404
by Michael Bowe
::
Rate this Message:
> -----Original Message-----
> From: Amos Jeffries [mailto:squid3@...] > I wonder. Is that actually 3.1.0.14 direct to origin? or perhapse going > through some older sub-cache? I see this at several of our sites Each site is a cluster of 3 or 4 v3.1.0.14 sibling-caches. No parent caches. > Are the two of you able to provide me with "tcpdump -s0" traces of the > data between apt and squid please? particularly for the transparent > mode > problems. Yes I will try and capture this for you. Michael. |
|
|
Re: Re: ubuntu apt-get update 404
by Matthew Morgan-3
::
Rate this Message:
Amos Jeffries wrote:
> Michael Bowe wrote: >>> -----Original Message----- >>> From: Matthew Morgan [mailto:atcs.matthew@...] >>> Sent: Saturday, 14 November 2009 7:59 AM >>> To: Squid Users >>> Subject: Re: [squid-users] Re: ubuntu apt-get update 404 >>> >> >>> Apparently I only get the dropped .bz2 extensions when using squid >>> transparently, which is how our network is set up. If I manually >>> specify http_proxy on my workstation to point to squid directly, I >>> don't >>> have any problems with apt-get update. Has anyone ever heard of this? >>> Here's my updated squid config (this is 3.0-STABLE20, btw). >> >> I've been having perhaps related problems with Debian servers behind >> Squid >> 3.1.0.14 TPROXY >> >> I am not getting 404's but am intermittently seeing "invalid reply >> header" >> errors. eg : >> >> Failed to fetch >> http://backports.org/debian/dists/etch-backports/main/binary-amd64/Packages. >> >> gz The HTTP server sent an invalid reply header >> >> Err http://security.debian.org lenny/updates Release.gpg >> The HTTP server sent an invalid reply header [IP: 150.203.164.38 80] >> >> W: Failed to fetch >> http://security.debian.org/dists/lenny/updates/Release.gpg The HTTP >> server >> sent an invalid reply header [IP: 150.203.164.38 80] >> >> As you say, if I specify HTTP_PROXY= to go direct to the cache rather >> than >> transparent then all works fine >> >> Michael. >> > > I wonder. Is that actually 3.1.0.14 direct to origin? or perhapse > going through some older sub-cache? > > Are the two of you able to provide me with "tcpdump -s0" traces of the > data between apt and squid please? particularly for the transparent > mode problems. > > Amos after switching to 3.x, it's hit or miss. Sometimes it's perfect, sometimes I get errors. After that, I get errors in two stages. Here's what happens: Either: apt-get update #1 - no errors apt-get update #2 - invalid header, and sometimes 404 errors apt-get update #3 and above - 404 errors only or: apt-get update #1 - invalid header, and sometimes 404 errors apt-get update #2 and above - 404 errors only The dump files I have uploaded match the second set of circumstances. server1.dump and client1.dump are from the first apt-get update after switching, and I got an invalid header error + 404 errors. server2.dump and client2.dump came from the second apt-get update attempt, and only 404 errors were returned. I hope this helps! Let me know if you need anything else. Just a reminder, on my setup I only have 1 squid server with 1 cache directory. For comparison, my server is Ubuntu 9.04 running kernel 2.6.28-16-server. I am not using TPROXY. Here are the files (I tried to attach them, but mailer-daemon kicked the email) http://lithagen.dyndns.org/server1.dump http://lithagen.dyndns.org/client1.dump http://lithagen.dyndns.org/server2.dump http://lithagen.dyndns.org/client2.dump |
|
|
Re: Re: ubuntu apt-get update 404
by Amos Jeffries-2
::
Rate this Message:
Matthew Morgan wrote:
> Amos Jeffries wrote: >> Michael Bowe wrote: >>>> -----Original Message----- >>>> From: Matthew Morgan [mailto:atcs.matthew@...] >>>> Sent: Saturday, 14 November 2009 7:59 AM >>>> To: Squid Users >>>> Subject: Re: [squid-users] Re: ubuntu apt-get update 404 >>>> >>> >>>> Apparently I only get the dropped .bz2 extensions when using squid >>>> transparently, which is how our network is set up. If I manually >>>> specify http_proxy on my workstation to point to squid directly, I >>>> don't >>>> have any problems with apt-get update. Has anyone ever heard of this? >>>> Here's my updated squid config (this is 3.0-STABLE20, btw). >>> >>> I've been having perhaps related problems with Debian servers behind >>> Squid >>> 3.1.0.14 TPROXY >>> >>> I am not getting 404's but am intermittently seeing "invalid reply >>> header" >>> errors. eg : >>> >>> Failed to fetch >>> http://backports.org/debian/dists/etch-backports/main/binary-amd64/Packages. >>> >>> gz The HTTP server sent an invalid reply header >>> >>> Err http://security.debian.org lenny/updates Release.gpg >>> The HTTP server sent an invalid reply header [IP: 150.203.164.38 80] >>> >>> W: Failed to fetch >>> http://security.debian.org/dists/lenny/updates/Release.gpg The HTTP >>> server >>> sent an invalid reply header [IP: 150.203.164.38 80] >>> >>> As you say, if I specify HTTP_PROXY= to go direct to the cache rather >>> than >>> transparent then all works fine >>> >>> Michael. >>> >> >> I wonder. Is that actually 3.1.0.14 direct to origin? or perhapse >> going through some older sub-cache? >> >> Are the two of you able to provide me with "tcpdump -s0" traces of the >> data between apt and squid please? particularly for the transparent >> mode problems. >> >> Amos > Ok, it seems to happen in stages. The first time I run apt-get update > after switching to 3.x, it's hit or miss. Sometimes it's perfect, > sometimes I get errors. After that, I get errors in two stages. Here's > what happens: > > > Either: > > apt-get update #1 - no errors > apt-get update #2 - invalid header, and sometimes 404 errors > apt-get update #3 and above - 404 errors only > > or: > > apt-get update #1 - invalid header, and sometimes 404 errors > apt-get update #2 and above - 404 errors only > > The dump files I have uploaded match the second set of circumstances. > server1.dump and client1.dump are from the first apt-get update after > switching, and I got an invalid header error + 404 errors. server2.dump > and client2.dump came from the second apt-get update attempt, and only > 404 errors were returned. > > I hope this helps! Let me know if you need anything else. Just a > reminder, on my setup I only have 1 squid server with 1 cache > directory. For comparison, my server is Ubuntu 9.04 running kernel > 2.6.28-16-server. I am not using TPROXY. > > Here are the files (I tried to attach them, but mailer-daemon kicked the > email) > > http://lithagen.dyndns.org/server1.dump > http://lithagen.dyndns.org/client1.dump > http://lithagen.dyndns.org/server2.dump > http://lithagen.dyndns.org/client2.dump Well, good news and sad news. Both traces show the same problems. The 404 is actually being generated by the us.archive.ubuntu.com server itself. There is something broken at the mirror or in apts local sources.list URLs. Squid-3.0 still has the deprecated default for caching of 404/5xx results for 5 minutes. You may get less of those errors and other temporary errors by adding this to your squid.conf: negative_ttl 0 seconds The invalid header problem appears to be a minor issue (should be no bad effect from it) caused by Squid sending back a Proxy-Connection: header to apt. That is meant to be Connection: on intercepted requests. Now fixed for the next release. Thank you. Amos -- Please be using Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20 Current Beta Squid 3.1.0.14 |
|
|
Re: Re: ubuntu apt-get update 404
by Matthew Morgan-3
::
Rate this Message:
Amos Jeffries wrote:
> Matthew Morgan wrote: >> Amos Jeffries wrote: >>> Michael Bowe wrote: >>>>> -----Original Message----- >>>>> From: Matthew Morgan [mailto:atcs.matthew@...] >>>>> Sent: Saturday, 14 November 2009 7:59 AM >>>>> To: Squid Users >>>>> Subject: Re: [squid-users] Re: ubuntu apt-get update 404 >>>>> >>>> >>>>> Apparently I only get the dropped .bz2 extensions when using squid >>>>> transparently, which is how our network is set up. If I manually >>>>> specify http_proxy on my workstation to point to squid directly, I >>>>> don't >>>>> have any problems with apt-get update. Has anyone ever heard of >>>>> this? >>>>> Here's my updated squid config (this is 3.0-STABLE20, btw). >>>> >>>> I've been having perhaps related problems with Debian servers >>>> behind Squid >>>> 3.1.0.14 TPROXY >>>> >>>> I am not getting 404's but am intermittently seeing "invalid reply >>>> header" >>>> errors. eg : >>>> >>>> Failed to fetch >>>> http://backports.org/debian/dists/etch-backports/main/binary-amd64/Packages. >>>> >>>> gz The HTTP server sent an invalid reply header >>>> >>>> Err http://security.debian.org lenny/updates Release.gpg >>>> The HTTP server sent an invalid reply header [IP: 150.203.164.38 80] >>>> >>>> W: Failed to fetch >>>> http://security.debian.org/dists/lenny/updates/Release.gpg The >>>> HTTP server >>>> sent an invalid reply header [IP: 150.203.164.38 80] >>>> >>>> As you say, if I specify HTTP_PROXY= to go direct to the cache >>>> rather than >>>> transparent then all works fine >>>> >>>> Michael. >>>> >>> >>> I wonder. Is that actually 3.1.0.14 direct to origin? or perhapse >>> going through some older sub-cache? >>> >>> Are the two of you able to provide me with "tcpdump -s0" traces of >>> the data between apt and squid please? particularly for the >>> transparent mode problems. >>> >>> Amos >> Ok, it seems to happen in stages. The first time I run apt-get >> update after switching to 3.x, it's hit or miss. Sometimes it's >> perfect, sometimes I get errors. After that, I get errors in two >> stages. Here's what happens: >> >> >> Either: >> >> apt-get update #1 - no errors >> apt-get update #2 - invalid header, and sometimes 404 errors >> apt-get update #3 and above - 404 errors only >> >> or: >> >> apt-get update #1 - invalid header, and sometimes 404 errors >> apt-get update #2 and above - 404 errors only >> >> The dump files I have uploaded match the second set of >> circumstances. server1.dump and client1.dump are from the first >> apt-get update after switching, and I got an invalid header error + >> 404 errors. server2.dump and client2.dump came from the second >> apt-get update attempt, and only 404 errors were returned. >> >> I hope this helps! Let me know if you need anything else. Just a >> reminder, on my setup I only have 1 squid server with 1 cache >> directory. For comparison, my server is Ubuntu 9.04 running kernel >> 2.6.28-16-server. I am not using TPROXY. >> >> Here are the files (I tried to attach them, but mailer-daemon kicked >> the email) >> >> http://lithagen.dyndns.org/server1.dump >> http://lithagen.dyndns.org/client1.dump >> http://lithagen.dyndns.org/server2.dump >> http://lithagen.dyndns.org/client2.dump > > Well, good news and sad news. > > Both traces show the same problems. > > The 404 is actually being generated by the us.archive.ubuntu.com > server itself. There is something broken at the mirror or in apts > local sources.list URLs. Everything works fine with the exact same sources.list when using squid 2.7, so there shouldn't be anything wrong with the file. us.archive.ubuntu.com must be treating squid 3.x different somehow, right? > > Squid-3.0 still has the deprecated default for caching of 404/5xx > results for 5 minutes. You may get less of those errors and other > temporary errors by adding this to your squid.conf: > negative_ttl 0 seconds > > > The invalid header problem appears to be a minor issue (should be no > bad effect from it) caused by Squid sending back a Proxy-Connection: > header to apt. That is meant to be Connection: on intercepted requests. > Now fixed for the next release. Thank you. > > Amos |
|
|
Re: Re: ubuntu apt-get update 404
by Amos Jeffries-2
::
Rate this Message:
Matthew Morgan wrote:
> Amos Jeffries wrote: >> Matthew Morgan wrote: <snip> >>> Ok, it seems to happen in stages. The first time I run apt-get >>> update after switching to 3.x, it's hit or miss. Sometimes it's >>> perfect, sometimes I get errors. After that, I get errors in two >>> stages. Here's what happens: >>> >>> >>> Either: >>> >>> apt-get update #1 - no errors >>> apt-get update #2 - invalid header, and sometimes 404 errors >>> apt-get update #3 and above - 404 errors only >>> >>> or: >>> >>> apt-get update #1 - invalid header, and sometimes 404 errors >>> apt-get update #2 and above - 404 errors only >>> >>> The dump files I have uploaded match the second set of >>> circumstances. server1.dump and client1.dump are from the first >>> apt-get update after switching, and I got an invalid header error + >>> 404 errors. server2.dump and client2.dump came from the second >>> apt-get update attempt, and only 404 errors were returned. >>> >>> I hope this helps! Let me know if you need anything else. Just a >>> reminder, on my setup I only have 1 squid server with 1 cache >>> directory. For comparison, my server is Ubuntu 9.04 running kernel >>> 2.6.28-16-server. I am not using TPROXY. >>> >>> Here are the files (I tried to attach them, but mailer-daemon kicked >>> the email) >>> >>> http://lithagen.dyndns.org/server1.dump >>> http://lithagen.dyndns.org/client1.dump >>> http://lithagen.dyndns.org/server2.dump >>> http://lithagen.dyndns.org/client2.dump >> >> Well, good news and sad news. >> >> Both traces show the same problems. >> >> The 404 is actually being generated by the us.archive.ubuntu.com >> server itself. There is something broken at the mirror or in apts >> local sources.list URLs. > So does squid 3.x have a different user agent string or something? No. > Everything works fine with the exact same sources.list when using squid > 2.7, so there shouldn't be anything wrong with the file. > us.archive.ubuntu.com must be treating squid 3.x different somehow, right? It does seem to be. Why is the big question. Amos -- Please be using Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20 Current Beta Squid 3.1.0.14 |
|
|
Re: Re: ubuntu apt-get update 404
by Matthew Morgan-3
::
Rate this Message:
Amos Jeffries wrote:
> Matthew Morgan wrote: >> Amos Jeffries wrote: >>> Matthew Morgan wrote: > <snip> >>>> Ok, it seems to happen in stages. The first time I run apt-get >>>> update after switching to 3.x, it's hit or miss. Sometimes it's >>>> perfect, sometimes I get errors. After that, I get errors in two >>>> stages. Here's what happens: >>>> >>>> >>>> Either: >>>> >>>> apt-get update #1 - no errors >>>> apt-get update #2 - invalid header, and sometimes 404 errors >>>> apt-get update #3 and above - 404 errors only >>>> >>>> or: >>>> >>>> apt-get update #1 - invalid header, and sometimes 404 errors >>>> apt-get update #2 and above - 404 errors only >>>> >>>> The dump files I have uploaded match the second set of >>>> circumstances. server1.dump and client1.dump are from the first >>>> apt-get update after switching, and I got an invalid header error + >>>> 404 errors. server2.dump and client2.dump came from the second >>>> apt-get update attempt, and only 404 errors were returned. >>>> >>>> I hope this helps! Let me know if you need anything else. Just a >>>> reminder, on my setup I only have 1 squid server with 1 cache >>>> directory. For comparison, my server is Ubuntu 9.04 running kernel >>>> 2.6.28-16-server. I am not using TPROXY. >>>> >>>> Here are the files (I tried to attach them, but mailer-daemon >>>> kicked the email) >>>> >>>> http://lithagen.dyndns.org/server1.dump >>>> http://lithagen.dyndns.org/client1.dump >>>> http://lithagen.dyndns.org/server2.dump >>>> http://lithagen.dyndns.org/client2.dump >>> >>> Well, good news and sad news. >>> >>> Both traces show the same problems. >>> >>> The 404 is actually being generated by the us.archive.ubuntu.com >>> server itself. There is something broken at the mirror or in apts >>> local sources.list URLs. >> So does squid 3.x have a different user agent string or something? > > No. > >> Everything works fine with the exact same sources.list when using >> squid 2.7, so there shouldn't be anything wrong with the file. >> us.archive.ubuntu.com must be treating squid 3.x different somehow, >> right? > > It does seem to be. Why is the big question. > > > Amos compare what headers and such are being sent from an otherwise identical setup? |
|
|
Re: Re: ubuntu apt-get update 404
by Amos Jeffries-2
::
Rate this Message:
Matthew Morgan wrote:
> Amos Jeffries wrote: >> Matthew Morgan wrote: >>> Amos Jeffries wrote: >>>> Matthew Morgan wrote: >> <snip> >>>>> Ok, it seems to happen in stages. The first time I run apt-get >>>>> update after switching to 3.x, it's hit or miss. Sometimes it's >>>>> perfect, sometimes I get errors. After that, I get errors in two >>>>> stages. Here's what happens: >>>>> >>>>> >>>>> Either: >>>>> >>>>> apt-get update #1 - no errors >>>>> apt-get update #2 - invalid header, and sometimes 404 errors >>>>> apt-get update #3 and above - 404 errors only >>>>> >>>>> or: >>>>> >>>>> apt-get update #1 - invalid header, and sometimes 404 errors >>>>> apt-get update #2 and above - 404 errors only >>>>> >>>>> The dump files I have uploaded match the second set of >>>>> circumstances. server1.dump and client1.dump are from the first >>>>> apt-get update after switching, and I got an invalid header error + >>>>> 404 errors. server2.dump and client2.dump came from the second >>>>> apt-get update attempt, and only 404 errors were returned. >>>>> >>>>> I hope this helps! Let me know if you need anything else. Just a >>>>> reminder, on my setup I only have 1 squid server with 1 cache >>>>> directory. For comparison, my server is Ubuntu 9.04 running kernel >>>>> 2.6.28-16-server. I am not using TPROXY. >>>>> >>>>> Here are the files (I tried to attach them, but mailer-daemon >>>>> kicked the email) >>>>> >>>>> http://lithagen.dyndns.org/server1.dump >>>>> http://lithagen.dyndns.org/client1.dump >>>>> http://lithagen.dyndns.org/server2.dump >>>>> http://lithagen.dyndns.org/client2.dump >>>> >>>> Well, good news and sad news. >>>> >>>> Both traces show the same problems. >>>> >>>> The 404 is actually being generated by the us.archive.ubuntu.com >>>> server itself. There is something broken at the mirror or in apts >>>> local sources.list URLs. >>> So does squid 3.x have a different user agent string or something? >> >> No. >> >>> Everything works fine with the exact same sources.list when using >>> squid 2.7, so there shouldn't be anything wrong with the file. >>> us.archive.ubuntu.com must be treating squid 3.x different somehow, >>> right? >> >> It does seem to be. Why is the big question. >> >> >> Amos > Should I send you a capture of my working 2.7 installation so you can > compare what headers and such are being sent from an otherwise identical > setup? > I've just ported that header fix down to 3.1. Try tomorrows snapshot and see if the header change fixes the issue at all. If the problem remains, then yes a copy of the 2.7 transactions would be useful to compare. Amos -- Please be using Current Stable Squid 2.7.STABLE7 or 3.0.STABLE20 Current Beta Squid 3.1.0.14 |
| Free embeddable forum powered by Nabble | Forum Help |
