Server hangs after 502 request

[Rolf Schaufelberger]

Hi,

I have a server running,  configured with a lightweiht frontend apache  
and a backend mod_perl apache.
Starting some weeks ago the server sometimes hangs with an out of  
memory problem.
(Kernel panic, no killable process found )
I currently have no idea where that problems comes from, the numer of  
backend apache is set to 15, so this should not be a problem.
Yet, what I'have found, that the last request   logged in the frontend  
aache log looks like :

"GET / HTTP/1.1" 502 465 "http://googleads.g.doubleclick.net/pagead/ads?client=ca-pub-8011132902902466&format=fp_al_lp&output=html&h=15&w=728&lmt=1256317728&channel=0471980043%2B7740394182&ea=0&flash=10.0.32.18&url=http%3A%2F%2Fwww.schulferien.org%2FBaden-Wurttemb_%2Fbaden-wurttemb_.html&dt=1256317728847&correlator=1256317728850&frm=0&ga_vid=795893907.1228664705&ga_sid=1256317729&ga_hid=250028746&ga_fc=1&u_tz=120&u_his=13&u_java=1&u_h=800&u_w=1280&u_ah=770&u_aw=1280&u_cd=32&u_nplug=0&u_nmime=0&biw=1276&bih=569&ref=http%3A%2F%2Fwww.google.de%2Fsearch%3Fhl%3Dde%26rlz%3D1T4SNYK_deDE295DE295%26q%3Dferien%2Bbaden%2Bw%25C3%25BCrttemberg%2B2010%26meta%3D%26aq%3D2%26oq%3Dferien%2Bbaden%2B&fu=0&ifi=1&dtd=271&kw_type=radlink&prev_fmts=728x15_0ads_al_s&rt=ChBK4eMkAAKqhwrMjcLbYhQ7Eg5KYWhyZXNrYWxlbmRlchoIyMwiMKeTu80oAVITCNvXmaHU050CFVGIzAodgFzQag&hl=de&kw0=Ferienkalender+NRW&kw1=Schulferien+2010&kw2=Jahreskalender&kw3=Kalender+Hessen+2008&kw4=Kalender+Schulferien&okw=Jahreskalender 
" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0;  
GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR  
1.1.4322; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"

It always has googleads as referer with a very long refer string and  
what das the 502 status mean?

Does anybody have some idea what is happening ?
Can I simulate this request with LWP ?


Rolf Schaufelberger

[awarnier]

Rolf Schaufelberger wrote:
>
> It always has googleads as referer with a very long refer string and
> what das the 502 status mean?
>
http://lmgtfy.com?q=http+status+502

[Rolf Schaufelberger]

Am 24.10.2009 um 12:13 schrieb André Warnier:

> Rolf Schaufelberger wrote:
>> It always has googleads as referer with a very long refer string  
>> and what das the 502 status mean?
> http://lmgtfy.com?q=http+status+502


Ok, I can read what's written there, but what does it mean ?
  I still have no idea, what leads to that error. As far as I  
understood this, my backend server send something else but no  HTTP  
message to the frontend server.
But  why?

Rolf Schaufelberger

[awarnier]

Rolf Schaufelberger wrote:

>
> Am 24.10.2009 um 12:13 schrieb André Warnier:
>
>> Rolf Schaufelberger wrote:
>>> It always has googleads as referer with a very long refer string and
>>> what das the 502 status mean?
>> http://lmgtfy.com?q=http+status+502
>
>
> Ok, I can read what's written there, but what does it mean ?
>  I still have no idea, what leads to that error. As far as I understood
> this, my backend server send something else but no  HTTP message to the
> frontend server.
> But  why?
>

Rolf,

502 Bad Gateway
     The server was acting as a gateway or proxy and received an invalid
response from the downstream server.

It means that your front-end server received this request ("GET /"),
from some client, which apparently was at the time displaying a page
which itself had been obtained from the long link that you see in the
message. (It does not mean that the request itself came from a Google
server; it just means that on some page of Google somewhere, there is a
link that points to your site, and that someone clicked on it).

Now, we don't know what the configuration of your front-end and back-end
servers are, nor what is exactly proxied to your back-end server when
the front-end server receives a request for "/".
But the error code 502 that is in the log (see description above) seems
to indicate that your front-end server passed this request somehow to a
back-end server, and that it received an invalid response from the
back-end server.
Why that is, is also not known to us, and quite impossible to figure out
with the information you provided.

If I had to make a guess, I would say that the problem has nothing to do
with this request in particular, and that your front-end server is just
configured with too many children or threads, compared to the physical
memory it really has.  So when many requests come in and it tries to
fork more children/threads to handle them, it runs out of memory.
Reducing MaxClients may be a first option.

About reproducing this access with LWP, sure :
lwp-request -m GET http://your-server-name/
but that is not going to tell you much.

[Rolf Schaufelberger]

Am 25.10.2009 um 19:36 schrieb André Warnier:

> Rolf Schaufelberger wrote:
>> Am 24.10.2009 um 12:13 schrieb André Warnier:
>>> Rolf Schaufelberger wrote:
>>>> It always has googleads as referer with a very long refer string  
>>>> and what das the 502 status mean?
>>> http://lmgtfy.com?q=http+status+502
>> Ok, I can read what's written there, but what does it mean ?
>> I still have no idea, what leads to that error. As far as I  
>> understood this, my backend server send something else but no  HTTP  
>> message to the frontend server.
>> But  why?
> Rolf,
>
> 502 Bad Gateway
>    The server was acting as a gateway or proxy and received an  
> invalid response from the downstream server.
>
> It means that your front-end server received this request ("GET /"),  
> from some client, which apparently was at the time displaying a page  
> which itself had been obtained from the long link that you see in  
> the message. (It does not mean that the request itself came from a  
> Google server; it just means that on some page of Google somewhere,  
> there is a link that points to your site, and that someone clicked  
> on it).
>
> Now, we don't know what the configuration of your front-end and back-
> end servers are, nor what is exactly proxied to your back-end server  
> when the front-end server receives a request for "/".
> But the error code 502 that is in the log (see description above)  
> seems to indicate that your front-end server passed this request  
> somehow to a back-end server, and that it received an invalid  
> response from the back-end server.
> Why that is, is also not known to us, and quite impossible to figure  
> out with the information you provided.
>

So far I agree, and a request to / is  just  normal  and handed over  
to the backend server.


> If I had to make a guess, I would say that the problem has nothing  
> to do with this request in particular, and that your front-end  
> server is just configured with too many children or threads,  
> compared to the physical memory it really has.  So when many  
> requests come in and it tries to fork more children/threads to  
> handle them, it runs out of memory.
> Reducing MaxClients may be a first option.
>

I've already dona this, the MaxClients for the backend ist set to 10  
and  there is not much  traffic on that server. I had another server  
running with 200 frontend and 30 backendserver with much less RAM.    
Actually I don't even know for sure, if apache is causing these  
problems,  It's just the only thing running there,( tha database runs  
on another server).
I' just desperately searching for anything that couses thtes problems.


> About reproducing this access with LWP, sure :
> lwp-request -m GET http://your-server-name/
> but that is not going to tell you much.

Of course, but what I meant was, can I tell LWP  to pass  the same  
referer string I found in my logfile,  to simultate exactly that  
request ?

Rolf Schaufelberger

[awarnier]

Rolf Schaufelberger wrote:
...

> Of course, but what I meant was, can I tell LWP  to pass  the same  
> referer string I found in my logfile,  to simultate exactly that request ?
>
I'm sure you can.  That "referrer" is just a HTTP header ("Referer:"
with only one r I think), and LWP allows you to add headers to the request.
lwp-request -h

I doubt this will help you however.  If you examine your access log, you
probably have lots of such kinds of request, which do not necessarily
bring down your server.

Maybe an easier way :
- on one of your servers (suppose it is called
"yourserver.yourcompany.com"), create a really long path under the
document root and put a html page in it.
Like :
/var/www/htdocs/some/very/long/path/to/a/document/page.html
- in that html page, put a link to your front-end server
- call up that page in a browser, and click on the link.
- then look in your access log

The browser, when you click on the link in the page, sends a request to
your front-end server.
In that request, it will add a header :
Referer:
http://yourserver.yourcompany.com/some/very/long/path/to/a/document/page.html

That's what you see in the log.

[Torsten Foertsch]

On Sat 24 Oct 2009, Rolf Schaufelberger wrote:
> Starting some weeks ago the server sometimes hangs with an out of  
> memory problem.

Assuming you are running on linux the following sysctls may help to find
the culprit.

vm.overcommit_memory=2
vm.overcommit_ratio=90

By default (overcommit_memory=0) when a process needs more memory linux
simply says "okay here you are" no matter if the memory is currently
available or not. This is based on the assumption that most processes
do not use all memory they allocate at all. Later on, when the process
really accesses the memory a page fault is generated and linux only
then allocates the memory for the process. But now it is too late to
signal the process out-of-memory. So, linux somehow has to obtain
memory no matter what. So, in short-of-memory conditions linux starts
the OOM killer. It implements some heuristic that chooses some "best
fitting" processes to be killed. These "best fitting" processes may be
totally unrelated to the original problem.

I had once a case where a perl program processed mailbox files (using
Mail::Box) on a box where a postgres database ran. Unfortunately
Mail::Box reads in the whole mailbox file. Normally our mailbox files
were about 1-10mb and the program had worked for years. But suddenly we
had one of >1Gb. Instead of signaling out-of-memory to the perl process
linux killed the postgres database.

Make sure you have enough swap space (at least the RAM size) before
experimenting with those sysctls.

Torsten

--
Need professional mod_perl support?
Just hire me: torsten.foertsch@...

[Rolf Schaufelberger]

Hi, Tortsen,


Am 26.10.2009 um 12:12 schrieb Torsten Foertsch:

> On Sat 24 Oct 2009, Rolf Schaufelberger wrote:
>> Starting some weeks ago the server sometimes hangs with an out of
>> memory problem.
>
> Assuming you are running on linux the following sysctls may help to  
> find
> the culprit.
>
> vm.overcommit_memory=2
> vm.overcommit_ratio=90
>

how about setting PERL_RLIMIT_DATA ?
see (http://perl.apache.org/docs/2.0/api/Apache2/Resource.html)

Would that work ?

> By default (overcommit_memory=0) when a process needs more memory  
> linux
> simply says "okay here you are" no matter if the memory is currently
> available or not. This is based on the assumption that most processes
> do not use all memory they allocate at all. Later on, when the process
> really accesses the memory a page fault is generated and linux only
> then allocates the memory for the process. But now it is too late to
> signal the process out-of-memory. So, linux somehow has to obtain
> memory no matter what. So, in short-of-memory conditions linux starts
> the OOM killer. It implements some heuristic that chooses some "best
> fitting" processes to be killed. These "best fitting" processes may be
> totally unrelated to the original problem.
>
> I had once a case where a perl program processed mailbox files (using
> Mail::Box) on a box where a postgres database ran. Unfortunately
> Mail::Box reads in the whole mailbox file. Normally our mailbox files
> were about 1-10mb and the program had worked for years. But suddenly  
> we
> had one of >1Gb. Instead of signaling out-of-memory to the perl  
> process
> linux killed the postgres database.
>
> Make sure you have enough swap space (at least the RAM size) before
> experimenting with those sysctls.
>
> Torsten
>
> --
> Need professional mod_perl support?
> Just hire me: torsten.foertsch@...

Mit freundlichen Grüßen
Rolf Schaufelberger

plusW GmbH
Stuttgarter Str. 26       Tel.   [phone]
73635 Rudersberg    Fax   [phone]

www.plusw.de
www.mypixler.com
www.calendrino.de

[Kulasekaran, Raja]

Hi,

I have configured the mod_perl with oracle persistent connection through Apache::DBI module. On every web page request It creates a process
something like below and It never be killed automatically when the request has completed.

Can you please suggest me how to do I handle this scenario ?.

oracle    8021     1  0 17:51 ?        00:00:01 oracleXE (LOCAL=NO)
oracle    8038     1  0 17:51 ?        00:00:00 oracleXE (LOCAL=NO)
oracle    8050     1  0 17:51 ?        00:00:01 oracleXE (LOCAL=NO)
oracle    8068     1  0 17:52 ?        00:00:00 oracleXE (LOCAL=NO)

Thanks,
Raja



-----Original Message-----
From: André Warnier [mailto:aw@...]
Sent: Monday, October 26, 2009 3:17 AM
To: modperl@...
Subject: Re: Server hangs after 502 request

Rolf Schaufelberger wrote:
...

> Of course, but what I meant was, can I tell LWP  to pass  the same  
> referer string I found in my logfile,  to simultate exactly that request ?
>
I'm sure you can.  That "referrer" is just a HTTP header ("Referer:"
with only one r I think), and LWP allows you to add headers to the request.
lwp-request -h

I doubt this will help you however.  If you examine your access log, you
probably have lots of such kinds of request, which do not necessarily
bring down your server.

Maybe an easier way :
- on one of your servers (suppose it is called
"yourserver.yourcompany.com"), create a really long path under the
document root and put a html page in it.
Like :
/var/www/htdocs/some/very/long/path/to/a/document/page.html
- in that html page, put a link to your front-end server
- call up that page in a browser, and click on the link.
- then look in your access log

The browser, when you click on the link in the page, sends a request to
your front-end server.
In that request, it will add a header :
Referer:
http://yourserver.yourcompany.com/some/very/long/path/to/a/document/page.html

That's what you see in the log.

[Perrin Harkins-3]

On Tue, Oct 27, 2009 at 8:33 AM, Kulasekaran, Raja
<Raja.Kulasekaran@...> wrote:
> I have configured the mod_perl with oracle persistent connection through Apache::DBI module. On every web page request It creates a process
> something like below and It never be killed automatically when the request has completed.

That is the intended behavior.  You should get one Oracle connection
for each apache child process.  They will stay connected for the life
of the process.

- Perrin

[Joe Niederberger-2]

Hello,

Does anyone have the combination of FreeBSD 7.2, Apache2.2, mod_perl2
and libapreq2 all installed and working fine?

My web-hoster is having exterme difficulties getting this set up. The
Apache2.so is
rife with undefined symbols that they cannot resolve. Simply making a call
to the cookie jar and the process segfaults.

Any help most appreciated.
Thanks in advance.

Joe Niederberger

[Joe Niederberger-2]

Oh - the perl version is 5.8.9 -


----- Original Message -----
From: "Joe Niederberger" <jniederberger@...>
To: "mod_perl list" <modperl@...>
Sent: Tuesday, October 27, 2009 10:04 PM
Subject: FreeBSD 7.2, mod_perl2 & Apache2::Cookie (libapreq2)

> Hello,
>
> Does anyone have the combination of FreeBSD 7.2, Apache2.2, mod_perl2
> and libapreq2 all installed and working fine?
>
> My web-hoster is having exterme difficulties getting this set up. The
> Apache2.so is
> rife with undefined symbols that they cannot resolve. Simply making a call
> to the cookie jar and the process segfaults.
>
> Any help most appreciated.
> Thanks in advance.
>
> Joe Niederberger
>

[Kulasekaran, Raja]

So, How to I control this ?. Is it possible to reuse the existing
connection ?.

Raja

-----Original Message-----
From: Perrin Harkins [mailto:pharkins@...]
Sent: Tuesday, October 27, 2009 11:47 PM
To: Kulasekaran, Raja
Cc: mod_perl list
Subject: Re: mod-perl child process

On Tue, Oct 27, 2009 at 8:33 AM, Kulasekaran, Raja
<Raja.Kulasekaran@...> wrote:
> I have configured the mod_perl with oracle persistent connection
through Apache::DBI module. On every web page request It creates a
process
> something like below and It never be killed automatically when the
request has completed.

That is the intended behavior.  You should get one Oracle connection
for each apache child process.  They will stay connected for the life
of the process.

- Perrin

[Perrin Harkins-3]

On Wed, Oct 28, 2009 at 12:32 AM, Kulasekaran, Raja
<Raja.Kulasekaran@...> wrote:
> So, How to I control this ?. Is it possible to reuse the existing
> connection ?.

It should be doing that already.  Each process will open one
connection and keep it open (unless you use different connection
parameters, like a different login), reusing it each time.  What you
can't do is share a single connection between multiple processes.

If the number of connections to Oracle is a problem for you, there are
several tools out there that can help.  You'll find discussion of them
in the mailing list archives.  The first thing to do is make sure you
have a proxy in front of your mod_perl server.

- Perrin

[Kulasekaran, Raja]

Hi,

The below method used to kill the child process after the successful
execution of web request.

$r->child_terminate();

How do I get the status that particular child process has been killed ?.
Any suggestion on this ?

Raja

-----Original Message-----
From: Kulasekaran, Raja
Sent: Wednesday, October 28, 2009 10:02 AM
To: Perrin Harkins
Cc: mod_perl list
Subject: RE: mod-perl child process


So, How to I control this ?. Is it possible to reuse the existing
connection ?.

Raja

-----Original Message-----
From: Perrin Harkins [mailto:pharkins@...]
Sent: Tuesday, October 27, 2009 11:47 PM
To: Kulasekaran, Raja
Cc: mod_perl list
Subject: Re: mod-perl child process

On Tue, Oct 27, 2009 at 8:33 AM, Kulasekaran, Raja
<Raja.Kulasekaran@...> wrote:
> I have configured the mod_perl with oracle persistent connection
through Apache::DBI module. On every web page request It creates a
process
> something like below and It never be killed automatically when the
request has completed.

That is the intended behavior.  You should get one Oracle connection
for each apache child process.  They will stay connected for the life
of the process.

- Perrin

[Perrin Harkins-3]

On Thu, Oct 29, 2009 at 10:05 AM, Kulasekaran, Raja
<Raja.Kulasekaran@...> wrote:
> How do I get the status that particular child process has been killed ?

Are you talking about the exit status of the process?  You can't get
that.  What are you trying to do?

- Perrin

[Joe Niederberger-2]

Should I conclude nobody is running mod_perl2 on the
latest version of FreeBSD?


----- Original Message -----
From: "Joe Niederberger" <jniederberger@...>
To: "mod_perl list" <modperl@...>
Sent: Tuesday, October 27, 2009 10:04 PM
Subject: FreeBSD 7.2, mod_perl2 & Apache2::Cookie (libapreq2)

> Hello,
>
> Does anyone have the combination of FreeBSD 7.2, Apache2.2, mod_perl2
> and libapreq2 all installed and working fine?
>
> My web-hoster is having exterme difficulties getting this set up. The
> Apache2.so is
> rife with undefined symbols that they cannot resolve. Simply making a call
> to the cookie jar and the process segfaults.
>
> Any help most appreciated.
> Thanks in advance.
>
> Joe Niederberger
>

[Sin-6]

Hello,


Few weeks back I joined this mailling list to find out about this myself.
I've been trying to get apache 2.x and mod_perl2 to run together for many
many months now.   My target application was OTRS.   I've seen apache2 and
mod_perl2 work with Debian no problem.  I figured I was doing something
wrong.   I'm not so sure i'm in the wrong at this point after following all
readme files to the letter.


I've gone back to using apache-1.3.41_1 / mod_perl-1.31_1


Seems like all versions of 1.3 work for the past years, but apache 2 just
doesn't want to interface with the application.  Its there, installs, but
doesn't work.  Log files never come up with anything to point to.

This apache 2.x and mod_perl2 failed on FreeBSD 6 and 7.  Haven't tried 8
yet.

----- Original Message -----
From: "Joe Niederberger" <jniederberger@...>
To: "mod_perl list" <modperl@...>
Sent: Tuesday, October 27, 2009 11:04 PM
Subject: FreeBSD 7.2, mod_perl2 & Apache2::Cookie (libapreq2)

> Hello,
>
> Does anyone have the combination of FreeBSD 7.2, Apache2.2, mod_perl2
> and libapreq2 all installed and working fine?
>
> My web-hoster is having exterme difficulties getting this set up. The
> Apache2.so is
> rife with undefined symbols that they cannot resolve. Simply making a call
> to the cookie jar and the process segfaults.
>
> Any help most appreciated.
> Thanks in advance.
>
> Joe Niederberger
>
>

[Joe Niederberger-2]

Sadly, has the signs of a not-well technology.
Anyone have opinions on a way forward?
Is the answer Linux?

----- Original Message -----
From: "Sin" <sinister@...>

>
> Few weeks back I joined this mailling list to find out about this myself.
> I've been trying to get apache 2.x and mod_perl2 to run together for many
> many months now.   My target application was OTRS.   I've seen apache2 and
> mod_perl2 work with Debian no problem.  I figured I was doing something
> wrong.   I'm not so sure i'm in the wrong at this point after following
all

> readme files to the letter.
>
>
> I've gone back to using apache-1.3.41_1 / mod_perl-1.31_1
>
>
> Seems like all versions of 1.3 work for the past years, but apache 2 just
> doesn't want to interface with the application.  Its there, installs, but
> doesn't work.  Log files never come up with anything to point to.
>
> This apache 2.x and mod_perl2 failed on FreeBSD 6 and 7.  Haven't tried 8
> yet.

[awarnier]

Joe Niederberger wrote:
> Sadly, has the signs of a not-well technology.
> Anyone have opinions on a way forward?
> Is the answer Linux?
>
I have no idea about FreeBSD, but I am using Apache 2.0 & 2.2, with
mod_perl 2.x, on quite a lot of Linux (Debian, Suse, RedHat) systems,
with a lot of satisfaction and very few problems (and the few there are
generally turn out to be in my code, not in mod_perl).