rouge IPs / user

[Daniel Ouellet]

Otto Moerbeek wrote:
> I did not follow the complete thread, but I like to mention one thing:
> there might be half open connections involved here.
>
> A client might do a half close (i.e. shutdown(SHUT_WR)) after sending
> a request. This will make the connection a half-duplex one.  iirc, after the
> shutdown, the server moves to CLOSE_WAIT, but will still be able to
> send data to the client, until it decides that it is done and closes
> down the connection.

Thanks for your feedback Otto!

I keep reading this and may be my English is not getting it fully. For
my own knowledge and understanding, when you say half open and haft
close. In both cases they needed and have done the full three step
handshake or may be skip some of it?

In other word they can or can't reach the CLOSE_WAIT without doing the
full complete and proper three step handshake?

I would appreciate to understand that for myself as I thought it was
clear, but I guess I still have a gap unfill. (;>

Best,

Daniel

[Daniel Ouellet]

knitti wrote:

> On 12/12/07, Daniel Ouellet <daniel@...> wrote:
>> net.inet.tcp.keepidle
>> net.inet.tcp.keepinittime
>> net.inet.tcp.keepintvl
>> net.inet.tcp.rstppslimit
>> net.inet.tcp.synbucketlimit
>> net.inet.tcp.syncachelimit
>
> nope, shoudn't apply, unless my TCP knowledge is wrong or there
> is a bug, which makes it affecting it unintentional

They would help at making more socket to httpd available sooner, but for
this problem, you are right. It wouldn't as if I understood properly the
reading I did, it will get to CLOSE_WAIT only after the FIN/ACK and then
get to that state from where it will then wait for the application to
close and then after the application is closed, the last ACK will be
sent. That's how I understand it, so you are right and I am wrong for
the CLOSE_WAIT state.

>>>> My point with PF here was that it would reduce the possible numbers of
>>>> close_wait state you could possibly see in the first place, witch is one
>>>> of the original goal of the question.
>>> Why?
>> OK, I could be wrong and I am sure someone with a huge stick will hit me
>> with it if I say something stupid, and/or there might be something I am
>> overlooking or not understanding fully, witch is sure possible as well. (;>
>>
>> But if httpd received a fake connection that do not do the full
>> handshake, isn't it there a socket open and/or use by httpd for that
>> fake connection anyway. Meaning it tries to communicate with that fake
>> source and can't and eventually will close and (that's where may be I am
>> failing here) will end up in close_wait may be?
>
> no fake connections involved, CLOSE_WAIT is a state _after_ having a
> fully established connection

Agree, unless it's somehow possible to reach that state without having
an establish and confirmed connection first and looks like it not
possible. So, you are right again. (;>

>> Or, are you saying that the ONLY possible way a socket end up in
>> close_wait state is ONLY when and ONLY possible if it was fully open
>> properly in the first place? If so, then I stand corrected and I was/am
>> wrong about that part of my suggestions. So, is it the case then?
>
> Yes. Random example:
> http://www4.informatik.uni-erlangen.de/Projects/JX/Projects/TCP/tcpstate.html

Thanks!

Daniel

[Daniel Ouellet]

knitti wrote:

> On 12/12/07, Daniel Ouellet <daniel@...> wrote:
>> knitti wrote:
>>> The problem would be to "forget" calling ap_bclose() after ending a
>>> connection, either because all data has been sent or the connection has
>>> been aborted. What I can read with some confidence, is that keeping a
>>> socket open beyond sending any data is not intentional, and there is
>>> nothing (for me) which suggests that it would happen at all.
>> Logically if that was the case, wouldn't you think you would run out of
>> sockets in just a few minutes after starting httpd? I am not saying
>> there isn't any bugs in httpd, or that there is. Fair to assume there is
>> some, but to that extend, I couldn't imagine so. Just think about it for
>> a second. What the effect of it would be if that was the case?
>
> I think you misunderstood me. I meant I don't see any obvious occasion
> in which the problem I assumed (forgetting ap_bclose() ) would occur.
> So I don't see any bug (surpise), but something occurs. So either I don't
> see the bug because its not obvious (surprise, again), or my
> assumption (ap_bclose() not called) is wrong.
>
> My question: would not calling ap_bclose() show this behaviour ?

May be, I am not sure, but based on your previous information provided
it would make sense I guess. A stupid idea, may be something like Ajax
might keep it going... No, I don't think so.

>> - Application needs sockets and send request to create and destroy them
>> and keep using them after they are created. Who does that, kernel or
>> application?
>
> I assume the kernel creates the actual socket, but the app keeps it as long
> as it wants (or longer ;-)
>
>> - Who receive the sockets creation and destroy requests and will create
>> them or destroy them and pass the handle to the application when ready.
>> The Kernel, or the applications?
>>
>> - Who is handling the signaling, meaning handshake, opening, close_wait,
>> retransmitions, etc. Application or kernel?
>>
>> - So, in the end, if a socket is in close_wait, is it the application,
>> or the kernel at that point? Meaning, was it already requested to be
>> close and is now a signaling issue, or an application that hasn't asked
>> to close the socket yet? (;>
>
> I *assume* that it is the application forgetting to close(), because if the
> kernel "forgets" to close() something what is more or less a file, we would
> also have massive stale open files being around.

Make sense.

>> - If jam in close_wait state, is it because it hasn't send the ACK on
>> the request from the client to close the socket?
>>
>> - Or is it that it did send the ACK to the client and is now waiting on
>> the final ACK from that client to do it?
>>
>> - Or is it that it reach that point because it was an none fully open
>> three way handshake establish connection to start with may be?
>>
>> - Or it is because the client just open a socket, get what it needed and
>>   didn't bother to do the proper closing of the sockets as it should be?
>
> _please_, read my last mails, or look at a TCP state diagram.

I did, thank you.

>> - Now, where is the application, in the case httpd involved here?
>
> CLOSE_WAIT is a defined state. The most simple explaination is not
> closing the socket even after recognizing there is nothing more to
> read from it.

One would think so. What would really be nice is to be able to create
these states I guess to understand how they are staying there in the
first place. I try to see if I could, but I can't right now.

>> - Where can keep alive in httpd help, or not?
>>
>> - Where pf proxy help or not?
>>
>> - Where keep alive in tcp stack (sysctl) help or not?
>
> these three questions I simply don't understand. Please rephrase.

It was intended as a reflexion more then questions to show where each
part could affect the efficiency of httpd, but in this case, we left the
part related to it and just focus in why CLOSE_WAIT was present.
However, there must be a relation as the original posted corrected his
problem of having to many CLOSE_WAIT state by reducing the KEEP_ALIVE to
10 seconds, instead of the wrong setup of 150 seconds. However thinking
about it, unless the wrong informations was provided, there shouldn't
have been a CLOSE_WAIT issue, but a lack of available sockets. I think
they should have been FIN_WAIT_2 and not CLOSE_WAIT if I think about it.
I must have mix the two then assuming it was FIN_WAIT_2, but continue
with CLOSE_WAIT. I think you can definitely help FIN_WAIT_2, but as for
CLOSE_WAIT as you put it and as I think more about it, you are right.
Would looks like an application issue somehow.

>> That's why there isn't a single answer to the questions here and it will
>> always depend on your specific setup, traffic patterns and load, etc.
>
> I seems we are here of different opinions. I'm more or less convinced
> now, that there is a bug not closing the socket even after httpd has
> nothing more to send. Under the assumption my interpretation of the
> problem is not fundamentally flawed.

May be not of a different opinion. There obviously was a part that I
miss understood and ready more on it corrected my understanding. So, if
you haven't fix the bug in httpd, if there is one. You fix the bug in me
however. (;>

>> Example, you could reduce the keep alive in sysctl a lots if you want to
>> help the close_wait, but at the same time this will increase all the
>> exchange messages between valid connections as well. So, on one hand to
>> will affect the delay in closing your sockets sooner, but at the same
>> time you will increase the load on other already active connections.
>
> well, I think turnig the wrong knobs will do harm, there you are right.
> tuning TCP keep alives would be the wrong knob

Not in all cases, but for this particular one here yes it would be.

>> left, unless it does give you a problem, other then a feeling of wanting
>> it to look different, you should put it to rest I think.
>
> unless I can reproduce it, I will also let it rest after being convinced
> of not finding the bug by reading the code alone ;)

Good luck in your research.

Daniel

[Daniel Ouellet]

knitti wrote:
> On 12/12/07, Daniel Ouellet <daniel@...> wrote:
>> I am only
>> saying that using PF in front of httpd will reduce the possible number
>> of httpd close_wait you might see. By default httpd can only support up
>> to 256 connections, unless you increase it and compile it again.
>
> I don't understand why pf would reduce this. Every single CLOSE_WAIT
> stems from a former established connection, and pf can nothing do
> to convince httpd to close its socket. No rogue clients involved here.

Yes you are right.

>> lead you in that path, then I am sorry. What will affect your close_wait
>> time (when you reach that point) are the tcp stack value, witch I am
>> reluctant to suggest to adjust as they sure can create way more harm
>> then goods.
>
> I don't think there is a systl for that. TCP connections don't expire by
> default, if you not make them, and the same should go for a half-closed
> one. There are perfectly legit reasons for long open half-closed
> TCP connections.

In my flow of emails, I mixed two things. Glad you cut it. It would help
for normal heavy traffic, or fake one and it's possible to affect the
keep alive from the tcp side making sockets be available sooner, etc.
But again, specifically for CLOSE_WAIT, I stand corrected.

>> My point with PF here was that it would reduce the possible numbers of
>> close_wait state you could possibly see in the first place, witch is one
>> of the original goal of the question.
>
> Why?

I miss spoke ( or in this case write ) I confuse myself between that
stage and idle/waiting/FIN_WAIT_x/etc of fake connections that would
take sockets of httpd away from legit traffic that PF would/could
definitely help, however, in this tread, I lost track of where we were
in the process and specially, I had a gap in my understanding that miss
lead me as well.

Was interesting however.

Thanks

Daniel

[Otto Moerbeek]

On Wed, Dec 12, 2007 at 08:57:18PM -0500, Daniel Ouellet wrote:

> Otto Moerbeek wrote:
>> I did not follow the complete thread, but I like to mention one thing:
>> there might be half open connections involved here. A client might do a
>> half close (i.e. shutdown(SHUT_WR)) after sending
>> a request. This will make the connection a half-duplex one.  iirc, after
>> the
>> shutdown, the server moves to CLOSE_WAIT, but will still be able to
>> send data to the client, until it decides that it is done and closes
>> down the connection.
>
> Thanks for your feedback Otto!
>
> I keep reading this and may be my English is not getting it fully. For my
> own knowledge and understanding, when you say half open and haft close. In
> both cases they needed and have done the full three step handshake or may
> be skip some of it?
>
> In other word they can or can't reach the CLOSE_WAIT without doing the full
> complete and proper three step handshake?
>
> I would appreciate to understand that for myself as I thought it was clear,
> but I guess I still have a gap unfill. (;>

I'm only talking about the tear-down. The three-way handshake happens
before that, both sockets are in ESTABSLIHED state. You have to read
half-close as a verb (action), and half-open as a description of the
state of the connection.

Check the state transition diagram, and maybe do some reading in
Stevens's TCP/Illustrated Volume I (which has the state diagram on the
inside cover ;-)

        -Otto

[Daniel Ouellet]

Otto Moerbeek wrote:
> I'm only talking about the tear-down. The three-way handshake happens
> before that, both sockets are in ESTABSLIHED state. You have to read
> half-close as a verb (action), and half-open as a description of the
> state of the connection.
>
> Check the state transition diagram, and maybe do some reading in
> Stevens's TCP/Illustrated Volume I (which has the state diagram on the
> inside cover ;-)

I did a lots of reading and I cleared my miss understanding looks like
and see where I confuse myself:

1) A socket gets into CLOSE_WAIT when the remote side sends a FIN and
the local side sends an ACK, but has not yet sent it's FIN.

2) A socket gets into FIN_WAIT_2 when the local side closes the socket
and sends a FIN (FIN_WAIT_1) and then receives an ACK for that FIN but
has not yet received a FIN from the remote side.

I got confuse as I can't never see any CLOSE_WAIT on my server, no
complain for sure but I see a FIN_WAIT_2 and obviously confuse the two.

I am puzzled as to why they would get so many CLOSE_WAIT and I guess it
might be an under power server may be that when the httpd child process
is done, that the master, or what not don't have time to process it and
as such keep the socket open because of that. Not sure, but I guess
possible.

In any case, I appreciate the reply and I sure clear that part of the
handshake anyway, so I will not make the same mistakes again. (;>

Best,

Daniel

[Otto Moerbeek]

On Thu, Dec 13, 2007 at 02:25:21AM -0500, Daniel Ouellet wrote:

> Otto Moerbeek wrote:
>> I'm only talking about the tear-down. The three-way handshake happens
>> before that, both sockets are in ESTABSLIHED state. You have to read
>> half-close as a verb (action), and half-open as a description of the
>> state of the connection. Check the state transition diagram, and maybe do
>> some reading in
>> Stevens's TCP/Illustrated Volume I (which has the state diagram on the
>> inside cover ;-)
>
> I did a lots of reading and I cleared my miss understanding looks like and
> see where I confuse myself:
>
> 1) A socket gets into CLOSE_WAIT when the remote side sends a FIN and the
> local side sends an ACK, but has not yet sent it's FIN.
>
> 2) A socket gets into FIN_WAIT_2 when the local side closes the socket and
> sends a FIN (FIN_WAIT_1) and then receives an ACK for that FIN but has not
> yet received a FIN from the remote side.
>
> I got confuse as I can't never see any CLOSE_WAIT on my server, no complain
> for sure but I see a FIN_WAIT_2 and obviously confuse the two.
>
> I am puzzled as to why they would get so many CLOSE_WAIT and I guess it
> might be an under power server may be that when the httpd child process is
> done, that the master, or what not don't have time to process it and as
> such keep the socket open because of that. Not sure, but I guess possible.
>
> In any case, I appreciate the reply and I sure clear that part of the
> handshake anyway, so I will not make the same mistakes again. (;>
>
> Best,
>
> Daniel

It's easy to force a socket into CLOSE_WAIT

On server:

$ nc -l 10000

On client:

$ nc 10000

That gets you ESTABLISHED on both sides.

^Z the server side

close the client side by typing ^D

The client moves to FIN_WAIT_2, and the server to CLOSE_WAIT.

fg the server process

The client moves to TIME_WAIT and the server socket will disappear.

It's depening on the application if sockets staying in CLOSE_WAIT are
a problem or not: it might be intentional (in the hulp duplex case),
or it might be a program "forgetting" to do a close.

        -Otto
       
BTW: it's volume II that has the state diagram on the inside cover.
Volume I has it in chapter 18, after the discussion of half-close.

[Stuart Henderson]

On 2007/12/13 09:09, Otto Moerbeek wrote:
> It's depening on the application if sockets staying in CLOSE_WAIT are
> a problem or not: it might be intentional (in the hulp duplex case),
> or it might be a program "forgetting" to do a close.

Does select() notify the application of FIN from the other side?

If not, that would explain things, it wouldn't be reasonable for
httpd to manually try and receive from all sockets in keepalive
to see whether it needs to close the socket, since it will only
wait KeepAliveTimeout (default 15s) before it closes them anyway.

[Hannah Schroeter]

Hi!

On Thu, Dec 13, 2007 at 11:10:51AM +0000, Stuart Henderson wrote:
>On 2007/12/13 09:09, Otto Moerbeek wrote:
>> It's depening on the application if sockets staying in CLOSE_WAIT are
>> a problem or not: it might be intentional (in the hulp duplex case),
>> or it might be a program "forgetting" to do a close.

>Does select() notify the application of FIN from the other side?

I haven't followed the rest of the thread, so answering this question
out of context: Yes, FIN from the other side says that there'll be no
other data from the other side, i.e. that'll lead to an EOF condition on
the data stream. So after the application has read all the remaining
data from the socket, the socket will be readable (in the sense of the
read bit from select(), or POLLIN from poll(), or EVFILT_READ for
kqueue/kevent) and if you read, it'll return 0, just the same as if you
have EOF on files.

>[...]

Kind regards,

Hannah.

[Otto Moerbeek]

On Thu, Dec 13, 2007 at 11:10:51AM +0000, Stuart Henderson wrote:

> On 2007/12/13 09:09, Otto Moerbeek wrote:
> > It's depening on the application if sockets staying in CLOSE_WAIT are
> > a problem or not: it might be intentional (in the hulp duplex case),

Strange typo by me... that's a Dutch word, but not very relevant in
this context. It should be half-duplex of course. Stupid autonomous
fingers ;-)

> > or it might be a program "forgetting" to do a close.
>
> Does select() notify the application of FIN from the other side?
>
> If not, that would explain things, it wouldn't be reasonable for
> httpd to manually try and receive from all sockets in keepalive
> to see whether it needs to close the socket, since it will only
> wait KeepAliveTimeout (default 15s) before it closes them anyway.

Nice suggestion, but if you've marked the fd for read I would expect
select to notify if the other side does a shutdown(SHUT_WR).

Other scenarios are also thinkable: like the server socket being
blocked because of outgoing data that cannot be written out. That
might prevent the server from doing a close too. But in the end the
close will happen, otherwise you would run out of fd's very soon.

        -Otto

[Raimo Niskanen-7]

On Thu, Dec 13, 2007 at 12:35:03PM +0100, Otto Moerbeek wrote:

> On Thu, Dec 13, 2007 at 11:10:51AM +0000, Stuart Henderson wrote:
>
> > On 2007/12/13 09:09, Otto Moerbeek wrote:
> > > It's depening on the application if sockets staying in CLOSE_WAIT are
> > > a problem or not: it might be intentional (in the hulp duplex case),
>
> Strange typo by me... that's a Dutch word, but not very relevant in
> this context. It should be half-duplex of course. Stupid autonomous
> fingers ;-)
>
> > > or it might be a program "forgetting" to do a close.
> >
> > Does select() notify the application of FIN from the other side?
> >
> > If not, that would explain things, it wouldn't be reasonable for
> > httpd to manually try and receive from all sockets in keepalive
> > to see whether it needs to close the socket, since it will only
> > wait KeepAliveTimeout (default 15s) before it closes them anyway.
>
> Nice suggestion, but if you've marked the fd for read I would expect
> select to notify if the other side does a shutdown(SHUT_WR).
>
> Other scenarios are also thinkable: like the server socket being
> blocked because of outgoing data that cannot be written out. That
> might prevent the server from doing a close too. But in the end the
> close will happen, otherwise you would run out of fd's very soon.
>

I have been doing a little ktrace:ing on the httpd worker process(es)
that gets stuck with a socket in CLOSE_WAIT.

It appears when it finally (after e.g 8.5 minutes) continues, it:
 28749 httpd    1197541469.470707 EMUL  "native"
 28749 httpd    53.276754 RET   write -1 errno 32 Broken pipe
 28749 httpd    0.000031 CALL  close(0x4)
 28749 httpd    0.000005 RET   close 0
 28749 httpd    0.000012 CALL  gettimeofday(0xcfbd12c8,0)
 28749 httpd    0.000002 RET   gettimeofday 0
 28749 httpd    0.000002 CALL  gettimeofday(0xcfbd12c8,0)
 28749 httpd    0.000001 RET   gettimeofday 0
 28749 httpd    0.000002 CALL  stat(0x2d5a3945,0xcfbd1370)
 28749 httpd    0.000002 NAMI  "/etc/resolv.conf"
 28749 httpd    0.000009 RET   stat 0
 28749 httpd    0.000012 CALL  open(0x2d5a1ffb,0,0x1b6)
 28749 httpd    0.000004 NAMI  "/etc/hosts"
 28749 httpd    0.000005 RET   open 4
 28749 httpd    0.000003 CALL  fstat(0x4,0xcfbd16d0)
 28749 httpd    0.000002 RET   fstat 0
 28749 httpd    0.000003 CALL  mprotect(0x8111e000,0x1000,0x3)
 28749 httpd    0.000005 RET   mprotect 0
 28749 httpd    0.000006 CALL  mprotect(0x8111e000,0x1000,0x1)
 28749 httpd    0.000003 RET   mprotect 0
 28749 httpd    0.000002 CALL  read(0x4,0x8103a000,0x4000)
 28749 httpd    0.000004 GIO   fd 4 read 725 bytes
that must be the start of finding my own hostname, it proceeds with
looking up the remote host over DNS and then writes a log entry:
 28749 httpd    0.000019 CALL  write(0x16,0x7cc0b4ac,0x7c)
 28749 httpd    0.000008 GIO   fd 22 wrote 124 bytes
:
 28749 httpd    0.000002 RET   write 124/0x7c
 28749 httpd    0.000011 CALL  close(0x3)
 28749 httpd    0.000016 RET   close 0
 28749 httpd    0.000003 CALL  sigaction(0x1e,0xcfbd1e00,0xcfbd1df0)
 28749 httpd    0.000003 RET   sigaction 0
 28749 httpd    0.000004 CALL  accept(0x12,0xcfbd1e50,0xcfbd1e3c)
 28749 httpd    2.751318 RET   accept 3
there comes the next connection. Note the close(0x3) call.
The lsof output said the HTTP connection socket had fd 3.

To compare with the end of a later successful request/reply
in the same log:
 28749 httpd    0.000004 CALL  write(0x3,0x85b7200c,0xb57)
 28749 httpd    0.000042 GIO   fd 3 wrote 2903 bytes
:
 28749 httpd    0.000010 RET   write 2903/0xb57
 28749 httpd    0.000006 CALL  gettimeofday(0xcfbd12c8,0)
 28749 httpd    0.000003 RET   gettimeofday 0
 28749 httpd    0.000002 CALL  gettimeofday(0xcfbd12c8,0)
 28749 httpd    0.000002 RET   gettimeofday 0
 28749 httpd    0.000001 CALL  stat(0x2d5a3945,0xcfbd1370)
 28749 httpd    0.000002 NAMI  "/etc/resolv.conf"
 28749 httpd    0.000009 RET   stat 0
and then lookup of my name and the other name, and write log:
 28749 httpd    0.000021 CALL  write(0x16,0x7cc0b2ac,0x97)
 28749 httpd    0.000008 GIO   fd 22 wrote 151 bytes
:
 28749 httpd    0.000002 RET   write 151/0x97
 28749 httpd    0.000011 CALL  shutdown(0x3,0x1)
 28749 httpd    0.000017 RET   shutdown 0
 28749 httpd    0.000003 CALL  select(0x4,0xcfbd1b70,0,0,0xcfbd1b68)
 28749 httpd    0.115966 RET   select 1
 28749 httpd    0.000009 CALL  read(0x3,0xcfbd1bf0,0x200)
 28749 httpd    0.000004 RET   read 0
 28749 httpd    0.000004 CALL  close(0x3)
 28749 httpd    0.000004 RET   close 0
 28749 httpd    0.000005 CALL  sigaction(0x1e,0xcfbd1e00,0xcfbd1df0)
 28749 httpd    0.000002 RET   sigaction 0
 28749 httpd    0.000002 CALL  munmap(0x8b60d000,0xa2b)
 28749 httpd    0.000018 RET   munmap 0
 28749 httpd    0.000016 CALL  accept(0x12,0xcfbd1e50,0xcfbd1e3c)

Note that close(0x3) follows after select() RET 1, read() RET 0.

In the fail case log, close(0x3) is called without select()
and read() probably because it is already known fd 3 is
dead due to the failing write().

So, the conclusion closest at hand is that it is a write of HTTP data
on fd 3 that times out with "errno 32 Broken pipe" after 8.5 minutes,
even though lsof reports the socket as being in CLOSE_WAIT.

Then the question is if this is a TCP stack bug, or a feature.
One might expect write() to return immediately when the
socket enters CLOSE_WAIT due to remote end socket close...



> -Otto

--

/ Raimo Niskanen, Erlang/OTP, Ericsson AB

[Raimo Niskanen-7]

On Thu, Dec 13, 2007 at 12:35:03PM +0100, Otto Moerbeek wrote:
:

> > > or it might be a program "forgetting" to do a close.
> >
> > Does select() notify the application of FIN from the other side?
> >
> > If not, that would explain things, it wouldn't be reasonable for
> > httpd to manually try and receive from all sockets in keepalive
> > to see whether it needs to close the socket, since it will only
> > wait KeepAliveTimeout (default 15s) before it closes them anyway.
>
> Nice suggestion, but if you've marked the fd for read I would expect
> select to notify if the other side does a shutdown(SHUT_WR).
>
> Other scenarios are also thinkable: like the server socket being
> blocked because of outgoing data that cannot be written out. That
> might prevent the server from doing a close too. But in the end the
> close will happen, otherwise you would run out of fd's very soon.
>
> -Otto

The behaviour is starting to make sense now. Scenario:

* The client connects to the server, sends its request and
  then closes the socket, that is shutdown() aka half-close.
  It can still read the reply.
* The server accepts the connection, reads the request,
  and may or may not notice that the client has done
  a shutdown() - it is not important. Nevertheless the
  server can not close the socket since it has a
  reply to deliver. And the server host TCP stack
  has noticed the shutdown() so the socket already
  enters CLOSE_WAIT.
* The server starts sending the reply which may be large
  e.g a file download. In the middle of this transfer
  the client's ethernet cable gets plugged out, the
  client host gets powered off, a firewall in the
  path goes bananas or whatnot.
* The server is now stuck in a write() call since the
  server host TCP stack has to wait quite a while
  to be sure the connection is really dead.
  And the state is still CLOSE_WAIT.

If the client program would die, the client host TCP
stack would close the socket and tell the server host
TCP stack, that would fail the hanging write() call.
So there must be a harder error such as network
outage or power outage to induce this problem.

If this scenario is correct, there is nothing to do
about it, except decreasing the likelyhood of the
server socket being half-closed while sending
the reply, and having KeepAliveTimeout in
httpd.conf at its default (15) or slightly lower
seems to do the trick. But I do not know how.

If there is some quirk in httpd's implementation
of the KeepAliveTimeout that makes it not notice
the half-close and keeps the socket open the whole
KeepAliveTimeout, that would explain it.

--

/ Raimo Niskanen, Erlang/OTP, Ericsson AB