tcp_close during async accept

[Joel Reymont]

I'm playing with Serge's "Non-blocking TCP server using OTP  
principles" [1]. When spawning a large number of connections almost at  
once, I'm seeing tcp_close messages on the client end, even before  
seeing inet_async on the client end.

I may spawn 700 client connections and only half may go through, even  
though my backlog option is 1024, e.g.

     Opts = [binary,
             {packet, 0},
             {reuseaddr, true},
             {keepalive, true},
             {backlog, 1024},
             {active, false}],
     case gen_tcp:listen(Port, Opts) of
         {ok, Socket} ->
             %% Create first accepting process
             {ok, Ref} = prim_inet:async_accept(Socket, -1),
...

Any suggestions on what's going on?

Is it kosher to use prim_inet:async_accept/2?

Am I using backlog correctly?

        Thanks, Joel

[1] http://www.trapexit.org/Building_a_Non-blocking_TCP_server_using_OTP_principles

---
Mac hacker with a performance bent
http://www.linkedin.com/in/joelreymont


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Eli Liang]

http://seekingalpha.com/article/147104-is-a-case-of-quant-trading-sabotage-about-to-destroy-goldman-sachs



     

[Benjamin Tolputt-2]

Eli Liang wrote:
> http://seekingalpha.com/article/147104-is-a-case-of-quant-trading-sabotage-about-to-destroy-goldman-sachs     
>  
I wouldn't say that the news is all bad. As far as Erlang/OTP is
concerned, it is part of a vital distributed, high-speed,
high-reliability, high-income producing system put together for one of
the world's leading financial firms.

The fact it is buried in a story regarding regarding corporate
sabotage/theft shouldn't take away from the best part of the story for
Erlang. That is, a system put together using Erlang/OTP (amongst other
technologies) is so critical to the "low latency trading" operations of
Goldman Sachs that when it was shut down (for legal, not technical
reasons) - it removed the top trading financial form from first place in
the NYSE reports to non-existent!

What better reference in the financial industry could you ask for?

--
Regards,

Benjamin Tolputt
Analyst Programmer


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Joel Reymont]

On Jul 6, 2009, at 11:55 PM, Eli Liang wrote:

> http://seekingalpha.com/article/147104-is-a-case-of-quant-trading-sabotage-about-to-destroy-goldman-sachs


I interviewed with Serge after he left Goldman, just a short while ago.

Decided to stay put in Tenerife rather than work for the trading  
startup in Chicago.

The official complaint is here:

http://www.scribd.com/doc/17118166/Complaint-Aleynikov

Also, Bloomberg has a much better story:

http://www.bloomberg.com/apps/news?pid=20601103&sid=axYw_ykTBokE

---
Mac hacker with a performance bent
http://www.linkedin.com/in/joelreymont


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Trevor Woollacott [ MTN - Innovation Centre ]]

What OS are you using? Maybe the maximum backlog on your platform is less than 1024?

Regards,
Trevor

> -----Original Message-----
> From: erlang-questions@... [mailto:erlang-questions@...]
> On Behalf Of Joel Reymont
> Sent: Monday, 06 July 2009 06:21 PM
> To: Erlang Users' List
> Subject: [erlang-questions] tcp_close during async accept
>
> I'm playing with Serge's "Non-blocking TCP server using OTP
> principles" [1]. When spawning a large number of connections almost at
> once, I'm seeing tcp_close messages on the client end, even before
> seeing inet_async on the client end.
>
> I may spawn 700 client connections and only half may go through, even
> though my backlog option is 1024, e.g.
>
>      Opts = [binary,
>              {packet, 0},
>              {reuseaddr, true},
>              {keepalive, true},
>              {backlog, 1024},
>              {active, false}],
>      case gen_tcp:listen(Port, Opts) of
>          {ok, Socket} ->
>              %% Create first accepting process
>              {ok, Ref} = prim_inet:async_accept(Socket, -1),
> ...
>
> Any suggestions on what's going on?
>
> Is it kosher to use prim_inet:async_accept/2?
>
> Am I using backlog correctly?
>
>       Thanks, Joel
>
> [1] http://www.trapexit.org/Building_a_Non-
> blocking_TCP_server_using_OTP_principles
>
> ---
> Mac hacker with a performance bent
> http://www.linkedin.com/in/joelreymont
>
>
> ________________________________________________________________
> erlang-questions mailing list. See http://www.erlang.org/faq.html
> erlang-questions (at) erlang.org

NOTE: This e-mail message is subject to the MTN Group disclaimer see http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Joel Reymont]

On Jul 7, 2009, at 12:10 PM, Trevor Woollacott [ MTN - Innovation  
Centre ] wrote:

> What OS are you using? Maybe the maximum backlog on your platform is  
> less than 1024?


Mac OSX 10.5.7

---
Mac hacker with a performance bent
http://www.linkedin.com/in/joelreymont


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Trevor Woollacott [ MTN - Innovation Centre ]]

I'm not sure for Max OSX, but I think it is 128. Although that wouldn't explain why only half of your 700 connections are successful. Maybe there is a fudge factor involved in the calculation of the actual backlog value, or maybe the listener process isn't accepting connections quickly enough?
I know for Windows Sockets 1.1 the maximum value is 5, and some versions of Windows (such as Windows 2000 Server) allow a backlog value of up to 200.

> -----Original Message-----
> From: Joel Reymont [mailto:joelr1@...]
> Sent: Tuesday, 07 July 2009 04:31 PM
> To: Trevor Woollacott [ MTN - Innovation Centre ]
> Cc: Erlang Users' List
> Subject: Re: [erlang-questions] tcp_close during async accept
>
>
> On Jul 7, 2009, at 12:10 PM, Trevor Woollacott [ MTN - Innovation
> Centre ] wrote:
>
> > What OS are you using? Maybe the maximum backlog on your platform is
> > less than 1024?
>
>
> Mac OSX 10.5.7
>
> ---
> Mac hacker with a performance bent
> http://www.linkedin.com/in/joelreymont

NOTE: This e-mail message is subject to the MTN Group disclaimer see http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Oscar Hellström-2]

There might be syn cookies involved as well? That way you kind of
believe (in the client) that the connection is successful, or at least
that's how I understand it. I also don't know about OSX, but at least in
Linux you can reconfigure most of this with sysctl...

Trevor Woollacott [ MTN - Innovation Centre ] wrote:

> I'm not sure for Max OSX, but I think it is 128. Although that wouldn't explain why only half of your 700 connections are successful. Maybe there is a fudge factor involved in the calculation of the actual backlog value, or maybe the listener process isn't accepting connections quickly enough?
> I know for Windows Sockets 1.1 the maximum value is 5, and some versions of Windows (such as Windows 2000 Server) allow a backlog value of up to 200.
>
>  
>> -----Original Message-----
>> From: Joel Reymont [mailto:joelr1@...]
>> Sent: Tuesday, 07 July 2009 04:31 PM
>> To: Trevor Woollacott [ MTN - Innovation Centre ]
>> Cc: Erlang Users' List
>> Subject: Re: [erlang-questions] tcp_close during async accept
>>
>>
>> On Jul 7, 2009, at 12:10 PM, Trevor Woollacott [ MTN - Innovation
>> Centre ] wrote:
>>
>>    
>>> What OS are you using? Maybe the maximum backlog on your platform is
>>> less than 1024?
>>>      
>> Mac OSX 10.5.7
>>
>> ---
>> Mac hacker with a performance bent
>> http://www.linkedin.com/in/joelreymont
>>    
>
>
> NOTE: This e-mail message is subject to the MTN Group disclaimer see http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx
>
> ________________________________________________________________
> erlang-questions mailing list. See http://www.erlang.org/faq.html
> erlang-questions (at) erlang.org
>
>  

--
Oscar Hellström, oscar@...
Office: [phone]
Mobile: [phone]
Erlang Training and Consulting Ltd
http://www.erlang-consulting.com/


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Valentin Micic-2]

In my experience, a backlog value is rarely an issue in practice (although
testing may be another matter). What's more often a problem when you do
high-volume testing is number of file descriptors (e.g. maybe you did change
it in ERTS but not at kernel level, or vice-versa), or even value for
TIME_WAIT period etc. What makes you believe that the problem is with
backlog?
 
V.

-----Original Message-----
From: erlang-questions@... [mailto:erlang-questions@...] On
Behalf Of Trevor Woollacott [ MTN - Innovation Centre ]
Sent: 08 July 2009 12:45 PM
To: Joel Reymont
Cc: erlang-questions@...
Subject: RE: [erlang-questions] tcp_close during async accept

I'm not sure for Max OSX, but I think it is 128. Although that wouldn't
explain why only half of your 700 connections are successful. Maybe there is
a fudge factor involved in the calculation of the actual backlog value, or
maybe the listener process isn't accepting connections quickly enough?
I know for Windows Sockets 1.1 the maximum value is 5, and some versions of
Windows (such as Windows 2000 Server) allow a backlog value of up to 200.

> -----Original Message-----
> From: Joel Reymont [mailto:joelr1@...]
> Sent: Tuesday, 07 July 2009 04:31 PM
> To: Trevor Woollacott [ MTN - Innovation Centre ]
> Cc: Erlang Users' List
> Subject: Re: [erlang-questions] tcp_close during async accept
>
>
> On Jul 7, 2009, at 12:10 PM, Trevor Woollacott [ MTN - Innovation
> Centre ] wrote:
>
> > What OS are you using? Maybe the maximum backlog on your platform is
> > less than 1024?
>
>
> Mac OSX 10.5.7
>
> ---
> Mac hacker with a performance bent
> http://www.linkedin.com/in/joelreymont

NOTE: This e-mail message is subject to the MTN Group disclaimer see
http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Jim Morris-3]

I am using the same async code in my server, and have tested several
thousand connections with no problem.

Are you testing client and server on same machine? if not are they on
the same network? Is there something in between that is doing TCP SYN
attack mitigation? Maybe OS/X does SYN attack protection builtin?
The trouble with testing so many simultaneous connections in a short
time is it does look like a SYN attack.

One thing I did find is my tester client had to have a very short
delay between the tcp opens, around 10ms was enough to significantly
speed up the test. Hitting my server with a thousand simultaneous
opens did cause the server some issues, however not the same as you
report, pacing the opens slightly had a significant impact on
performance.

On Jul 8, 5:41 am, "Valentin Micic" <v...@...> wrote:

> In my experience, a backlog value is rarely an issue in practice (although
> testing may be another matter). What's more often a problem when you do
> high-volume testing is number of file descriptors (e.g. maybe you did change
> it in ERTS but not at kernel level, or vice-versa), or even value for
> TIME_WAIT period etc. What makes you believe that the problem is with
> backlog?
>
> V.
>
> -----Original Message-----
> From: erlang-questi...@... [mailto:erlang-questi...@...] On
>
> Behalf Of Trevor Woollacott [ MTN - Innovation Centre ]
> Sent: 08 July 2009 12:45 PM
> To: Joel Reymont
> Cc: erlang-questi...@...
> Subject: RE: [erlang-questions] tcp_close during async accept
>
> I'm not sure for Max OSX, but I think it is 128. Although that wouldn't
> explain why only half of your 700 connections are successful. Maybe there is
> a fudge factor involved in the calculation of the actual backlog value, or
> maybe the listener process isn't accepting connections quickly enough?
> I know for Windows Sockets 1.1 the maximum value is 5, and some versions of
> Windows (such as Windows 2000 Server) allow a backlog value of up to 200.
>
> > -----Original Message-----
> > From: Joel Reymont [mailto:joe...@...]
> > Sent: Tuesday, 07 July 2009 04:31 PM
> > To: Trevor Woollacott [ MTN - Innovation Centre ]
> > Cc: Erlang Users' List
> > Subject: Re: [erlang-questions] tcp_close during async accept
>
> > On Jul 7, 2009, at 12:10 PM, Trevor Woollacott [ MTN - Innovation
> > Centre ] wrote:
>
> > > What OS are you using? Maybe the maximum backlog on your platform is
> > > less than 1024?
>
> > Mac OSX 10.5.7
>
> > ---
> > Mac hacker with a performance bent
> >http://www.linkedin.com/in/joelreymont
>
> NOTE: This e-mail message is subject to the MTN Group disclaimer seehttp://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx
>
> ________________________________________________________________
> erlang-questions mailing list. Seehttp://www.erlang.org/faq.html
> erlang-questions (at) erlang.org
>
> ________________________________________________________________
> erlang-questions mailing list. Seehttp://www.erlang.org/faq.html
> erlang-questions (at) erlang.org
>
>

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Trevor Woollacott]

Thats true, running out of file descriptors is likely. If that is the case
then you should be seeing EMFILE or ENFILE errors.

I had assumed that the TIME_WAIT period wouldn't be a problem because you
are using SO_REUSEADDR in the listener. But I suppose it could be a problem
if the clients have their ports defined manually. For example, if the client
on PORT A is connected to the remote server on PORT B, and then the
connection closes, the client will still be in TIME_WAIT state. The same
client on PORT A will not be able to reconnect to the servers remote address
until the TIME_WAIT state is over, even with SO_REUSEADDR.

The best thing to do is snoop the network traffic to see exactly what is
going on.

----- Original Message -----
From: "Valentin Micic" <v@...>
To: "'Trevor Woollacott [ MTN - Innovation Centre ]'" <Woolla_T@...>;
"'Joel Reymont'" <joelr1@...>
Cc: <erlang-questions@...>
Sent: Wednesday, July 08, 2009 2:41 PM
Subject: RE: [erlang-questions] tcp_close during async accept

> In my experience, a backlog value is rarely an issue in practice (although
> testing may be another matter). What's more often a problem when you do
> high-volume testing is number of file descriptors (e.g. maybe you did
> change
> it in ERTS but not at kernel level, or vice-versa), or even value for
> TIME_WAIT period etc. What makes you believe that the problem is with
> backlog?
>
> V.
>
> -----Original Message-----
> From: erlang-questions@... [mailto:erlang-questions@...] On
> Behalf Of Trevor Woollacott [ MTN - Innovation Centre ]
> Sent: 08 July 2009 12:45 PM
> To: Joel Reymont
> Cc: erlang-questions@...
> Subject: RE: [erlang-questions] tcp_close during async accept
>
> I'm not sure for Max OSX, but I think it is 128. Although that wouldn't
> explain why only half of your 700 connections are successful. Maybe there
> is
> a fudge factor involved in the calculation of the actual backlog value, or
> maybe the listener process isn't accepting connections quickly enough?
> I know for Windows Sockets 1.1 the maximum value is 5, and some versions
> of
> Windows (such as Windows 2000 Server) allow a backlog value of up to 200.
>
>> -----Original Message-----
>> From: Joel Reymont [mailto:joelr1@...]
>> Sent: Tuesday, 07 July 2009 04:31 PM
>> To: Trevor Woollacott [ MTN - Innovation Centre ]
>> Cc: Erlang Users' List
>> Subject: Re: [erlang-questions] tcp_close during async accept
>>
>>
>> On Jul 7, 2009, at 12:10 PM, Trevor Woollacott [ MTN - Innovation
>> Centre ] wrote:
>>
>> > What OS are you using? Maybe the maximum backlog on your platform is
>> > less than 1024?
>>
>>
>> Mac OSX 10.5.7
>>
>> ---
>> Mac hacker with a performance bent
>> http://www.linkedin.com/in/joelreymont
>
>
> NOTE: This e-mail message is subject to the MTN Group disclaimer see
> http://www.mtn.co.za/SUPPORT/LEGAL/Pages/EmailDisclaimer.aspx
>
> ________________________________________________________________
> erlang-questions mailing list. See http://www.erlang.org/faq.html
> erlang-questions (at) erlang.org
>
>
> ________________________________________________________________
> erlang-questions mailing list. See http://www.erlang.org/faq.html
> erlang-questions (at) erlang.org

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Joel Reymont]

On Jul 8, 2009, at 4:19 PM, Jim Morris wrote:

> Are you testing client and server on same machine?

Yes.

> if not are they on
> the same network? Is there something in between that is doing TCP SYN
> attack mitigation?

I believe I saw the same effect on Amazon EC2.

> One thing I did find is my tester client had to have a very short
> delay between the tcp opens, around 10ms was enough to significantly
> speed up the test.

I'm reconnecting after a random delay when this happens. I was  
wondering why it's a successful connection on the client side and then  
a tcp_close since it's not even getting to the Erlang accept on the  
server side.

        Thanks, Joel

---
Mac hacker with a performance bent
http://www.linkedin.com/in/joelreymont


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Joel Reymont]

On Jul 8, 2009, at 4:45 PM, Trevor Woollacott wrote:

> Thats true, running out of file descriptors is likely. If that is  
> the case then you should be seeing EMFILE or ENFILE errors.

This is not the case in my test, although I did report a related issue:

Mac OSX 10.5.7

ulimit -n
10240

Erlang R13B01 (erts-5.7.2) [source] [smp:2:2] [rq:2] [async-threads:0]  
[kernel-poll:true]

=ERROR REPORT==== 6-Jul-2009::20:51:54 ===
driver_select(0x00000041, 1024, ERL_DRV_WRITE ERL_DRV_USE, 1) by  
tcp_inet driver #Port<0.3137> failed: fd=1024 is larger than the  
largest allowed fd=1023

=ERROR REPORT==== 6-Jul-2009::20:51:55 ===
File operation error: system_limit. Target: s3erl/ebin/random.beam.  
Function: get_file. Process: code_server.

---
Mac hacker with a performance bent
http://www.linkedin.com/in/joelreymont


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Jim Morris-3]

Oh BTW my server is running on a recent Linux Kernel...

Erlang R13B (erts-5.7.1) [source] [smp:2:2] [rq:2] [async-threads:10]
[hipe] [kernel-poll:true]

So maybe it is an OS/X thing?

On Jul 8, 9:27 am, Joel Reymont <joe...@...> wrote:

> On Jul 8, 2009, at 4:19 PM, Jim Morris wrote:
>
> > Are you testing client and server on same machine?
>
> Yes.
>
> > if not are they on
> > the same network? Is there something in between that is doing TCP SYN
> > attack mitigation?
>
> I believe I saw the same effect on Amazon EC2.
>
> > One thing I did find is my tester client had to have a very short
> > delay between the tcp opens, around 10ms was enough to significantly
> > speed up the test.
>
> I'm reconnecting after a random delay when this happens. I was  
> wondering why it's a successful connection on the client side and then  
> a tcp_close since it's not even getting to the Erlang accept on the  
> server side.
>
>         Thanks, Joel
>
> ---
> Mac hacker with a performance benthttp://www.linkedin.com/in/joelreymont
>
> ________________________________________________________________
> erlang-questions mailing list. Seehttp://www.erlang.org/faq.html
> erlang-questions (at) erlang.org

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Javier París]

El 08/07/2009, a las 18:27, Joel Reymont escribió:
>
> I'm reconnecting after a random delay when this happens. I was  
> wondering why it's a successful connection on the client side and  
> then a tcp_close since it's not even getting to the Erlang accept on  
> the server side.

The easiest way to debug these kind of errors is to have a look at the  
actual packet trace with a sniffer. Look at wether the server is  
sending syn+acks to all the connection attempts. You can also look for  
rst that may explain that "open in the client but immediately close"  
behaviour.

Regarding what Oscar suggested about syn cookies, they are indeed a  
syn flood prevention technique, but they do not actually drop any  
connection so even if it is being used here it should not be an issue.  
You can guess if it is indeed used by the server if you see that some  
of the syn+acks replies it sends do not have any tcp option.

Regards.


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Steve Davis-5]

Joel et al.

I'm not sure if this is related but I'm seeing the same thing about
30-50$ of the time on a client socket connection when using...

  ok = inet:setopts(Socket, [{packet, http_bin}, {keepalive, true}]),
  case gen_tcp:recv(Socket, 0, Timeout) of
  ...

...possible bug in R13B01, or perhaps a bug in my usage of the socket?

/sd


On Jul 6, 11:20 am, Joel Reymont <joe...@...> wrote:

> I'm playing with Serge's "Non-blocking TCP server using OTP  
> principles" [1]. When spawning a large number of connections almost at  
> once, I'm seeing tcp_close messages on the client end, even before  
> seeing inet_async on the client end.
>
> I may spawn 700 client connections and only half may go through, even  
> though my backlog option is 1024, e.g.
>
>      Opts = [binary,
>              {packet, 0},
>              {reuseaddr, true},
>              {keepalive, true},
>              {backlog, 1024},
>              {active, false}],
>      case gen_tcp:listen(Port, Opts) of
>          {ok, Socket} ->
>              %% Create first accepting process
>              {ok, Ref} = prim_inet:async_accept(Socket, -1),
> ...
>
> Any suggestions on what's going on?
>

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Jim Morris-3]

I doubt this is the problem, but I notice you both use tcp keep alive.
In the past this option has had issues in various TCP stacks, and if
my memory serves me correctly its use was frowned upon. That may be
old data, but I have been writing servers for many many years and have
never used that option.

I prefer to have the application protocol include some type of ping
that the client sends to the server every 30 seconds, this keeps any
intermediate NAT/Firewall box from losing the connection, and allows
the server to know when a client has vanished due to a cable unplug or
router going away (ie it never sees the FIN or RST). This works well
for HTTP long-poll type clients as well as proprietary clients.


On Jul 8, 3:17 pm, Steve Davis <steven.charles.da...@...> wrote:

> Joel et al.
>
> I'm not sure if this is related but I'm seeing the same thing about
> 30-50$ of the time on a client socket connection when using...
>
>   ok = inet:setopts(Socket, [{packet, http_bin}, {keepalive, true}]),
>   case gen_tcp:recv(Socket, 0, Timeout) of
>   ...
>
> ...possible bug in R13B01, or perhaps a bug in my usage of the socket?
>
> /sd
>
> On Jul 6, 11:20 am, Joel Reymont <joe...@...> wrote:
>
>
>
> > I'm playing with Serge's "Non-blocking TCP server using OTP  
> > principles" [1]. When spawning a large number of connections almost at  
> > once, I'm seeing tcp_close messages on the client end, even before  
> > seeing inet_async on the client end.
>
> > I may spawn 700 client connections and only half may go through, even  
> > though my backlog option is 1024, e.g.
>
> >      Opts = [binary,
> >              {packet, 0},
> >              {reuseaddr, true},
> >              {keepalive, true},
> >              {backlog, 1024},
> >              {active, false}],
> >      case gen_tcp:listen(Port, Opts) of
> >          {ok, Socket} ->
> >              %% Create first accepting process
> >              {ok, Ref} = prim_inet:async_accept(Socket, -1),
> > ...
>
> > Any suggestions on what's going on?
>
> ________________________________________________________________
> erlang-questions mailing list. Seehttp://www.erlang.org/faq.html
> erlang-questions (at) erlang.org

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Steve Davis-5]

Hi Jim,

Indeed you are correct. I don't need it and my expectation was wrong
about its underlying meaning! I was just coming back to post that the
bug was in my brain/code and not the platform code.

Thank you for response (sic!).

Regards,
Steve

On Jul 8, 6:16 pm, Jim Morris <wolfma...@...> wrote:

> I doubt this is the problem, but I notice you both use tcp keep alive.
> In the past this option has had issues in various TCP stacks, and if
> my memory serves me correctly its use was frowned upon. That may be
> old data, but I have been writing servers for many many years and have
> never used that option.
>
> I prefer to have the application protocol include some type of ping
> that the client sends to the server every 30 seconds, this keeps any
> intermediate NAT/Firewall box from losing the connection, and allows
> the server to know when a client has vanished due to a cable unplug or
> router going away (ie it never sees the FIN or RST). This works well
> for HTTP long-poll type clients as well as proprietary clients.
>
> On Jul 8, 3:17 pm, Steve Davis <steven.charles.da...@...> wrote:
>
>
>
>
>
> > Joel et al.
>
> > I'm not sure if this is related but I'm seeing the same thing about
> > 30-50$ of the time on a client socket connection when using...
>
> >   ok = inet:setopts(Socket, [{packet, http_bin}, {keepalive, true}]),
> >   case gen_tcp:recv(Socket, 0, Timeout) of
> >   ...
>
> > ...possible bug in R13B01, or perhaps a bug in my usage of the socket?
>
> > /sd
>
> > On Jul 6, 11:20 am, Joel Reymont <joe...@...> wrote:
>
> > > I'm playing with Serge's "Non-blocking TCP server using OTP  
> > > principles" [1]. When spawning a large number of connections almost at  
> > > once, I'm seeing tcp_close messages on the client end, even before  
> > > seeing inet_async on the client end.
>
> > > I may spawn 700 client connections and only half may go through, even  
> > > though my backlog option is 1024, e.g.
>
> > >      Opts = [binary,
> > >              {packet, 0},
> > >              {reuseaddr, true},
> > >              {keepalive, true},
> > >              {backlog, 1024},
> > >              {active, false}],
> > >      case gen_tcp:listen(Port, Opts) of
> > >          {ok, Socket} ->
> > >              %% Create first accepting process
> > >              {ok, Ref} = prim_inet:async_accept(Socket, -1),
> > > ...
>
> > > Any suggestions on what's going on?
>
> > ________________________________________________________________
> > erlang-questions mailing list. Seehttp://www.erlang.org/faq.html
> > erlang-questions (at) erlang.org
>
> ________________________________________________________________
> erlang-questions mailing list. Seehttp://www.erlang.org/faq.html
> erlang-questions (at) erlang.org

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Jim Morris-3]

Joel, one other thing to try is use [kernel-poll:true] by adding +K
true to the erl command line, on both the client and server, see if
that helps.

I just double checked and I can open 1000 virtually simultaneous
connections (sleep(1) between each open on client) using the code from
http://www.trapexit.org/Building_a_Non-blocking_TCP_server_using_OTP_principles
with no problems on my Linux box using +K true. (Haven't tried without
that though).



On Jul 8, 9:29 am, Joel Reymont <joe...@...> wrote:

> On Jul 8, 2009, at 4:45 PM, Trevor Woollacott wrote:
>
> > Thats true, running out of file descriptors is likely. If that is  
> > the case then you should be seeing EMFILE or ENFILE errors.
>
> This is not the case in my test, although I did report a related issue:
>
> Mac OSX 10.5.7
>
> ulimit -n
> 10240
>
> Erlang R13B01 (erts-5.7.2) [source] [smp:2:2] [rq:2] [async-threads:0]  
> [kernel-poll:true]
>
> =ERROR REPORT==== 6-Jul-2009::20:51:54 ===
> driver_select(0x00000041, 1024, ERL_DRV_WRITE ERL_DRV_USE, 1) by  
> tcp_inet driver #Port<0.3137> failed: fd=1024 is larger than the  
> largest allowed fd=1023
>
> =ERROR REPORT==== 6-Jul-2009::20:51:55 ===
> File operation error: system_limit. Target: s3erl/ebin/random.beam.  
> Function: get_file. Process: code_server.
>
> ---
> Mac hacker with a performance benthttp://www.linkedin.com/in/joelreymont
>
> ________________________________________________________________
> erlang-questions mailing list. Seehttp://www.erlang.org/faq.html
> erlang-questions (at) erlang.org

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Joel Reymont]

Jim,

Take a look at the Erlang prompt right below where ulimit -n gives  
10240, i.e.

>> Erlang R13B01 (erts-5.7.2) [source] [smp:2:2] [rq:2] [async-threads:
>> 0]
>> [kernel-poll:true]

I thought this means I do have kernel poll enabled. I should since I  
pass +K.

The part that I'm surprised about is that select is still used.


On Jul 9, 2009, at 12:55 AM, Jim Morris wrote:

> Joel, one other thing to try is use [kernel-poll:true] by adding +K
> true to the erl command line, on both the client and server, see if
> that helps.
>
> I just double checked and I can open 1000 virtually simultaneous
> connections (sleep(1) between each open on client) using the code from
> http://www.trapexit.org/Building_a_Non-blocking_TCP_server_using_OTP_principles
> with no problems on my Linux box using +K true. (Haven't tried without
> that though).
>
>
>
> On Jul 8, 9:29 am, Joel Reymont <joe...@...> wrote:
>> On Jul 8, 2009, at 4:45 PM, Trevor Woollacott wrote:
>>
>>> Thats true, running out of file descriptors is likely. If that is
>>> the case then you should be seeing EMFILE or ENFILE errors.
>>
>> This is not the case in my test, although I did report a related  
>> issue:
>>
>> Mac OSX 10.5.7
>>
>> ulimit -n
>> 10240
>>
>> Erlang R13B01 (erts-5.7.2) [source] [smp:2:2] [rq:2] [async-threads:
>> 0]
>> [kernel-poll:true]
>>
>> =ERROR REPORT==== 6-Jul-2009::20:51:54 ===
>> driver_select(0x00000041, 1024, ERL_DRV_WRITE ERL_DRV_USE, 1) by
>> tcp_inet driver #Port<0.3137> failed: fd=1024 is larger than the
>> largest allowed fd=1023
>>
>> =ERROR REPORT==== 6-Jul-2009::20:51:55 ===
>> File operation error: system_limit. Target: s3erl/ebin/random.beam.
>> Function: get_file. Process: code_server.
>>
>> ---
>> Mac hacker with a performance benthttp://www.linkedin.com/in/joelreymont
>>
>> ________________________________________________________________
>> erlang-questions mailing list. Seehttp://www.erlang.org/faq.html
>> erlang-questions (at) erlang.org
>
> ________________________________________________________________
> erlang-questions mailing list. See http://www.erlang.org/faq.html
> erlang-questions (at) erlang.org
>

---
Mac hacker with a performance bent
http://www.linkedin.com/in/joelreymont


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org