maximum number of outgoing connections

[Igor Sysoev]

It seems that FreeBSD can not make more than

net.inet.ip.portrange.last - net.inet.ip.portrange.first

simultaneous outgoing connections, i.e., no more than about 64k.

If I made ~64000 connections 127.0.0.1:XXXX > 127.0.0.1:80, then
connect() to an external address returns EADDRNOTAVAIL.

net.inet.ip.portrange.randomized is 0.

sockets, etc. are enough:

ITEM        SIZE     LIMIT      USED      FREE  REQUESTS  FAILURES
socket:      356,   204809,    13915,   146443, 148189452,        0
inpcb:       180,   204820,    20375,   137277, 147631805,        0
tcpcb:       464,   204800,    13882,   142102, 147631805,        0
tcptw:        48,    41028,     6493,    11213, 29804665,        0

I saw it on 6.2-STABLE.


--
Igor Sysoev
http://sysoev.ru/en/
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Tom Judge]

Igor Sysoev wrote:

> It seems that FreeBSD can not make more than
>
> net.inet.ip.portrange.last - net.inet.ip.portrange.first
>
> simultaneous outgoing connections, i.e., no more than about 64k.
>
> If I made ~64000 connections 127.0.0.1:XXXX > 127.0.0.1:80, then
> connect() to an external address returns EADDRNOTAVAIL.
>
> net.inet.ip.portrange.randomized is 0.
>
> sockets, etc. are enough:
>
> ITEM        SIZE     LIMIT      USED      FREE  REQUESTS  FAILURES
> socket:      356,   204809,    13915,   146443, 148189452,        0
> inpcb:       180,   204820,    20375,   137277, 147631805,        0
> tcpcb:       464,   204800,    13882,   142102, 147631805,        0
> tcptw:        48,    41028,     6493,    11213, 29804665,        0
>
> I saw it on 6.2-STABLE.
>
>

In an ideal world (Not sure if this is quite correct for FreeBSD) TCP
connections are tracked with a pair of tupels  source-addr:src-port ->
dst-addr:dst-port

As your always connecting to the same destination service 127.0.0.1:80
and always from the same source IP 127.0.0.1 then you only have one
variable left to change, the source port.  If you where to use the hole
of the whole of the port range minus the reserved ports you would only
ever be able to make 64512 simultaneous connections.  In order to make
more connections the first thing that you may want to start changing is
the source IP. If you added a second IP to you lo0 interface (say
127.0.0.2) and used a round robin approach to making your out bound
connections then you could make around 129k outbound connections.

I am not sure if there are any other constraints that need to be taken
into account such as the maximum number of sockets, RAM etc....

Tom
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Igor Sysoev]

On Mon, Aug 20, 2007 at 05:19:14PM +0100, Tom Judge wrote:

> Igor Sysoev wrote:
> >It seems that FreeBSD can not make more than
> >
> >net.inet.ip.portrange.last - net.inet.ip.portrange.first
> >
> >simultaneous outgoing connections, i.e., no more than about 64k.
> >
> >If I made ~64000 connections 127.0.0.1:XXXX > 127.0.0.1:80, then
> >connect() to an external address returns EADDRNOTAVAIL.
> >
> >net.inet.ip.portrange.randomized is 0.
> >
> >sockets, etc. are enough:
> >
> >ITEM        SIZE     LIMIT      USED      FREE  REQUESTS  FAILURES
> >socket:      356,   204809,    13915,   146443, 148189452,        0
> >inpcb:       180,   204820,    20375,   137277, 147631805,        0
> >tcpcb:       464,   204800,    13882,   142102, 147631805,        0
> >tcptw:        48,    41028,     6493,    11213, 29804665,        0
> >
> >I saw it on 6.2-STABLE.
> >
> >
>
> In an ideal world (Not sure if this is quite correct for FreeBSD) TCP
> connections are tracked with a pair of tupels  source-addr:src-port ->
> dst-addr:dst-port
>
> As your always connecting to the same destination service 127.0.0.1:80
> and always from the same source IP 127.0.0.1 then you only have one
> variable left to change, the source port.  If you where to use the hole
> of the whole of the port range minus the reserved ports you would only
> ever be able to make 64512 simultaneous connections.  In order to make
> more connections the first thing that you may want to start changing is
> the source IP. If you added a second IP to you lo0 interface (say
> 127.0.0.2) and used a round robin approach to making your out bound
> connections then you could make around 129k outbound connections.

Connections to 127.0.0.1 were via lo0, external connections are via bge0.

> I am not sure if there are any other constraints that need to be taken
> into account such as the maximum number of sockets, RAM etc....

No, there are no constraints in memory, sockets, mbufs, clusters, etc.
If there's contraint in memory, then FreeBSD simply panics.
If there's contraint in mbuf clusters, then process stucks in zonelimit
state forever.

I suspect that local address in in_pcbbind_setup() is 0.0.0.0 so there
is 64K limit.


--
Igor Sysoev
http://sysoev.ru/en/
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[John-Mark Gurney]

Igor Sysoev wrote this message on Mon, Aug 20, 2007 at 19:11 +0400:
> It seems that FreeBSD can not make more than
>
> net.inet.ip.portrange.last - net.inet.ip.portrange.first
>
> simultaneous outgoing connections, i.e., no more than about 64k.
>
> If I made ~64000 connections 127.0.0.1:XXXX > 127.0.0.1:80, then
> connect() to an external address returns EADDRNOTAVAIL.

Isn't this more of a limitation of TCP/IP than FreeBSD?  because you
need to treat the srcip/srcport/dstip/dstport as a unique value, and
in your test, you are only changing one of the four...  Have you tried
running a second we server on port 8080, and see if you can connect
another ~64000 connections to that port too?

--
  John-Mark Gurney Voice: [phone]

     "All that I will do, has been done, All that I have, has not."
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Igor Sysoev]

On Mon, Aug 20, 2007 at 09:53:55AM -0700, John-Mark Gurney wrote:

> Igor Sysoev wrote this message on Mon, Aug 20, 2007 at 19:11 +0400:
> > It seems that FreeBSD can not make more than
> >
> > net.inet.ip.portrange.last - net.inet.ip.portrange.first
> >
> > simultaneous outgoing connections, i.e., no more than about 64k.
> >
> > If I made ~64000 connections 127.0.0.1:XXXX > 127.0.0.1:80, then
> > connect() to an external address returns EADDRNOTAVAIL.
>
> Isn't this more of a limitation of TCP/IP than FreeBSD?  because you
> need to treat the srcip/srcport/dstip/dstport as a unique value, and
> in your test, you are only changing one of the four...  Have you tried
> running a second we server on port 8080, and see if you can connect
> another ~64000 connections to that port too?

No, TCP/IP limitation is for XXXX in 127.0.0.1:XXXX <> 127.0.0.1:80,
but FreeBSD limits all outgoing connections to the port range, i.e.

    local part      remote part
  127.0.0.1:5000 <> 127.0.0.1:80
192.168.1.1:5000 <> 10.0.0.1:25

can not exist simultaneously, if both connections were started from
local host.

I can not write a simple test-case program, but I can offer simple setup:

cd /usr/ports/www/nginx && make install

create simple nginx.conf:

------------
events {
    worker_connections  20000;
}

http {
    server {
        listen        8080;
        server_name   test;

        location = /loop {
            proxy_pass  http://127.0.0.1:8080;

            error_page  502 = /yahoo;
        }

        location = /yahoo {
            proxy_pass  http://www.yahoo.com;
        }
    }
}
------------

set

sysctl net.inet.ip.portrange.randomized=0
sysctl net.inet.ip.portrange.first=1024
sysctl net.inet.ip.portrange.last=5000

to see the case with default small number of files, sockets, etc.

and run as root:

/usr/local/sbin/nginx -c ./nginx.conf

then ask http://host:8080/loop in browser. nginx will cycle to itslef, then
after first error

2007/08/20 22:05:16 [crit] 29669#0: *94165 connect() to 127.0.0.1:8080 failed (49: Can't assign requested address) while connecting to upstream, client: 127.0.0.1, server: test, URL: "/loop", upstream: "http://127.0.0.1:8080/loop", host: "127.0.0.1:8080"

you will see the second error:

2007/08/20 22:05:16 [crit] 29669#0: *94165 connect() to 87.248.113.14:80 failed (49: Can't assign requested address) while connecting to upstream, client: 127.0.0.1, server: test, URL: "/loop", upstream: "http://87.248.113.14:80/loop", host: "127.0.0.1:8080"

If you think it may be nginx fault, run this under ktrace/truss and see
syscalls.


--
Igor Sysoev
http://sysoev.ru/en/
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Igor Sysoev]

On Mon, Aug 20, 2007 at 10:30:12PM +0400, Igor Sysoev wrote:

> On Mon, Aug 20, 2007 at 09:53:55AM -0700, John-Mark Gurney wrote:
>
> > Igor Sysoev wrote this message on Mon, Aug 20, 2007 at 19:11 +0400:
> > > It seems that FreeBSD can not make more than
> > >
> > > net.inet.ip.portrange.last - net.inet.ip.portrange.first
> > >
> > > simultaneous outgoing connections, i.e., no more than about 64k.
> > >
> > > If I made ~64000 connections 127.0.0.1:XXXX > 127.0.0.1:80, then
> > > connect() to an external address returns EADDRNOTAVAIL.
> >
> > Isn't this more of a limitation of TCP/IP than FreeBSD?  because you
> > need to treat the srcip/srcport/dstip/dstport as a unique value, and
> > in your test, you are only changing one of the four...  Have you tried
> > running a second we server on port 8080, and see if you can connect
> > another ~64000 connections to that port too?
>
> No, TCP/IP limitation is for XXXX in 127.0.0.1:XXXX <> 127.0.0.1:80,
> but FreeBSD limits all outgoing connections to the port range, i.e.
>
>     local part      remote part
>   127.0.0.1:5000 <> 127.0.0.1:80
> 192.168.1.1:5000 <> 10.0.0.1:25
>
> can not exist simultaneously, if both connections were started from
> local host.

To be exact - if connect() was called on unbound socket.


--
Igor Sysoev
http://sysoev.ru/en/
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Igor Sysoev]

On Mon, Aug 20, 2007 at 08:34:43PM +0400, Igor Sysoev wrote:

> On Mon, Aug 20, 2007 at 05:19:14PM +0100, Tom Judge wrote:
>
> > Igor Sysoev wrote:
> > >It seems that FreeBSD can not make more than
> > >
> > >net.inet.ip.portrange.last - net.inet.ip.portrange.first
> > >
> > >simultaneous outgoing connections, i.e., no more than about 64k.
> > >
> > >If I made ~64000 connections 127.0.0.1:XXXX > 127.0.0.1:80, then
> > >connect() to an external address returns EADDRNOTAVAIL.
> > >
> > >net.inet.ip.portrange.randomized is 0.
> > >
> > >sockets, etc. are enough:
> > >
> > >ITEM        SIZE     LIMIT      USED      FREE  REQUESTS  FAILURES
> > >socket:      356,   204809,    13915,   146443, 148189452,        0
> > >inpcb:       180,   204820,    20375,   137277, 147631805,        0
> > >tcpcb:       464,   204800,    13882,   142102, 147631805,        0
> > >tcptw:        48,    41028,     6493,    11213, 29804665,        0
> > >
> > >I saw it on 6.2-STABLE.
> > >
> > >
> >
> > In an ideal world (Not sure if this is quite correct for FreeBSD) TCP
> > connections are tracked with a pair of tupels  source-addr:src-port ->
> > dst-addr:dst-port
> >
> > As your always connecting to the same destination service 127.0.0.1:80
> > and always from the same source IP 127.0.0.1 then you only have one
> > variable left to change, the source port.  If you where to use the hole
> > of the whole of the port range minus the reserved ports you would only
> > ever be able to make 64512 simultaneous connections.  In order to make
> > more connections the first thing that you may want to start changing is
> > the source IP. If you added a second IP to you lo0 interface (say
> > 127.0.0.2) and used a round robin approach to making your out bound
> > connections then you could make around 129k outbound connections.
>
> Connections to 127.0.0.1 were via lo0, external connections are via bge0.
>
> > I am not sure if there are any other constraints that need to be taken
> > into account such as the maximum number of sockets, RAM etc....
>
> No, there are no constraints in memory, sockets, mbufs, clusters, etc.
> If there's contraint in memory, then FreeBSD simply panics.
> If there's contraint in mbuf clusters, then process stucks in zonelimit
> state forever.
>
> I suspect that local address in in_pcbbind_setup() is 0.0.0.0 so there
> is 64K limit.

Recently I looked the issue again and find (with Ruslan Ermilov's help)
that if I set the SO_REUSEADDR option, FreeBSD allows to use the same
local port for different connections:

        192.168.1.1:5000  > 10.0.0.1:80
        192.168.1.1:5000  > 10.0.0.2:80

Linux allows this by default. I believe FreeBSD should set SO_REUSEADDR
internally while connect().

BTW, bind()ing socket to a local address does not help.


--
Igor Sysoev
http://sysoev.ru/en/
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."