kpoll still using select

[Joel Reymont]

Any suggestions?

        Thanks in advance, Joel

---

Mac OSX 10.5.7

ulimit -n
10240

Erlang R13B01 (erts-5.7.2) [source] [smp:2:2] [rq:2] [async-threads:0]  
[kernel-poll:true]

=ERROR REPORT==== 6-Jul-2009::20:51:54 ===
driver_select(0x00000041, 1024, ERL_DRV_WRITE ERL_DRV_USE, 1) by  
tcp_inet driver #Port<0.3137> failed: fd=1024 is larger than the  
largest allowed fd=1023

=ERROR REPORT==== 6-Jul-2009::20:51:55 ===
File operation error: system_limit. Target: s3erl/ebin/random.beam.  
Function: get_file. Process: code_server.

---
Mac hacker with a performance bent
http://www.linkedin.com/in/joelreymont


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Sverker Eriksson]

Joel Reymont wrote:

> Mac OSX 10.5.7
> ulimit -n
> 10240
> Erlang R13B01 (erts-5.7.2) [source] [smp:2:2] [rq:2] [async-threads:0]
> [kernel-poll:true]
>
> =ERROR REPORT==== 6-Jul-2009::20:51:54 ===
> driver_select(0x00000041, 1024, ERL_DRV_WRITE ERL_DRV_USE, 1) by
> tcp_inet driver #Port<0.3137> failed: fd=1024 is larger than the
> largest allowed fd=1023
>
> =ERROR REPORT==== 6-Jul-2009::20:51:55 ===
> File operation error: system_limit. Target: s3erl/ebin/random.beam.
> Function: get_file. Process: code_server.

A late answer:

What makes you think select is still used? driver_select will use select,
poll or any kernel poll variant depending on configuration and if kernel
poll is turn on or not. In case of kernel poll, driver_select may in some
cases fall back to use select or poll for individual file descriptors if the
OS does not support kernel poll for that device.


/Sverker, Erlang/OTP, Ericsson


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Joel Reymont]

On Jul 9, 2009, at 10:28 AM, Sverker Eriksson wrote:

> In case of kernel poll, driver_select may in some
> cases fall back to use select or poll for individual file  
> descriptors if the
> OS does not support kernel poll for that device.


./configure --enable-threads --enable-smp-support --enable-kernel-poll  
--enable-hipe
...
checking for working poll()... broken or based on select()
checking whether kqueue() fd can be select()ed on... yes
checking whether kernel poll support should be enabled... yes; kqueue
...

It should be using kqueue and the Erlang prompt reports that it's  
using kqueue. In fact, I distinctly remember being able to go to at  
least 20k connections (descriptors) when banging on OpenPoker last year.

It puzzles me greatly that I cannot do better than 1024 this time  
around.

Any suggestions on how to troubleshoot?

        Thanks, Joel

---
Mac hacker with a performance bent
http://www.linkedin.com/in/joelreymont


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Sverker Eriksson]

Joel Reymont wrote:

>
> On Jul 9, 2009, at 10:28 AM, Sverker Eriksson wrote:
>
>> In case of kernel poll, driver_select may in some
>> cases fall back to use select or poll for individual file descriptors
>> if the
>> OS does not support kernel poll for that device.
>
> ./configure --enable-threads --enable-smp-support --enable-kernel-poll
> --enable-hipe
> ...
> checking for working poll()... broken or based on select()
> checking whether kqueue() fd can be select()ed on... yes
> checking whether kernel poll support should be enabled... yes; kqueue
> ...
>
> It should be using kqueue and the Erlang prompt reports that it's
> using kqueue. In fact, I distinctly remember being able to go to at
> least 20k connections (descriptors) when banging on OpenPoker last year.
>
> It puzzles me greatly that I cannot do better than 1024 this time around.
>
> Any suggestions on how to troubleshoot?
>

Put some printf's in erts/emulator/sys/common/erl_poll.c. This is where
the file descriptor limit is set at start:

void
ERTS_POLL_EXPORT(erts_poll_init)(void)
{
    erts_smp_spinlock_init(&pollsets_lock, "pollsets_lock");
    pollsets = NULL;

    errno = 0;

#if defined(VXWORKS)
    max_fds = erts_vxworks_max_files;
#elif !defined(NO_SYSCONF)
    max_fds = sysconf(_SC_OPEN_MAX);
#elif ERTS_POLL_USE_SELECT
    max_fds = NOFILE;
#else
    max_fds = OPEN_MAX;
#endif

#if ERTS_POLL_USE_SELECT && defined(FD_SETSIZE)
    if (max_fds > FD_SETSIZE)
    max_fds = FD_SETSIZE;
#endif

    if (max_fds < 0)
    fatal_error("erts_poll_init(): Failed to get max number of files: %s\n",
            erl_errno_id(errno));

#ifdef ERTS_POLL_DEBUG_PRINT
    print_misc_debug_info();
#endif
}


/Sverker, Erlang/OTP


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Joel Reymont]

On Jul 9, 2009, at 11:08 AM, Sverker Eriksson wrote:

> Put some printf's in erts/emulator/sys/common/erl_poll.c. This is  
> where the file descriptor limit is set at start:


Would this work instead?

Breakpoint 2, driver_select (port=86, event=86, mode=86, on=1) at sys/
unix/sys.c:275
275 }
(gdb) p max_fds
$3 = -1

        Thanks, Joel

---
Mac hacker with a performance bent
http://www.linkedin.com/in/joelreymont


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Joel Reymont]

On Jul 9, 2009, at 11:08 AM, Sverker Eriksson wrote:

> Put some printf's in erts/emulator/sys/common/erl_poll.c. This is  
> where the file descriptor limit is set at start:


Another data point:

(gdb) p (int)erts_poll_max_fds_kp()
$2 = 1024

Why?

---
Mac hacker with a performance bent
http://www.linkedin.com/in/joelreymont


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Joel Reymont]

It goes like this...

Breakpoint 2, erts_poll_init_kp () at sys/common/erl_poll.c:2174
2174    max_fds = sysconf(_SC_OPEN_MAX);

reakpoint 3, erts_poll_init_kp () at sys/common/erl_poll.c:2183
2183 max_fds = FD_SETSIZE;
(gdb) p max_fds
$1 = -1

kernel poll is enabled with kqueue (as per configure), thus the _kp  
suffix.

ERTS_POLL_USE_SELECT is still defined, though, thus the clipping.

I have a strange feeling of deja vu, having had looked into this last  
year.

I remember that ERTS_POLL_USE_SELECT must be defined because kernel  
poll may fall back to select under certain exceptional circumstances.  
I don't understand why I'm forced to 1024 fds under normal kernel poll  
conditions, though.

Any explanation or workarounds?

        Thanks, Joel

---

2171 #if defined(VXWORKS)
2172    max_fds = erts_vxworks_max_files;
2173 #elif !defined(NO_SYSCONF)
2174    max_fds = sysconf(_SC_OPEN_MAX);
2175 #elif ERTS_POLL_USE_SELECT
2176    max_fds = NOFILE;
2177 #else
2178    max_fds = OPEN_MAX;
2179 #endif
2180
2181 #if ERTS_POLL_USE_SELECT && defined(FD_SETSIZE)
2182    if (max_fds > FD_SETSIZE)
2183 max_fds = FD_SETSIZE;
2184 #endif
2185
2186    if (max_fds < 0)
2187 fatal_error("erts_poll_init(): Failed to get max number of  
files: %s\n",
2188    erl_errno_id(errno));

---
Mac hacker with a performance bent
http://www.linkedin.com/in/joelreymont


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Joel Reymont]

It's all coming back to me...

http://erlang.org/pipermail/erlang-questions/2008-August/037574.html
http://erlang.org/pipermail/erlang-questions/2008-August/037595.html

I think it's wrong for the OTP team to hide behind broken poll on Mac  
OSX to clip fds to 1024 when kernel poll is used. The whole point of  
using kernel poll is to effectively manage more than 1024 descriptors.

I also think it's wrong to force users to edit system header files to  
bump the available number of file descriptors. I bumped FD_SETSIZE to  
30720 in /usr/include/sys/select.h (/usr/include/sys/_structs.h  
technically) so select may choke if it falls back.

I'm fine with select choking on a larger fd if it falls back. I don't  
want any select fallbacks, otherwise I would not be using kernel poll.

---
Mac hacker with a performance bent
http://www.linkedin.com/in/joelreymont


________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Sverker Eriksson]

Joel Reymont wrote:

> It's all coming back to me...
>
> http://erlang.org/pipermail/erlang-questions/2008-August/037574.html
> http://erlang.org/pipermail/erlang-questions/2008-August/037595.html
>
> I think it's wrong for the OTP team to hide behind broken poll on Mac
> OSX to clip fds to 1024 when kernel poll is used. The whole point of
> using kernel poll is to effectively manage more than 1024 descriptors.
>
> I also think it's wrong to force users to edit system header files to
> bump the available number of file descriptors. I bumped FD_SETSIZE to
> 30720 in /usr/include/sys/select.h (/usr/include/sys/_structs.h
> technically) so select may choke if it falls back.
>
> I'm fine with select choking on a larger fd if it falls back. I don't
> want any select fallbacks, otherwise I would not be using kernel poll.
>
>

Disabling fallback on select is not an option. That will fail to poll
the tty device and make the emulator unable to read from STDIN.

To make select fail for large fd's would probably work better. But you
will have an emulator with unstable semantics. driver_select may fail
arbitrary depending on the value the fd happended to get. If you want
this, you can patch erl_poll.c as suggested in the earlier thread. You
don't have to edit any system headers.

Let's hope for a healthier poll() in the next release of OSX.

/Sverker, Erlang/OTP




________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org

[Sverker Eriksson]

I wrote:
> You don't have to edit any system headers.
>
Incorrect. I see your point.
You patch FD_SETSIZE to make the bit-mask in fd_set larger and avoid
memory overwrites by FD_SET and FD_CLR.

But that will give you even more dangerous semantics. A call to
driver_select with a large fd that kernel poll does not support
will be silently accepted but you will never get any events from it.

What would be needed is a patch to erl_poll.c that checks fd against
FD_SETSIZE before calling FD_SET or FD_CLR.

/Sverker, Erlang/OTP

________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org